A motion control model training method and device, and a robot motion control method

Abstract

The application relates to a motion control model training method and device and a robot motion control method, wherein the motion control model training method comprises the following steps: at each time step, a history ontology observation sequence of a robot is acquired and input into a variational autoencoder to generate environment encoding; ontology observation data and the environment encoding at a current time step are input into a policy network to output an action instruction to control the motion of the robot; current observation data and privileged information are input into a critic network to evaluate state value; actual observation data and a termination flag at a next time step are collected to determine a task reward value; a style reward value is determined according to the current observation data and the actual observation data, and a comprehensive reward value is obtained by combining the task reward value; the above data is stored as experience data, and a motion control model is trained based on experience data of multiple time steps. Through the application, the policy network can perceive the physical environment and generate a natural and coordinated motion style.

Description

Technical Field

[0001] This application relates to the field of robot perception and motion control technology, and in particular to a motion control model training method and device, and a robot motion control method. Background Technology

[0002] Traditional robot motion control methods, such as model predictive control, require accurate robot dynamics models and have poor adaptability to complex terrains and nonlinear systems. In recent years, reinforcement learning-based control methods have attracted widespread attention because they can automatically learn complex strategies through interaction with the environment. However, applying reinforcement learning to real-world quadruped robots still faces two major challenges: first, the gap between simulation and reality—the physical parameters in the simulation environment differ from those in the real world, causing a sharp decline in performance when strategies trained in simulation are deployed on real robots; second, unnatural movements—policies trained purely through reinforcement learning, in order to maximize task rewards, may produce unnatural movements such as high-frequency tremors and stiff gait, which not only affect performance but may also damage the robot.

[0003] Currently, a common solution in related technologies is domain randomization, which randomizes various physical parameters during simulation training, hoping that the trained policy can cover the possible parameter range in the real world. Another approach is system identification, which first estimates the robot's real physical parameters through specific steps, and then uses these parameters to configure the simulation environment or adjust the policy. However, the policy trained by the domain randomization method is usually an average policy for all randomization cases, and cannot be finely adjusted according to changes in the actual physical environment after deployment. The system identification method requires additional and time-consuming identification processes and cannot cope with dynamically changing environments. At the same time, the core optimization goal of the above two approaches is usually to complete a specific task, which can easily lead to unnatural and uncoordinated actions by the policy.

[0004] Currently, no effective solution has been proposed for the problem of how to enable the motion strategy of a quadruped robot trained by reinforcement learning to adapt to different physical environment parameters and achieve efficient and robust transfer from simulation to the real world without explicit system identification and ensuring the naturalness of motion. Summary of the Invention

[0005] This application provides a motion control model training method and apparatus, and a robot motion control method, to at least solve the problem in related technologies of how to enable the motion strategy of a quadruped robot trained by reinforcement learning to adapt to different physical environment parameters without explicit system identification and ensuring the naturalness of motion.

[0006] In a first aspect, embodiments of this application provide a method for training a motion control model, wherein the motion control model includes a variational autoencoder, an adversarial motion prior discriminator, and a policy evaluation model; the policy evaluation model includes a policy network and a critic network; the method includes:

[0007] The motion control model includes a variational autoencoder and a policy evaluation model; the policy evaluation model includes a policy network and a critic network; the method includes:

[0008] At each time step, the robot's historical ontology observation sequence is acquired and input into the variational autoencoder. The variational autoencoder generates an environment code and predicts the predicted ontology observation data for the next time step based on the environment code.

[0009] The ontology observation data at the current time step and the environment code are input into the policy network, which outputs action commands and controls the robot to execute the action commands.

[0010] The robot's current observation data and privileged information are input into the critic network, which then evaluates the state value.

[0011] Collect the actual observation data and termination flag at the next time step after the robot executes the action command, and determine the task reward value at the current time step;

[0012] Based on the current observation data and the actual observation data, the adversarial motion prior discriminator determines the style reward value at the current time step, and determines the comprehensive reward value at the current time step based on the style reward value and the task reward value; the style reward value is used to guide the policy network to generate action instructions that approximate a preset reference motion trajectory;

[0013] The historical ontology observation sequence, the environment code, the predicted ontology observation data, the current observation data, the action command, the comprehensive reward value, the actual observation data, the termination flag, and the state value are used as empirical data at the current time step.

[0014] The motion control model is trained based on empirical data at multiple time steps to obtain a well-trained motion control model.

[0015] Secondly, embodiments of this application provide a motion control model training device, wherein the motion control model includes a variational autoencoder, an adversarial motion prior discriminator, and a policy evaluation model; the policy evaluation model includes a policy network and a critic network; the device includes: an acquisition module, a processing module, and a training module;

[0016] The acquisition module is used to acquire the robot's historical ontology observation sequence at each time step and input it into the variational autoencoder. The variational autoencoder generates an environment code and predicts the predicted ontology observation data for the next time step based on the environment code.

[0017] The processing module is used to input the ontology observation data and the environment code at the current time step into the policy network, and the policy network outputs action commands and controls the robot to execute the action commands.

[0018] The processing module is also used to input the robot's current observation data and privileged information into the commentator network, so that the commentator network can evaluate the state value;

[0019] The acquisition module is also used to collect the actual observation data and termination flag at the next time step after the robot executes the action command, and to determine the task reward value at the current time step;

[0020] The processing module is further configured to determine the style reward value at the current time step by the adversarial motion prior discriminator based on the current observation data and the actual observation data, and to determine the comprehensive reward value at the current time step based on the style reward value and the task reward value; the style reward value is used to guide the policy network to generate action instructions that approximate a preset reference motion trajectory;

[0021] The processing module is further configured to use the historical ontology observation sequence, the environment code, the predicted ontology observation data, the current observation data, the action command, the comprehensive reward value, the actual observation data, the termination flag, and the state value as empirical data at the current time step;

[0022] The training module is used to train the motion control model based on empirical data at multiple time steps to obtain a trained motion control model.

[0023] Thirdly, embodiments of this application provide a robot motion control method, the robot motion control method comprising:

[0024] Obtain the robot's historical ontological observation sequence;

[0025] The historical ontology observation sequence is input into the variational autoencoder in the trained motion control model, so that the variational autoencoder obtains the environment code based on the historical ontology observation sequence; wherein, the motion control model is trained by the motion control model training method described in the first aspect;

[0026] The ontology observation data at the current time step and the environment encoding are input into the policy network in the motion control model so that the policy network can output action commands.

[0027] The robot is controlled to move according to the action instructions.

[0028] Compared to related technologies, the motion control model training method and device, and robot motion control method provided in this application introduce a variational autoencoder to implicitly extract environmental codes from the robot's historical ontology observation sequence and combine them with a policy network to achieve adaptive control. At the same time, the style reward value guides the policy network to generate motion instructions for natural style motion. This solves the problem in related technologies that require explicit system identification during the deployment stage or cannot cope with dynamic environmental changes. It achieves adaptation to the changing physical environment without the need for system identification and the naturalization of motion style, thereby improving the success rate of policy transfer from simulation to the real world.

[0029] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0030] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0031] Figure 1 A flowchart of the motion control model training method provided in this application;

[0032] Figure 2 A schematic diagram of the structure of a variational autoencoder is shown as an exemplary embodiment of this application;

[0033] Figure 3 This is a schematic diagram illustrating the global network structure and data flow direction as an exemplary embodiment of this application;

[0034] Figure 4 A structural block diagram of the motion control model training device provided in this application;

[0035] Figure 5 A flowchart of the robot motion control method provided in this application;

[0036] Figure 6 A hardware structure block diagram of the terminal for the motion control model training method provided in this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0038] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0039] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0040] To facilitate understanding of the technical solution of this application, the following terms used throughout the document will first be explained with a consistent meaning:

[0041] Body observation data: refers to data that can be directly measured by the sensors carried by the robot itself, including joint angles, joint angular velocities, body posture, body angular velocity, and gravity direction.

[0042] External observation data refers to the true values ​​of physical parameters that are available in the simulation training environment but cannot be accurately measured by the real robot's own sensors. These include the absolute linear velocity of the robot body, the coefficient of friction against the ground, the true value of the robot body mass, and the true value of the center of mass position. External observation data is mainly used for supervision of explicit estimation by variational autoencoders and for constructing policy state transition pairs for adversarial motion prior discriminators, and is only used during the training phase.

[0043] Current observation data: refers to the collection of ontological observation data and external observation data at the current time step.

[0044] Actual observation data refers to the observation data returned by the environment at the next time step after the robot executes the action command. During the training phase, actual observation data includes both the actual body observation data and the actual external observation data at the next time step; during the deployment phase, actual observation data refers only to the actual body observation data at the next time step.

[0045] Privileged information refers to other environmental or state truth information provided in the simulation training environment that exceeds the scope of the aforementioned external observation data. Examples include low-dimensional compressed tensors of environmental elevation maps and low-dimensional privileged tensors of external perturbations. Privileged information is only used as input to the critic network to assist in more accurate state value assessment during the training phase; it is neither available nor needed during the deployment phase.

[0046] The above terms will be used consistently throughout the text and will not be repeated hereafter.

[0047] This embodiment provides a method for training a motion control model. Figure 1 The flowchart of the motion control model training method provided in this application is as follows: Figure 1 As shown, the process includes the following steps:

[0048] In step S110, at each time step, the robot's historical ontology observation sequence is acquired and input into the variational autoencoder. The variational autoencoder generates an environment code and predicts the predicted ontology observation data for the next time step based on the environment code.

[0049] Among them, the historical ontology observation sequence is composed of the current time step The time series consisting of the previous k consecutive time steps of ontology observation data can be represented as:

[0050] .

[0051] Specifically, at each time step, the robot's historical ontology observation sequence is... The input is fed to the variational autoencoder. The variational autoencoder takes the historical ontology observation sequence from the input. The system identifies implicit, latent information related to the characteristics of the current physical environment and generates an environmental code to characterize those characteristics. This environment code It can implicitly express the dynamic characteristics of the robot's current environment, such as the robot's load mass, center of mass position, and ground friction characteristics at the current time step. Subsequently, the decoder of the variational autoencoder is based on this environmental encoding. Decode and recover the data based on the environment encoding. By analyzing the implicit environmental context and system state evolution patterns, we can deduce the possible ontological observation state that the robot may reach in the next time step under the current environmental conditions, and thus output the predicted ontological observation data for the next time step. Through the above steps, by inputting the historical ontology observation sequence into the variational autoencoder and having the variational autoencoder identify the implicit physical environment feature information in the historical sequence, the problem of requiring explicit system identification or relying on additional sensors to obtain environmental parameters in related technologies is solved. This achieves online implicit perception of dynamically changing environmental parameters, enabling the robot strategy to autonomously adapt to different physical environment conditions without relying on external sensors or identification processes, and providing an effective information foundation for the adaptive decision-making of the subsequent policy network.

[0052] In step S120, the ontology observation data and environment code at the current time step are input into the policy network, which outputs action instructions and controls the robot to execute the action instructions; the robot's current observation data and privileged information are input into the critic network, which evaluates the state value.

[0053] Specifically, the policy network and the critic network constitute the Actor-Critic architecture. This involves using the ontology observation data at the current time step. and the environment code generated in step S110 The inputs are simultaneously fed into the policy network, which then performs reasoning based on these two parts of input information and outputs action commands. This action instruction An actuator model is applied to the robot, and through this actuator model, the motion is converted into specific target joint angles or target joint torques to drive the robot's movement. Simultaneously, current observation data and privileged information are input into a critic network, which outputs the state value corresponding to the current state. The value of this state. This network is used to measure the quality of the state under the current policy, providing a stable benchmark signal for subsequent parameter updates of the policy network. It should be noted that this critic network is only used during the training phase and does not need to be run during model deployment.

[0054] Through the above steps, the environment at the current time step is encoded. Compared with the ontology observation data at the current time step Both serve as inputs to the policy network, enabling the policy network to encode information based on the environment. The dynamic adjustment of the control output based on the current physical environment characteristics solves the problem that domain randomization strategies in related technologies cannot be finely adjusted for specific deployment environments, and achieves adaptive control for complex environmental conditions such as unknown loads and variable terrain. At the same time, during the training phase, by introducing privileged information into the commentator network, the commentator network can obtain more comprehensive environmental and state information than the policy network, thereby providing the policy network with more accurate gradient signals and further accelerating the training and convergence process of the policy network.

[0055] Step S130: Collect the actual observation data and termination flag at the next time step after the robot executes the action command, and determine the task reward value at the current time step; based on the current observation data and the actual observation data, the adversarial motion prior discriminator determines the style reward value at the current time step, and determines the comprehensive reward value at the current time step based on the style reward value and the task reward value; the above style reward value is used to guide the policy network to generate action commands that approach the preset reference motion trajectory.

[0056] Among them, the termination mark It refers to the end marker, used to indicate whether the current round ends due to the robot falling down, the task timeout, or other preset conditions.

[0057] Specifically, this step includes the following sub-steps:

[0058] Step S131: Collect the action commands output by the robot execution strategy network. Subsequently, the volumetric observation data returned by the simulation environment at the next time step. Task reward value and the termination mark Among them, the task reward value The reward function can be calculated based on a preset task objective. For example, the reward function can be designed based on indicators such as speed tracking error, energy consumption, or aircraft stability.

[0059] Step S132: Collect external observation data at the next time step, including the external observation data at the next time step and the ontological observation data at the previous next time step. This constitutes the actual observation data for the next time step mentioned above.

[0060] Step S133: The current observation data and the actual observation data are concatenated and input into the pre-built adversarial motion prior discriminator to obtain the style reward value at the current time step. This style's reward value This is used to guide the strategy network to generate motion instructions that approximate a preset reference motion trajectory.

[0061] Step S134, transfer the task reward value Style bonus value Perform a weighted summation to obtain the overall reward value at the current time step. , can be represented as:

[0062] ,in, Style-based reward weights.

[0063] Through the above steps, by introducing a style reward value into the comprehensive reward value, the naturalness of the motion style is incorporated into the optimization objective of reinforcement learning. This solves the problems of unnatural motion and high-frequency tremors caused by simply optimizing based on the task objective in related technologies. As a result, the policy network can achieve a smooth, coordinated, and natural motion style that approximates the preset reference motion trajectory while completing the task objective.

[0064] Step S140: The historical ontology observation sequence, environment encoding, predicted ontology observation data, current observation data, action instructions, comprehensive reward value, actual observation data, termination flag, and state value are used as empirical data for the current time step; the above motion control model is trained based on the empirical data for multiple time steps to obtain the trained motion control model.

[0065] In this context, empirical data refers to the complete set of interactive data generated at each time step during training, used for subsequent updates of relevant network parameters. Simultaneously, a corresponding empirical buffer can be deployed to store empirical data accumulated over multiple time steps, allowing each network to sample from this data for parameter updates during the training phase.

[0066] Specifically, at the end of each time step, the data generated in the current time step is uniformly encapsulated into a complete empirical data set and written into the empirical buffer. This complete empirical data set for the current time step may include: historical ontology observation sequences. Environment coding Predicted ontological observation data for the next time step Current observation data (volume observation data at the current time step) External observation data at the current time step), action commands Comprehensive reward value (Task reward value) Style reward value Actual observation data (body observation data at the next time step) (Next time step external observation data), termination marker and state value The external observation data may include the absolute linear velocity of the fuselage, the coefficient of friction with the ground, the true value of the fuselage mass, and the true value of the center of mass position.

[0067] Once a sufficient amount of experience data has accumulated in the experience buffer, experience data from multiple time steps is read to perform model updates.

[0068] In this embodiment of the application, during the training process, the variational autoencoder and the policy evaluation model can be jointly optimized, wherein the network parameters of the variational autoencoder are based on the historical ontology observation sequence. Predicted ontological observation data for the next time step Next time step, the body observation data The system updates its environmental feature extraction and future state prediction based on ontology observation data at the current time step, as well as external observation data. Action instructions Comprehensive reward value Next time step, the body observation data State value and termination mark Updates are performed to improve the policy network's task execution capabilities and adaptive control level. Simultaneously, the adversarial motion prior discriminator is iteratively updated based on policy state transition pairs composed of current and actual observation data to continuously improve its ability to discriminate the authenticity of motion styles.

[0069] It should be further clarified that, in this application, the pre-trained motion control model includes a pre-trained variational autoencoder and a pre-trained policy evaluation model, but does not include the critic network and the adversarial motion prior discriminator. In this application, the critic network is used to provide accurate state value estimates for the policy network during the training phase to guide the parameter updates of the policy network; the adversarial motion prior discriminator is used to evaluate the similarity between the generated motion and the expert's motion style during the training phase, thereby providing a style reward signal; both are only used during the training phase and are not required in actual deployment.

[0070] Through the above steps, by observing the historical ontology sequence Environment coding Predicted ontological observation data for the next time step Current observation data (volume observation data at the current time step) External observation data at the current time step), action commands Comprehensive reward value (Task reward value) Style reward value Actual observation data (body observation data at the next time step) (Next time step external observation data), termination marker and state value By uniformly encapsulating training data into empirical data and performing batch joint model updates based on the empirical buffer, the problem of scattered management and low update efficiency of multi-module training data in related technologies is solved, and standardized management and efficient reuse of training data are realized.

[0071] Through steps S110 to S140, environmental features are implicitly extracted from the historical ontology observation sequence by a variational autoencoder and an environmental code is generated. The policy network then combines the environmental code to output adaptive control action commands. At the same time, the style reward value and the task reward value together form a comprehensive reward value to guide the optimization of the policy network. This solves the problems in related technologies, such as the need for explicit system identification during the deployment stage, the inability of the policy to dynamically adapt to environmental changes, and unnatural motion under pure task reward drive. It realizes online implicit perception and adaptive control of the variable physical environment without the need for system identification, and generates smooth, coordinated, and natural motion that approaches the preset reference motion trajectory. Furthermore, by constructing an Actor-Critic architecture with the policy network and the critic network, and by introducing privileged information into the critic network for value evaluation during the training phase, the problem of slow training convergence caused by inaccurate policy evaluation is solved, and privileged information is used to accelerate convergence during training. In addition, by jointly optimizing the variational autoencoder and the policy evaluation model under the same training framework, the environmental perception capability and task execution capability are mutually promoted, solving the problem of limited overall performance when the two are trained independently in related technologies. End-to-end collaborative optimization is achieved, and the resulting motion control model has higher robustness and success rate in the transfer from simulation to the real world.

[0072] In some embodiments, a motion control model is trained based on empirical data at multiple time steps to obtain a trained motion control model, including:

[0073] The parameters of the variational autoencoder are updated based on historical ontology observation sequences, predicted ontology observation data, and actual observation data.

[0074] Based on the ontology observation data, action instructions, comprehensive rewards, termination flag, actual observation data, and parameters of the state value update strategy evaluation model at the current time step.

[0075] In this model, the network parameter updates of the variational autoencoder and the policy evaluation model can be performed within the same training iteration, and both can use the same optimizer. The update goal of the variational autoencoder is to continuously improve its ability to extract environmental features from historical ontology observation sequences and accurately predict future states, while the update goal of the policy evaluation model is to continuously improve the policy network's ability to complete task objectives and generate natural motion styles. The joint training of the two enables environmental perception and motion control to mutually promote and synergistically improve each other.

[0076] Specifically, during the training phase, when the training time reaches the preset update cycle, a batch of empirical data for each time step is read from the empirical buffer, and the network parameters of the variational autoencoder and the policy evaluation model are updated based on this batch of empirical data.

[0077] In detail, for the variational autoencoder, it uses historical ontological observation sequences. As input, the predicted ontology observation data at the following time step and the next time step of the body observation data The differences between them serve as the main basis for calculating the reconstruction loss, and the network parameters of the variational autoencoder are updated through backpropagation.

[0078] For the policy evaluation model, the proximal policy optimization (PPO) algorithm can be used, based on the ontology observation data at the current time step. Action instructions Comprehensive rewards Termination mark and state value The advantage function is calculated, and the policy loss and value loss are constructed. The parameters of the policy network and the critic network are updated under the constraint of the policy update magnitude. Furthermore, the introduced PPO algorithm, through a pruning mechanism, limits the difference between the old and new policies, effectively avoiding drastic fluctuations during the policy update process and ensuring training stability.

[0079] Through the above steps, by jointly updating the network parameters of the variational autoencoder and the policy evaluation model based on their respective empirical data, the problem of the difficulty in coordinating the overall performance of the environment perception module and the control policy module when they are trained independently in related technologies is solved. This achieves the synchronous improvement of environment perception accuracy and task execution capability, enabling the implicit perception of the environment by the variational autoencoder to effectively serve the adaptive decision-making of the policy network.

[0080] In some embodiments, the variational autoencoder includes an encoder, a linear processor, and a decoder; the variational autoencoder generates an environment code, and predicts the predicted ontology observation data for the next time step based on the environment code, including:

[0081] Feature vectors are obtained by extracting features from historical ontology observation sequences using an encoder;

[0082] The feature vector is processed by a linear processor to obtain explicit estimation information and implicit distribution parameters. The implicit distribution parameters include the mean and log-variance, which are used to characterize the distribution characteristics of the potential information in the feature vector that is related to the current physical environment.

[0083] A linear processor samples random noise vectors from a preset standard normal distribution, and implicit coding is obtained based on the mean, standard deviation, and random noise vectors mentioned above.

[0084] The explicit estimation information and implicit coding are concatenated by a linear processor to obtain the environment coding;

[0085] The environment encoding is decoded by a decoder to obtain the predicted ontology observation data.

[0086] The encoder described above can be implemented using a multilayer perceptron (MLP) to extract high-dimensional feature representations containing temporal information and environmental features from the historical sequencing ontology column. The linear processor described above can be implemented using multiple parallel linear layers to map feature vectors into output tensors with different functional uses. The decoder described above can also be implemented using a multilayer perceptron to reconstruct the prediction tensor of the ontology observation data for the next time step from the environmental encoding, thereby enabling the prediction of the robot's ontology observation data for the next time step.

[0087] Specifically, Figure 2 This is a schematic diagram of the structure of a variational autoencoder shown in an exemplary embodiment of this application, as follows: Figure 2 As shown, the variational autoencoder may include an encoder, a linear processor, and a decoder. Next, we will combine... Figure 2 The working process of the variational autoencoder in this application is described in detail, and the process may include the following sub-steps:

[0088] Step S210, the encoder receives data from continuous... The historical ontological observation sequence consists of ontological observation data at each time step. After undergoing multiple nonlinear transformations by the encoder, a dense feature vector is output. This feature vector contains key information extracted from historical sequences related to the robot's motion state and environmental interaction characteristics.

[0089] Step S220, the linear processor receives the feature vector. The mapping process is performed through multiple parallel linear layers.

[0090] In detail, step S220 may include the following sub-steps:

[0091] Step S221: Explicit estimation information is output through the first linear layer. This explicit estimation information is generated by the linear processor based on the feature vector. The physical parameter estimates are inferred from implicit features related to the current environment and robot dynamics. This explicit estimation information may include the estimated body velocity at the current time step. Estimated value of center of gravity shift and estimated fuselage mass Among them, the estimated fuselage speed Used to adjust the velocity tracking accuracy of the policy network; centroid offset estimate Used to enable the policy network to adapt to shifts in center of gravity caused by load changes or external disturbances; fuselage mass estimation. Used to enable the policy network to adapt to different physical loads.

[0092] It should be further noted that the first part of the linear layer consists of multiple parallel fully connected layers. The specific number of layers and the number of neurons in each layer can be set according to the actual implementation requirements. No specific limitation is made here, and it does not constitute a limitation on the scope of protection of this application.

[0093] Step S222: Output implicit distribution parameters through the second linear layer, which include the mean. Sum of logarithmic variance Used to characterize feature vectors The distribution characteristics of potential information related to the current physical environment.

[0094] Similarly, the second part of the linear layer consists of multiple parallel fully connected layers. The specific number of layers and the number of neurons in each layer can be set according to the actual implementation requirements. No specific limitation is made here, and it does not constitute a limitation on the scope of protection of this application.

[0095] In step S230, the linear processor processes the implicit distribution parameters to obtain the implicit encoding. .

[0096] Furthermore, implicit coding Different computation methods exist in the training and deployment phases:

[0097] (1) During the training phase, the implicit code can be obtained by sampling from the Gaussian distribution using the reparameterization technique. .

[0098] In detail, we can start with the standard normal distribution. Medium-sampled random noise vector Subsequently, based on the log-variance of the implicit distribution... Calculate its corresponding standard deviation , can be represented as:

[0099] ;

[0100] Finally, the implicit code is calculated according to the reparameterization formula. , can be represented as:

[0101] .

[0102] (2) During the deployment phase, since random exploration and distribution regularization training are no longer required after actual deployment, and it is necessary to ensure the consistency and determinism of the output under the same input, the mean can be directly used at this time. As implicit encoding .

[0103] Step S240, after obtaining the implicit encoding Then, the explicitly estimated information is processed by a linear processor. , , ) and implicit encoding The data is then concatenated to obtain the final environment code. , can be represented as:

[0104] .

[0105] Step S250, the decoder receives the environment code. The environment is encoded using a multilayer perceptron. Decoding and reconstruction are performed, i.e., encoding is performed based on the environment. Based on the implicit current physical environment characteristics and robot motion trends, the system predicts the ontology observation data for the next time step and outputs the predicted ontology observation data for the next time step. .

[0106] Through the steps S210~S250 above, by designing the variational autoencoder into a structure including an encoder, a linear processor, and a decoder, and having the linear processor output explicit physical feature information and implicit distributed parameters respectively, the problems of insufficient environmental feature extraction capability and the separation between physical parameter perception and motion prediction in related technologies are solved. This achieves the fusion of explicit physical parameter estimation and implicit context feature expression of the current physical environment, so that the generated environment code can simultaneously provide interpretable physical parameter information and rich potential environmental context information for the policy network, thereby improving the accuracy and robustness of subsequent adaptive control.

[0107] In some embodiments, the method further includes, before concatenating the explicit estimation information with the implicit coding:

[0108] Based on the current training progress, the explicit estimation information is replaced with the true value of the physical parameters corresponding to the explicit estimation information using an adaptive probability; wherein, as the number of training iterations increases, the adaptive probability gradually decreases from 1 to 0.

[0109] The true values ​​of physical parameters refer to the accurate physical quantity values ​​in the external observation data that correspond to the explicit estimation information of the variational autoencoder. Specifically, these include the true values ​​of fuselage velocity (corresponding to the absolute linear velocity of the fuselage in the external observation data), center of gravity offset (corresponding to the true value of the center of mass position in the external observation data), and fuselage mass (corresponding to the true value of the fuselage mass in the external observation data). Adaptive probabilities are used to control the selection ratio between the explicit estimation information and its corresponding true values ​​of physical parameters in the input of the policy network.

[0110] Understandably, in the early stages of training, because the parameters of the variational autoencoder have not yet converged, the explicit estimation information output by it may contain significant errors. Directly inputting this information into the policy network could lead to ineffective learning or even training divergence. Therefore, using the true physical parameters with a high probability in the early stages of training ensures that the policy network receives accurate environmental feature information, thereby quickly establishing basic motion control and environmental adaptation capabilities. As training progresses, the estimation accuracy of the variational autoencoder gradually improves, the adaptation probability gradually decreases, and the proportion of explicit estimation information in the policy network input gradually increases, eventually transitioning entirely to using the estimates from the variational autoencoder to adapt to the actual situation where the true physical parameters cannot be obtained during real-device deployment. This is precisely the purpose of the technical solution in this embodiment.

[0111] Specifically, let the current training iteration number be... The maximum number of adaptive rounds is Then adaptive probability It can be represented as:

[0112] .

[0113] At each training time step, with probability Choose to replace the explicit estimates with the corresponding true values ​​of the physical parameters, in probability. Preserve explicit estimation information from the variational autoencoder output. When hour, The policy network uses the true values ​​of physical parameters completely; when hour, The policy network fully utilizes the explicit estimation information output by the variational autoencoder. This learning mechanism constructs a smooth transition path from environment-dependent ground truth to network-dependent estimates, ensuring both stability and efficiency in the early stages of training, and guaranteeing that the trained model can be directly deployed and run on real robots without additional parameter identification steps.

[0114] Through the above steps, by designing the explicit estimation information in sections and introducing an adaptive course learning strategy, the problems of inaccurate estimation in the early stage of variational autoencoder training leading to unstable policy network training and the inability to obtain the true value of physical parameters during model deployment are solved. A smooth transition from relying on the true value to relying on the estimated value is achieved, ensuring the unity of training stability and deployment feasibility.

[0115] In some embodiments, the parameters of the variational autoencoder are updated based on historical ontology observation sequences, predicted ontology observation data, and actual observation data, including:

[0116] Calculate the explicit estimation loss based on the explicit estimation information and the true values ​​of the physical parameters corresponding to the explicit estimation information;

[0117] Calculate the reconstruction loss based on the predicted ontology observation data and the actual ontology observation data from the actual observation data;

[0118] Calculate the divergence loss based on the divergence between the implicit distribution parameters and the preset prior distribution;

[0119] Based on the explicit estimation loss, reconstruction loss, and divergence loss, the total loss function of the variational autoencoder is constructed;

[0120] The network parameters of the variational autoencoder are updated based on the total loss function.

[0121] Among them, the explicit estimation loss is used to measure the explicit estimation information of the variational autoencoder output. , , The difference between the predicted ontology observations and the true values ​​of the corresponding physical parameters provided by the simulation environment is used to guide the variational autoencoder to learn to accurately estimate the physical parameters related to the current environment. The reconstruction loss measures the difference between the predicted ontology observations output by the decoder and the actual ontology observations at the next time step. It guides the variational autoencoder to compress and reconstruct key information in the observation sequence by learning forward and inverse dynamics. The divergence loss constrains the learned latent variable distribution from deviating excessively from the preset prior distribution (standard normal distribution). It applies regularization to the latent variable space, ensuring its continuity and smoothness, so that the implicit codes sampled from this space can generate reasonable prediction results.

[0122] Specifically, the variational autoencoder can be trained based on the total loss function, and the network parameters of the encoder, linear processor, and decoder of the variational autoencoder can be updated simultaneously through gradient backpropagation.

[0123] The total loss function can be obtained through the following steps:

[0124] Step S310: Calculate the explicit estimated loss. .

[0125] Specifically, explicitly estimate the loss. It can be determined based on the velocity estimation error, mass estimation error, and center of gravity offset estimation error, and can be expressed as:

[0126] ;

[0127] in , , These represent the true values ​​of the physical parameters corresponding to the explicit estimation information in the external observation data, namely the true values ​​of the fuselage absolute velocity, fuselage mass, and center of gravity offset.

[0128] Step S320: Calculate reconstruction loss .

[0129] Specifically, reconstruction losses The predicted ontological observation data can be obtained through the next time step. With the actual ontological observation data at the next time step The error between them is used to determine this, and can be expressed as:

[0130] .

[0131] Step S330, calculate divergence loss .

[0132] The divergence loss can be calculated based on the divergence between the implicit distribution parameters and the preset prior distribution, including:

[0133] For each dimension of the implicit distribution parameters, the posterior Gaussian distribution corresponding to that dimension is defined by the mean and log-variance of that dimension.

[0134] Calculate the KL divergence between the posterior Gaussian distribution and the preset prior distribution corresponding to this dimension to obtain the KL divergence value corresponding to this dimension;

[0135] The divergence loss is calculated based on the KL divergence value corresponding to each dimension and the preset free position threshold.

[0136] The presupposed prior distribution is the standard normal distribution. The divergence loss It is used to measure the difference between the latent variable distribution and the standard normal distribution.

[0137] Specifically, divergence loss The calculation includes the following sub-steps:

[0138] Step S331: Calculate the posterior Gaussian distribution of each dimension of the implicit distribution parameters.

[0139] In detail, let the dimension of the implicit distribution parameter be... For the first Each dimension has a mean that can be expressed as: Its logarithmic variance can be expressed as Then, the posterior Gaussian distribution corresponding to this dimension can be expressed as: .

[0140] Step S332: Based on the above posterior Gaussian distribution and the preset prior distribution, calculate the KL divergence value corresponding to each dimension of the implicit distribution parameter.

[0141] In detail, according to and standard normal distribution The KL divergence value corresponding to each dimension of the implicit distribution parameter can be expressed as:

[0142] .

[0143] Step S333: Calculate the divergence loss based on the KL divergence values ​​of each dimension and the preset free position threshold. .

[0144] In detail, the preset free position threshold is denoted as Then the divergence loss It can be represented as:

[0145] ;

[0146] in, It can be set according to actual needs. Through adjustment... The value of can control the degree of constraint on the lower limit of the information capacity of the latent space.

[0147] It should be further noted that in the training of standard variational autoencoders, if the sole objective is to minimize the KL divergence, the network may tend to make the posterior distribution of all dimensions completely approximate the prior distribution (i.e., the standard normal distribution), causing the latent variables to degenerate into informationless random noise in each dimension, completely losing the ability to encode the environmental context. This is the "posterior collapse" problem commonly seen in variational autoencoder training. To avoid this problem, this application introduces a "free bit" mechanism, which sets a minimum information tolerance. This mechanism ensures that each dimension only incurs divergence loss when its KL divergence value exceeds a certain threshold; otherwise, the loss is zero. Consequently, each dimension is allowed to retain a minimum number of encoding degrees of freedom without sacrificing its own information to conform to prior constraints. This allows each dimension of the latent variable to carry effective and independent environmental feature information, preventing the entire latent space from collapsing due to over-regularization. The implicit encoding generated through this mechanism can more fully capture contextual information such as implicit terrain features and physical dynamic differences, providing richer and more robust latent feature representations for the adaptive decision-making of the policy network.

[0148] Step S340: Calculate the total loss function of the variational autoencoder.

[0149] Specifically, based on the above explicit loss estimation Reconstruction losses and divergence loss The total loss of constructing a variational autoencoder , can be represented as:

[0150] ;

[0151] Among them, the weights of the divergence loss It can be customized according to actual needs. Assign a value.

[0152] Optionally, the weights of this divergence loss An annealing strategy can be used for dynamic adjustment. In the early stages of training, the weights can be adjusted... The preset minimum value is set to 0. In this case, the total loss is mainly dominated by explicit estimation loss and reconstruction loss. The variational autoencoder focuses on learning accurate estimation of physical parameters and precise reconstruction of observed data, avoiding the limitation of the network's learning capacity due to premature intervention of distribution constraints. As the number of training iterations increases, the weights... Gradually increase from the preset minimum value to the preset maximum value. As the divergence loss gradually increases in proportion to the total loss, the network, while continuing to learn reconstruction and estimation, is gradually guided to learn to compress and normalize the distribution of latent variables. Subsequently, the weights... Maintain at the preset maximum value The annealing process, by keeping the latent space structure unchanged under sufficiently strong distribution regularization constraints, aims to make it more regular, which is beneficial for generalization and subsequent implicit encoding generation. This annealing strategy is designed to balance reconstruction accuracy and latent space regularization, avoiding network capacity limitations in the early stages of training while ensuring a regular latent space distribution structure in the later stages. It is understandable that the aforementioned preset maximum value... The value can be set based on the relative magnitudes of the explicit estimation loss, reconstruction loss, and divergence loss. In practical implementation, The specific values ​​can be adjusted according to actual needs.

[0153] Specifically, let the current training iteration number be... The maximum number of annealing iterations is , The maximum value is ,but The calculation formula can be expressed as:

[0154] ;

[0155] Among them, hour, ;along with Gradually increase to , linearly increasing from 0 to , and thereafter maintain constant.

[0156] Through the above steps, by dynamically adjusting the divergence loss weight using the KL divergence annealing strategy, the problem of balancing the reconstruction target and the distribution constraint target during the training of variational autoencoders in related technologies is solved. Based on rapid convergence and accurate estimation of physical parameters in the early stage of training, latent space regularization is gradually introduced, and finally a variational autoencoder model with both good reconstruction capability and regular latent space structure is obtained.

[0157] Through the above steps S310~S340, by designing the total loss function of the variational autoencoder as a combination of explicit estimation loss, reconstruction loss and divergence loss, the problem of focusing only on reconstruction quality and ignoring the accuracy of physical parameter estimation during training of variational autoencoders in related technologies is solved. This achieves joint optimization of physical parameter estimation, observation reconstruction and latent space regularization, enabling the variational autoencoder to output accurate physical parameter estimates and effective implicit representations of the environment at the same time.

[0158] In some embodiments, the network parameters of the encoder are updated via backpropagation using a joint loss consisting of explicit estimation loss, reconstruction loss, and divergence loss; the network parameters of the decoder are updated via backpropagation using the gradient of the reconstruction loss.

[0159] The joint loss refers to the weighted sum of the explicit estimation loss, reconstruction loss, and divergence loss. The encoder, as the front-end processing part of the variational autoencoder, outputs (feature vectors) that, after passing through a linear processor, simultaneously serve explicit estimation, implicit encoding generation, and subsequent decoding and reconstruction. Therefore, the encoder's parameter updates require backpropagation gradients from the losses of the three parts mentioned above. The decoder is only responsible for reconstructing the predicted ontology observation data from the environment encoding; its parameter updates mainly depend on the gradient signal of the reconstruction loss.

[0160] Specifically, in a single training iteration, the total loss function of the variational autoencoder is first calculated. Then, the gradient of the total loss with respect to the encoder and decoder parameters is calculated using an automatic differentiation mechanism, and the parameters are updated using an optimizer (such as the Adam optimizer). Since the explicit estimation information is mapped from the feature vector output by the linear processor, the gradient of the explicit estimation loss is backpropagated to the encoder through the linear processor, enabling the encoder to learn to extract features that are helpful for physical parameter estimation. Similarly, the gradient of the reconstruction loss is backpropagated to the encoder through the decoder and the linear processor. The gradient of the divergence loss is also backpropagated to the encoder through the linear processor. This multi-path gradient backpropagation mechanism ensures that the encoder can simultaneously optimize in directions that are beneficial to physical parameter estimation, observation reconstruction, and latent space regularization.

[0161] Through the above steps, by designing the encoder's parameter update to receive the multipath gradient of the joint loss, while the decoder's parameter update focuses on reconstructing the loss gradient, the problem of fragmented training signals among the functional modules of variational autoencoders in related technologies is solved. This enables the encoder to learn three types of objectives—physical parameter estimation, data reconstruction, and latent space regularization—in a unified manner, ensuring that the extracted features can simultaneously serve multiple functional requirements.

[0162] In some embodiments, the style reward value at the current time step is determined by an adversarial motion prior discriminator based on current observation data and actual observation data, including:

[0163] By concatenating the current observation data with the actual observation data, the policy state transition pair at the current time step is obtained;

[0164] The strategy state transition pair is input into the adversarial motion prior discriminator, which outputs a realism score. This realism score is used to characterize the degree of similarity between the strategy state transition pair and the preset reference motion trajectory.

[0165] Based on the authenticity score, the style reward value is calculated using a preset mapping function.

[0166] Here, the current time step is used. The preset reference motion trajectory refers to a pre-collected reference motion trajectory, constructed from expert controller or real animal motion data, possessing a natural and coordinated style. The preset mapping function is used to convert the authenticity score output by the discriminator into a scalar value that can be used for reinforcement learning rewards. The higher the authenticity score (the closer it is to the expert style), the greater the corresponding style reward value.

[0167] It should be noted that Adversarial Motion Priors (AMP) is a method that uses adversarial learning to learn motion styles or prior knowledge from reference motion data. In related technologies, strategies trained by pure reinforcement learning may produce unnatural movements such as high-frequency tremors and stiff gait in order to maximize task rewards. Although these movements may complete the task in simulation, they will fail on real robots due to complex factors such as friction and inertia, or lead to low energy efficiency and accelerated wear of mechanical structures. The embodiments of this application introduce an adversarial motion prior discriminator, using real animal or robot motion data as expert samples, and use adversarial training to guide the policy network to generate natural, smooth, and coordinated movements. It is understood that the above-mentioned adversarial motion prior discriminator can be implemented by neural network structures such as multilayer perceptrons. This is a conventional implementation method that can be chosen by those skilled in the art based on the disclosure of this application, and is not specifically limited here, nor does it constitute a limitation on the scope of protection of this application.

[0168] Specifically, at each time step, the current observation data at the current time step is concatenated with the actual observation data at the next time step to construct a policy state transition pair. This policy state transition pair contains complete kinematic and dynamic state changes of the robot between two consecutive time steps. This policy state transition pair is input into an adversarial motion prior discriminator, which outputs a real-valued score. The score The closer the value is to 1, the closer the input policy state transition pair is to the expert's movement style; the closer it is to -1, the closer it is to the style generated by the policy. Then, the score is converted into a style reward value using a preset mapping function. For example, a quadratic reward mapping function can be used to implement the above mapping, which can be expressed as:

[0169] .

[0170] Among them, when the adversarial motion prior discriminator outputs The closer it is to 1, the closer its corresponding style reward value is to 1; when When the value deviates from 1, the corresponding style reward value gradually decreases to 0.

[0171] Compared to the linear mapping or mapping function that directly uses the discriminator's raw output as the reward signal used in traditional schemes, the above-mentioned quadratic reward mapping function is more effective when the adversarial motion prior discriminator output is close to the expert style. Approaching 1) a relatively flat gradient provides a stable reward signal to the policy network, preventing drastic changes in style reward due to small fluctuations in the output of the adversarial motion prior discriminator. Simultaneously, when the output of the adversarial motion prior discriminator deviates from the expert style, the aforementioned quadratic reward mapping function provides a gradually increasing gradient signal, effectively guiding the policy network to optimize towards generating motion styles closer to the expert's. This reward design encourages the policy network to generate policy state transition pairs that can "deceive" the discriminator, causing the discriminator to misclassify the policy-generated data as expert data, thereby guiding the policy network to learn and generate motions that are stylistically indistinguishable from expert data.

[0172] Through the above steps, by introducing an adversarial motion prior discriminator into the reinforcement learning framework to evaluate the style of the motion generated by the policy, and feeding the evaluation results back to the policy network in the form of reward signals, the problems of unnatural motion and high-frequency tremors under pure task reward-driven methods in related technologies are solved. This enables the policy network to generate a smooth, coordinated, and natural motion style that is close to the expert trajectory while completing the task objective, which significantly improves the motion quality and mechanical structure safety of the robot.

[0173] Furthermore, the training process of the aforementioned adversarial motion prior discriminator includes:

[0174] Expert state transition pairs are constructed based on a preset reference motion trajectory; wherein, the expert state transition pairs are constructed from continuous state data in the preset reference motion trajectory;

[0175] The policy state transition generated by the policy network is input to the adversarial motion prior discriminator to obtain the first authenticity score.

[0176] The expert state is transferred to the input adversarial motion prior discriminator to obtain the second authenticity score;

[0177] The network parameters of the adversarial motion prior discriminator are updated based on the first and second authenticity scores.

[0178] Among them, the preset reference motion trajectory refers to high-quality reference motion data that serves as expert samples for adversarial motion prior discriminators, and is used to provide a natural and coordinated motion benchmark for adversarial training with style rewards.

[0179] Specifically, when training the adversarial motion prior discriminator, state transition pairs can be sampled from two data sources, namely, expert state transition pairs can be sampled from the expert dataset. Policy state transition pairs are obtained by sampling from the replay buffer generated by the interaction between the policy network and the simulation environment. .Will Input the discriminator to obtain the first authenticity score. The first authenticity score is used to characterize the degree of deviation between the motion state transitions generated by the strategy and the expert's motion style. Input the discriminator to obtain the second authenticity score. The second authenticity score is used to characterize the degree of closeness between the expert's motion state transition and the discriminator's ideal output. Subsequently, the loss function of the adversarial motion prior discriminator can be determined based on the first and second authenticity scores, and can be expressed as:

[0180] ;

[0181] The first term in the formula Used to cause the discriminator's output for the expert state transition pair to approach 1; the second term Used to cause the discriminator's output for the policy state transition pair to approach -1; the third term For gradient penalty term, where The gradient penalty coefficient is used to constrain the gradient norm of the adversarial motion prior discriminator on expert samples, preventing it from becoming overly sensitive near expert samples and thus improving the stability of adversarial training. The physical meaning of this loss function is to ensure that the adversarial motion prior discriminator accurately distinguishes between expert data and policy data, while the policy network aims to generate policy state transition pairs that make the discriminator output as close to 1 as possible, forming an adversarial game relationship. During training, the discriminator and policy network are updated alternately, gradually improving the discriminator's discrimination ability and the policy network's motion generation quality.

[0182] Furthermore, to improve training efficiency and stability, separate buffers can be maintained for expert samples and policy samples. The expert buffer pre-samples and stores a fixed number of expert state transition pairs randomly from the expert database, while the policy buffer is continuously updated as training progresses, retaining policy state transition pairs generated by the policy network during the most recent training phase. Each time the discriminator is updated, the same number of samples are randomly sampled from both buffers, the loss is calculated, and the parameters are updated.

[0183] Optionally, a model predictive control (MPC) method can be used to generate motion data of the robot in various gaits through real-time optimization calculations, obtain the above-mentioned preset reference motion trajectory, and use it as expert data.

[0184] In detail, obtaining this expert data may include the following steps:

[0185] First, the command space is defined, and the commands are sampled at equal intervals. The commands include the robot's linear velocity in the x-direction and y-direction in the body coordinate system, as well as its yaw angular velocity around the yaw direction. Within a pre-defined range of velocity dimensions, equal-interval sampling is performed on each velocity dimension to obtain a combination of velocity commands covering multiple motion modes.

[0186] Secondly, for each sampled velocity command, the robot operates under that command for a preset duration. Model predictive control methods are then used to generate, through rolling time-domain optimization, a joint control sequence and corresponding robot motion trajectory that satisfy the command in real time. This preset duration can be set as needed, for example, 4 seconds, to obtain time trajectory data containing the complete motion cycle.

[0187] During continuous operation, the robot's full state information is collected and recorded synchronously. The robot state data saved in each complete time trajectory includes: body position, body posture, joint angles, joint angular velocities, and foot position, etc.

[0188] Finally, all collected data can be stored in a compressed format (such as npz format) to reduce the dataset size and improve reading efficiency.

[0189] The expert motion data generated by the model predictive control method not only meets the motion quality requirements under optimization constraints but also possesses a physically meaningful natural motion style, providing a high-quality reference standard for the adversarial motion prior discriminator. When training the adversarial motion prior discriminator, state data from two consecutive time points can be extracted from this dataset to construct expert state transition pairs, serving as a benchmark for judging the true motion style. Guided by this benchmark, the policy network continuously optimizes based on style reward values, ultimately generating natural motion that approximates this reference standard.

[0190] Through the above steps, authenticity scores are obtained based on expert state transition pairs and policy state transition pairs respectively. The discriminator loss function is constructed by adding gradient penalty to mean squared error. This solves the problems of instability in adversarial training process and difficulty in guaranteeing the reliability and training efficiency of discriminator gradient signal in related technologies. It realizes stable training of adversarial motion prior discriminator, enabling it to provide accurate and effective motion style guidance signals for policy network.

[0191] This application also provides a system architecture embodiment to illustrate a specific implementation architecture and complete data flow of the above-described motion control model training method. Figure 3 This is a schematic diagram illustrating the global network structure and data flow direction of an exemplary embodiment of this application, as shown below. Figure 3As shown, the core components involved in training in this global network structure include: a simulation interaction environment, a robot, a variational autoencoder, a policy network, a commentator network, an adversarial motion prior discriminator, and an experience buffer for storing experience data during training. Figure 3 (Not shown in the image), used to provide batches of empirical data for the PPO training phase. The components work collaboratively according to a defined data flow and timing sequence.

[0192] The simulation interactive environment is used to simulate the physical environment in which the robot operates. At the beginning of each training round, environmental physical parameters (such as body mass, joint friction, ground friction coefficient / terrain course, center of mass position / environmental gravity, external random disturbances, etc.) are randomized to simulate various physical condition changes that may occur in the real world. The robot (i.e., the robot model) in the simulation interactive environment collects its own state information through sensors, forming ontological observation data, including joint angles, joint angular velocities, body posture, body angular velocity, and gravity direction. Simultaneously, the simulation interactive environment also provides external observation data and privileged information. External observation data includes the absolute linear velocity of the body, ground friction coefficient, true value of body mass, and true value of center of mass position; privileged information may include privileged observation information formed from privileged observations (such as external disturbance force information) and elevation map observation information formed from elevation map observations.

[0193] The policy network and the critic network constitute the Actor-Critic architecture. The policy network (Actor) is responsible for outputting action commands based on the current ontology observation data and environment encoding to control the robot's movement; the critic network (Critic) is responsible for evaluating the value of the current state, outputting the state value, and providing gradient signals for updating the policy network.

[0194] The adversarial motion prior discriminator is used to perform style evaluation on the motion state transition (i.e. policy state) pairs generated by the policy network. By comparing them with expert state transitions obtained from the expert action library, it outputs a authenticity score, thereby quantifying the similarity between the current motion and the preset reference motion trajectory, and mapping the authenticity score into a style reward value to feed back to the policy network.

[0195] The experience buffer is used to store the complete interaction data generated at each time step, which can then be used for batch updates of subsequent network components.

[0196] Specifically, such as Figure 3 As shown, the global data flow and execution sequence in this embodiment are as follows:

[0197] First, at the start of each training round, the simulation interactive environment randomizes the environmental physical parameters. The purpose of environmental randomization is to expose the policy to diverse environmental conditions during training, thereby learning a control policy with generalization ability and robustness. Parameters for environmental randomization may include: random scaling of fuselage mass within a certain range, random perturbation of the center of mass position within a certain offset range, random sampling of the ground friction coefficient, and random perturbation of joint friction torque and damping, etc.

[0198] Time step in each control cycle The robot collects current body observation data through sensors. and maintain a length of Historical ontological observation sequence This sequence contains consecutive sequences preceding the current time. Ontology observation data at each time step.

[0199] like Figure 3 As shown, the data first flows to the variational autoencoder module. The variational autoencoder receives the historical ontology observation sequence. The encoder extracts feature vectors, and then a linear processor outputs explicit estimation information and implicit distribution parameters. Implicit encoding is obtained through reparameterized sampling (training phase) or by directly taking the mean (deployment phase). Finally, the explicit estimation information and implicit encoding are concatenated to generate the environment encoding. Simultaneously, the decoder of the variational autoencoder predicts the predicted ontology observation data for the next time step based on the environment encoding. .

[0200] Subsequently, the data flows to the policy network. The policy network receives the current ontology observation data. and the environment encoding of the variational autoencoder output Output continuous control action commands The action command is sent to the robot's actuator model, driving the robot to move. Simultaneously, the critic network receives current ontological observation data, external observation data, and privileged information to evaluate the state value of the current state. .

[0201] Robot executes action instructions Then, the simulation interactive environment returns the actual observation data for the next time step (including the volumetric observation data for the next time step). (and the actual external observation data at the next time step) and the termination marker Simultaneously, the environment calculates and returns the basic task reward based on the task objective. Based on this, the system uses the current observation data at the current time step and the actual observation data at the next time step as the states, respectively. and Construct strategy state transition pairs The input is fed into the adversarial motion prior discriminator, which outputs a truth score. Then, it is converted into a style reward value through a preset mapping function. The total reward value is obtained by weighted summing of the task reward value and the style reward value: In addition, the policy state transition will also be applied to... Store the policy sample in the cache for subsequent discriminator updates.

[0202] At this point, a complete set of experience data generated at the current time step is uniformly encapsulated and written into the experience buffer. This complete experience data includes historical ontology observation sequences, environment codes, predicted ontology observation data, current observation data (current ontology observation data and external observation data), action instructions, comprehensive reward values ​​(task reward values ​​and style reward values), actual observation data (ontology observation data and external observation data at the next time step), termination flags, and state values.

[0203] Once a sufficient amount of empirical data has accumulated in the experience buffer, the system enters the model update phase. A batch of data is read from the experience buffer to jointly update the parameters of the variational autoencoder and the policy evaluation model. The variational autoencoder update is based on the joint loss consisting of explicit estimation loss, reconstruction loss, and divergence loss; the policy network and critic network update are based on the PPO algorithm, using the combined reward value and state value to calculate the advantage function, and constructing the policy loss and value loss for backpropagation. Simultaneously, expert action transition pairs and policy state transition pairs are sampled from the expert database storing preset expert motion trajectories and the policy sample cache, respectively, to update the parameters of the adversarial motion prior discriminator.

[0204] The above process is repeated iteratively in multiple training rounds until the model converges, and finally a trained motion control model is obtained.

[0205] Through the above system architecture embodiments, by constructing a system architecture that includes a simulation interactive environment, a variational autoencoder, a policy network, a commentator network, and an adversarial motion prior discriminator, the complete data flow and execution sequence from environmental observation and acquisition, construction of historical ontology observation sequences, generation of environmental encoding by the variational autoencoder, output of policy action instructions, environmental interaction feedback, calculation of comprehensive reward value, storage of experience data, and joint updates of various components are clarified. This enables each step in the above method embodiments to be efficiently implemented in a specific system architecture, and the environmental perception accuracy, task execution capability, and motion style naturalness promote each other and improve synchronously in a training process.

[0206] The following uses a quadruped robot as an example, combined with typical application scenarios of the technical solution in actual deployment, to further illustrate the technical effects of the technical solution of this application.

[0207] Scenario 1: Transporting unknown loads.

[0208] When a quadruped robot performs a transportation task carrying goods of unknown weight, its actual dynamic characteristics change significantly. At this time, the variational autoencoder module can continuously collect historical on-body observation sequences from the robot's own sensors to perceive implicit information such as the magnitude of body descent and the trend of joint torque changes. It can then quickly infer the physical parameter characteristics corresponding to the current load change and generate a corresponding environmental code online. Based on this environmental code, the policy network adaptively adjusts the gait amplitude, support phase duration, and joint driving force to maintain stable walking without the need for pre-weighing or retraining.

[0209] Scene 2: Traversing varied terrain.

[0210] When a quadruped robot moves from a high-friction surface (such as concrete) to a low-friction surface (such as sand), the contact dynamics between its feet and the ground change abruptly. At this point, the variational autoencoder module can capture subtle changes such as foot slippage and fluctuations in the robot's angular velocity from recent observation sequences and output an environmental code characterizing the current ground friction properties. The policy network dynamically adjusts its landing strategy and foot trajectory based on this code to suppress slippage and maintain stability. Simultaneously, guided by the style reward value output by the adversarial motion prior discriminator, the adaptively adjusted gait maintains a smooth, coordinated, and natural movement style close to the preset reference trajectory, avoiding stiff and uncoordinated movements resulting from environmental adaptation.

[0211] The above application scenarios demonstrate that this application combines the online implicit perception capability of the environmental dynamics of the variational autoencoder with the constraint of the naturalness of the motion style by the adversarial motion prior discriminator, enabling the quadruped robot to autonomously generate robust control behavior that combines task completion capability and natural motion style when facing unknown load changes and varied terrain conditions, without relying on explicit system identification and external sensors.

[0212] It should be noted that the module division described in this embodiment is for the purpose of clearly illustrating the data flow and functional division during the training process, and does not constitute a unique limitation on the corresponding device modules. Furthermore, the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0213] This embodiment also provides a motion control model training device for implementing the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0214] This application also provides a motion control model training device, wherein the motion control model includes a variational autoencoder, an adversarial motion prior discriminator, and a policy evaluation model; the policy evaluation model includes a policy network and a critic network. Figure 4 The structural block diagram of the motion control model training device provided in this application is as follows: Figure 4 As shown, the device includes: an acquisition module 410, a processing module 420, and a training module 430;

[0215] The acquisition module 410 is used to acquire the robot's historical ontology observation sequence at each time step and input it into the variational autoencoder. The variational autoencoder generates an environment code and predicts the predicted ontology observation data for the next time step based on the environment code.

[0216] The processing module 420 is used to input the ontology observation data and environment encoding at the current time step into the policy network, and the policy network outputs action instructions and controls the robot to execute the action instructions.

[0217] The processing module 420 is also used to input the robot's current observation data and privileged information into the critic network, which evaluates the state value.

[0218] The acquisition module 410 is also used to collect the actual observation data and termination flag at the next time step after the robot executes the action command, and to determine the task reward value at the current time step;

[0219] The processing module 420 is also used to determine the style reward value at the current time step by the adversarial motion prior discriminator based on the current observation data and the actual observation data, and to determine the comprehensive reward value at the current time step based on the style reward value and the task reward value; the style reward value is used to guide the policy network to generate action instructions that approach the preset reference motion trajectory;

[0220] The processing module 420 is also used to use historical ontology observation sequences, environment codes, predicted ontology observation data, current observation data, action instructions, comprehensive reward values, actual observation data, termination flags, and state values ​​as empirical data at the current time step;

[0221] Training module 430 is used to train the motion control model based on empirical data at multiple time steps to obtain a trained motion control model.

[0222] This application also provides a robot motion control method. Figure 5 The flowchart of the robot motion control method provided in this application is as follows: Figure 5 As shown, the method includes the following steps:

[0223] Step S510: Obtain the robot's historical ontology observation sequence.

[0224] Among them, the historical ontology observation sequence consists of consecutive observations before the current time step. A time series consisting of ontology observation data at each time step. Ontology observation data refers to data directly measured by the robot's own sensors, including but not limited to joint angles, joint angular velocities, body posture, body angular velocity, and gravity direction. During the model deployment phase, the robot can only acquire this type of ontology observation data and cannot access external observation data or privileged information available during simulation training. The length of the historical ontology observation sequence... Maintain consistency with the training phase to ensure the consistency of the variational autoencoder input data distribution.

[0225] Specifically, during actual deployment and operation, the robot uses its onboard joint encoders, inertial measurement units, and other sensors to collect body observation data at each time step in real time, and then... The ontological observation data at each time step are organized in chronological order to form a historical ontological observation sequence.

[0226] Through the above steps, by collecting historical ontological observation sequences from the robot's own sensors, the availability of input data during the deployment phase and the consistency of the input data format with those during the training phase are ensured, providing a reliable data foundation for subsequent environmental feature extraction by the variational autoencoder.

[0227] Step S520: Input the historical ontology observation sequence into the variational autoencoder in the trained motion control model so that the variational autoencoder can obtain the environment code based on the historical ontology observation sequence; wherein, the motion control model is trained by any of the above-mentioned motion control model training methods.

[0228] In this deployment phase, the variational autoencoder uses the network parameters learned during the training phase, but its operation differs from that during training: when generating implicit codes, it does not perform a sampling step for random noise vectors, but directly uses the mean of the linear processor output as the implicit code. This design ensures that, under the same input conditions, the environment code output by the variational autoencoder has determinism and consistency, eliminating output fluctuations that may be introduced by random sampling. The environment code can implicitly represent the dynamic characteristics of the current physical environment, providing environmental context information for the subsequent adaptive decision-making of the policy network.

[0229] It's important to note that during the model deployment phase, only the encoder and linear processor parts of the pre-trained variational autoencoder, along with the policy network, are needed; the critic network and adversarial motion prior discriminator do not require execution. The critic network and discriminator only assist model learning during training and do not participate in inference or consume computational resources during deployment. This lightweight deployment architecture enables this method to run directly on real-world robotic embedded platforms with limited computing power.

[0230] Specifically, the historical ontology observation sequence obtained in step S510 is input into the encoder of the trained variational autoencoder, which extracts features to obtain a feature vector. The linear processor receives the feature vector and outputs explicit estimation information and implicit distribution parameters through a fully connected layer. In the deployment phase, the mean of the implicit distribution parameters is directly used as the implicit code and concatenated with the explicit estimation information to obtain the environment code, without the need to sample random noise vectors from the standard normal distribution.

[0231] Through the above steps, by using a trained variational autoencoder to extract the environment code from the historical ontology observation sequence and replacing the sampling with the mean as the implicit code output, the determinism of environment perception during the deployment phase is guaranteed. At the same time, only the encoder, linear processor and policy network are run, without relying on the critic network and discriminator, which solves the problem of requiring additional sensors or complex identification processes during deployment in related technologies, and realizes lightweight and efficient operation on real robots.

[0232] Step S530: Input the ontology observation data and environment encoding at the current time step into the policy network in the motion control model so that the policy network can output action commands.

[0233] The input to the policy network during the deployment phase consists of two parts: the ontology observation data at the current time step and the environment code generated in step S520. The ontology observation data at the current time step provides information on the robot's current motion state, while the environment code provides an implicit feature representation of the current physical environment. The combination of these two elements enables the policy network to adaptively adjust its control strategy and generate action commands adapted to the current environment under different physical environmental conditions (such as different loads and different ground friction characteristics). Since the input to the policy network relies solely on the ontology sensor data and the output of the variational autoencoder throughout the deployment process, without depending on any external observation data or privileged information, this method demonstrates good practical deployment feasibility.

[0234] Specifically, the body observation data collected by sensors at the current time step is concatenated or combined with the environment code output in step S520, serving as the input tensor for the policy network. Based on the mapping relationships learned during the training phase, the policy network infers and outputs continuous control actions, such as target joint angles or target joint torques. This inference process is a forward propagation computation, which has low computational cost and fast response speed, meeting the requirements of real-time robot control.

[0235] Through the above steps, by inputting ontological observation data and environmental coding into the policy network, the policy network can make adaptive decisions based on environmental context information. This solves the problem in related technologies that the policy cannot be finely adjusted according to different actual environmental conditions during deployment, and realizes online adaptive control of variable physical environments.

[0236] Step S540: Control the robot's movement according to the action instructions.

[0237] In this context, motion commands are continuous control quantities output by the policy network, which can directly act on the robot's underlying actuators. Depending on the robot's control architecture, motion commands can be target joint angles, target joint torques, or target joint position gains, etc. The robot's underlying controller receives these motion commands and drives each joint actuator to complete the corresponding movements, thereby achieving a complete perception-decision-control closed loop.

[0238] Specifically, the motion commands output by the policy network, after necessary scaling or transformation, are sent to the drive units of each joint actuator via the robot's communication bus. The drive units control the motors to rotate according to the received commands, causing the robot's joints to move, thereby changing the robot's pose and motion state. In the next control cycle, the robot's sensors re-acquire new body observation data, starting a new round of perception-decision-control loop.

[0239] Through the above steps, by directly applying the action commands output by the policy network to the robot actuator, a complete closed-loop control link is realized, from historical state perception, environmental feature extraction, adaptive decision-making to motion execution.

[0240] Through steps S510 to S540, the robot's historical ontology observation sequence is acquired, and environmental codes are extracted from it using a trained variational autoencoder. The environmental codes and the current ontology observation data are then input into the policy network to generate action commands, ultimately driving the robot to move. This solves the problems in related technologies where model deployment requires reliance on external sensors for system identification and the policy cannot adapt to different physical environments. It enables quadruped robots to implicitly perceive and adaptively control environmental factors such as load changes and terrain changes in the real physical world using only their own sensor data, while generating smooth, coordinated, and natural motion. This significantly improves the success rate of transferring from simulation to reality and the robustness of motion control.

[0241] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination. Specific examples in this embodiment can be found in the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0242] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 6 This is a hardware structure block diagram of the terminal for the motion control model training method provided in this application. (See diagram below.) Figure 6 As shown, a terminal may include one or more ( Figure 6 Only one is shown in the diagram. A processor 602 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 604 for storing data are also shown. Optionally, the terminal may further include a transmission device 606 for communication functions and an input / output device 608. Those skilled in the art will understand that... Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 6 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0243] The memory 604 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the motion control model training method in this embodiment. The processor 602 executes various functional applications and data processing by running the computer program stored in the memory 604, thus implementing the aforementioned method. The memory 604 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 604 may further include memory remotely located relative to the processor 602, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0244] The transmission device 606 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 606 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 606 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0245] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described motion control model training method embodiments.

[0246] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0247] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0248] S1. At each time step, the robot's historical ontology observation sequence is acquired and input into the variational autoencoder. The variational autoencoder generates an environment code and predicts the ontology observation data for the next time step based on the environment code.

[0249] S2: Input the ontology observation data and environment code at the current time step into the policy network, which outputs action instructions and controls the robot to execute the action instructions; input the robot's current observation data and privileged information into the critic network, which evaluates the state value.

[0250] S3: Collect the actual observation data and termination flag at the next time step after the robot executes the action command, and determine the task reward value at the current time step; based on the current observation data and the actual observation data, the adversarial motion prior discriminator determines the style reward value at the current time step, and determines the comprehensive reward value at the current time step based on the style reward value and the task reward value; the above style reward value is used to guide the policy network to generate action commands that approach the preset reference motion trajectory.

[0251] S4. The historical ontology observation sequence, environment encoding, predicted ontology observation data, current observation data, action instructions, comprehensive reward value, actual observation data, termination flag, and state value are used as empirical data at the current time step. The above motion control model is trained based on the empirical data at multiple time steps to obtain the trained motion control model.

[0252] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0253] Furthermore, in conjunction with the motion control model training methods described in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the motion control model training methods described in the above embodiments.

[0254] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0255] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0256] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0257] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for training a motion control model, characterized in that, The motion control model includes a variational autoencoder, an adversarial motion prior discriminator, and a policy evaluation model; the policy evaluation model includes a policy network and a critic network; the method includes: At each time step, the robot's historical ontology observation sequence is acquired and input into the variational autoencoder. The variational autoencoder generates an environment code and predicts the predicted ontology observation data for the next time step based on the environment code. The ontology observation data at the current time step and the environment code are input into the policy network, which outputs action commands and controls the robot to execute the action commands. The robot's current observation data and privileged information are input into the critic network, which then evaluates the state value. Collect the actual observation data and termination flag at the next time step after the robot executes the action command, and determine the task reward value at the current time step; Based on the current observation data and the actual observation data, the adversarial motion prior discriminator determines the style reward value at the current time step, and determines the comprehensive reward value at the current time step based on the style reward value and the task reward value; the style reward value is used to guide the policy network to generate action instructions that approximate a preset reference motion trajectory; The historical ontology observation sequence, the environment code, the predicted ontology observation data, the current observation data, the action command, the comprehensive reward value, the actual observation data, the termination flag, and the state value are used as empirical data at the current time step. The motion control model is trained based on empirical data at multiple time steps to obtain a well-trained motion control model. The variational autoencoder includes an encoder, a linear processor, and a decoder; the step of generating an environment code from the variational autoencoder and predicting the predicted ontology observation data for the next time step based on the environment code includes: The encoder is used to extract features from the historical ontology observation sequence to obtain a feature vector; The linear processor processes the feature vector to obtain explicit estimation information and implicit distribution parameters. The explicit estimation information includes fuselage speed estimates, center of gravity offset estimates, and fuselage mass estimates. The fuselage speed estimates are used by the policy network to adjust speed tracking accuracy. The center of gravity offset estimates are used by the policy network to adapt to center of gravity position shifts caused by load changes or external disturbances. The fuselage mass estimates are used by the policy network to adapt to different physical loads. The implicit distribution parameters include the mean and logarithmic variance, used to characterize the distribution characteristics of latent information related to the current physical environment in the feature vector. The linear processor samples random noise vectors from a preset standard normal distribution and obtains implicit coding based on the mean, the logarithmic variance, and the random noise vectors. The explicit estimation information and the implicit coding are concatenated by the linear processor to obtain the environment coding; The environment encoding is decoded by the decoder to obtain the predicted ontology observation data; Training the motion control model based on empirical data at multiple time steps includes: Calculate the explicit estimation loss based on the explicit estimation information and the true values ​​of the physical parameters corresponding to the explicit estimation information; The reconstruction loss is calculated based on the predicted ontology observation data and the actual ontology observation data. The divergence loss is calculated based on the divergence between the implicit distribution parameters and the preset prior distribution. Based on the explicit estimation loss, the reconstruction loss, and the divergence loss, the total loss function of the variational autoencoder is constructed; The network parameters of the variational autoencoder are updated based on the total loss function.

2. The motion control model training method according to claim 1, characterized in that, The step of training the motion control model based on empirical data at multiple time steps to obtain a trained motion control model includes: The parameters of the variational autoencoder are updated based on the historical ontology observation sequence, the predicted ontology observation data, and the actual observation data. The parameters of the strategy evaluation model are updated based on the ontology observation data at the current time step, the action command, the comprehensive reward, the termination flag, the actual observation data, and the state value.

3. The motion control model training method according to claim 2, characterized in that, Before concatenating the explicit estimation information with the implicit encoding, the method further includes: Based on the current training progress, the explicit estimation information is replaced with the true value of the physical parameters corresponding to the explicit estimation information using an adaptive probability selection; wherein, as the number of training iterations increases, the adaptive probability gradually decreases from 1 to 0.

4. The motion control model training method according to claim 3, characterized in that, The step of calculating the divergence loss based on the divergence between the implicit distribution parameters and the preset prior distribution includes: For each dimension of the implicit distribution parameters, the posterior Gaussian distribution corresponding to that dimension is defined by the mean and the log-variance of that dimension. Calculate the KL divergence between the posterior Gaussian distribution corresponding to this dimension and the preset prior distribution to obtain the KL divergence value corresponding to this dimension; The divergence loss is calculated based on the KL divergence value corresponding to each dimension and the preset free position threshold.

5. The motion control model training method according to claim 4, characterized in that, Based on the explicit estimation loss, the reconstruction loss, and the divergence loss, the total loss function of the variational autoencoder is constructed, including: The total loss function is obtained by weighting each loss according to the weight of the explicit estimation loss, the weight of the reconstruction loss, and the weight of the divergence loss; wherein, as the number of iterations increases, the weight of the divergence loss gradually increases from a preset minimum value to a preset maximum value.

6. The motion control model training method according to claim 4, characterized in that, The network parameters of the encoder are updated via backpropagation using the joint loss consisting of the explicit estimation loss, the reconstruction loss, and the divergence loss; the network parameters of the decoder are updated via backpropagation using the gradient of the reconstruction loss.

7. The motion control model training method according to claim 1, characterized in that, The step of determining the style reward value at the current time step by the adversarial motion prior discriminator based on the current observation data and the actual observation data includes: The current observation data is concatenated with the actual observation data to obtain the policy state transition pair at the current time step; The strategy state transition pair is input into an adversarial motion prior discriminator, which outputs a realism score. The realism score is used to characterize the similarity between the strategy state transition pair and a preset reference motion trajectory. The style reward value is calculated based on the authenticity score using a preset mapping function.

8. The motion control model training method according to claim 7, characterized in that, The training process of the adversarial motion prior discriminator includes: An expert state transition pair is constructed based on a preset reference motion trajectory; wherein, the expert state transition pair is constructed from continuous state data in the preset reference motion trajectory; The policy state transition pair generated by the policy network is input into the adversarial motion prior discriminator to obtain the first authenticity score; The expert state transition pair is input into the adversarial motion prior discriminator to obtain a second authenticity score; The network parameters of the adversarial motion prior discriminator are updated based on the first authenticity score and the second authenticity score.

9. A motion control model training device, characterized in that, The motion control model includes a variational autoencoder, an adversarial motion prior discriminator, and a policy evaluation model; the policy evaluation model includes a policy network and a critic network; the device includes: an acquisition module, a processing module, and a training module; wherein, The acquisition module is used to acquire the robot's historical ontology observation sequence at each time step and input it into the variational autoencoder. The variational autoencoder generates an environment code and predicts the predicted ontology observation data for the next time step based on the environment code. The processing module is used to input the ontology observation data and the environment code at the current time step into the policy network, and the policy network outputs action commands and controls the robot to execute the action commands. The processing module is also used to input the robot's current observation data and privileged information into the commentator network, so that the commentator network can evaluate the state value; The acquisition module is also used to collect the actual observation data and termination flag at the next time step after the robot executes the action command, and to determine the task reward value at the current time step; The processing module is further configured to determine the style reward value at the current time step by the adversarial motion prior discriminator based on the current observation data and the actual observation data, and to determine the comprehensive reward value at the current time step based on the style reward value and the task reward value; the style reward value is used to guide the policy network to generate action instructions that approximate a preset reference motion trajectory; The processing module is further configured to use the historical ontology observation sequence, the environment code, the predicted ontology observation data, the current observation data, the action command, the comprehensive reward value, the actual observation data, the termination flag, and the state value as empirical data at the current time step; The training module is used to train the motion control model based on empirical data at multiple time steps to obtain a trained motion control model. The variational autoencoder includes an encoder, a linear processor, and a decoder; the step of generating an environment code from the variational autoencoder and predicting the predicted ontology observation data for the next time step based on the environment code includes: The encoder is used to extract features from the historical ontology observation sequence to obtain a feature vector; The linear processor processes the feature vector to obtain explicit estimation information and implicit distribution parameters. The explicit estimation information includes fuselage speed estimates, center of gravity offset estimates, and fuselage mass estimates. The fuselage speed estimates are used by the policy network to adjust speed tracking accuracy. The center of gravity offset estimates are used by the policy network to adapt to center of gravity position shifts caused by load changes or external disturbances. The fuselage mass estimates are used by the policy network to adapt to different physical loads. The implicit distribution parameters include the mean and logarithmic variance, used to characterize the distribution characteristics of latent information related to the current physical environment in the feature vector. The linear processor samples random noise vectors from a preset standard normal distribution and obtains implicit coding based on the mean, the logarithmic variance, and the random noise vectors. The explicit estimation information and the implicit coding are concatenated by the linear processor to obtain the environment coding; The environment encoding is decoded by the decoder to obtain the predicted ontology observation data; Training the motion control model based on empirical data at multiple time steps includes: Calculate the explicit estimation loss based on the explicit estimation information and the true values ​​of the physical parameters corresponding to the explicit estimation information; The reconstruction loss is calculated based on the predicted ontology observation data and the actual ontology observation data. The divergence loss is calculated based on the divergence between the implicit distribution parameters and the preset prior distribution. Based on the explicit estimation loss, the reconstruction loss, and the divergence loss, the total loss function of the variational autoencoder is constructed; The network parameters of the variational autoencoder are updated based on the total loss function.

10. A robot motion control method, characterized in that, The robot motion control method includes: Obtain the robot's historical ontological observation sequence; The historical ontology observation sequence is input into the variational autoencoder in the trained motion control model, so that the variational autoencoder obtains the environment code based on the historical ontology observation sequence; wherein, the motion control model is trained by the motion control model training method according to any one of claims 1 to 8; The ontology observation data at the current time step and the environment encoding are input into the policy network in the motion control model so that the policy network can output action commands. The robot is controlled to move according to the action instructions.

Images