Robot motion control method and device based on world model, equipment and medium

Abstract

The robot motion control method based on a world model disclosed by the embodiments of the present application comprises: obtaining current observation information of a robot in a real machine environment, first action parameters and next state observation information; encoding the current and next state observation information into first and second reference latent variables respectively through a world model encoder; inputting the first reference latent variable and the first action parameter into a latent space dynamics model of the world model to predict a third reference latent variable; and determining a second action parameter for controlling the motion of the robot by the world model based on the state deviation between the second reference latent variable and the third reference latent variable after calibrating the initial world model. The state deviation between the real latent variable calculated based on the real machine data and the latent variable predicted by the simulation world model is used to calibrate the world model, which improves the accuracy of the action prediction model trained by the world model compared with the scheme of only applying the simulation world model.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of robot motion control technology, and in particular to a robot motion control method, device, equipment, and medium based on a world model. Background Technology

[0002] In the model-based reinforcement learning framework, the latent space world model, as an environment simulator within the reinforcement learning algorithm, can predict the latent variables, rewards, or values ​​at the next moment during the reinforcement learning update process, thereby assisting the action prediction model in policy updates and action evaluation.

[0003] Existing latent world models used in reinforcement learning are first trained in a simulation environment and then deployed to real robots. However, there are multi-dimensional differences between real robots and simulation environments, including differences in dynamics, control latency, joint execution errors, friction and contact model deviations, and camera timing differences. These differences lead to poor accuracy of the latent world models trained in simulation in real environments, which in turn affects the accuracy of the action prediction models trained by reinforcement learning. Summary of the Invention

[0004] This application provides a robot motion control method, device, equipment, and medium based on a world model, which can improve the accuracy of prediction results of the latent space world model in a real machine environment.

[0005] In a first aspect, embodiments of this application provide a robot motion control method based on a world model, including:

[0006] Acquire real interaction data, wherein the real interaction data includes the robot's current observation information, current first action parameters and next state observation information collected from the real physical environment, and any observation information includes the robot's own state information and the robot's observation image collected at the same time.

[0007] The current observation information is encoded into a first reference latent variable by the encoder of the first world model, wherein the first world model is trained in a simulation environment;

[0008] The encoder encodes the observation information of the next state into a second reference latent variable;

[0009] The first reference latent variable and the current first action parameter are input into the latent space dynamics model of the first world model to predict the third reference latent variable of the next state;

[0010] The first world model is calibrated based on the state deviation between the second reference latent variable and the third reference latent variable to obtain the second world model;

[0011] The second world model is deployed on the robot so that the robot determines the second motion parameters based on the second world model and controls the robot's movement based on the second motion parameters.

[0012] In some embodiments, calibrating the first world model based on the state deviation between the second reference latent variable and the third reference latent variable to obtain a second world model includes:

[0013] Calculate the first consistency error between the second reference latent variable and the third reference latent variable;

[0014] When the first consistency error is less than the first threshold, and the difference between the first consistency error and the simulation verification error is less than the second threshold, the first world model is updated based on the first consistency error to obtain the second world model. The simulation verification error is the error value between the latent variables of the next state calculated by the world model based on historical simulation data in the simulation environment and the latent variables of the actual next state given by the simulation environment.

[0015] In some embodiments, updating the first world model based on the consistency error to obtain the second world model includes:

[0016] The model parameters of the remaining modules in the first world model, except for the latent space dynamics model, remain fixed.

[0017] The model parameters of the latent space dynamics model in the first world model are updated based on the first consistency error to obtain the second world model.

[0018] In some embodiments, calibrating the first world model based on the state deviation between the second reference latent variable and the third reference latent variable to obtain a second world model includes:

[0019] When the first consistency error is greater than or equal to the first threshold, or the difference between the first consistency error and the simulation verification error is greater than or equal to the second threshold, a difference correction module is configured at the input or output end of the encoder. The difference between the first model parameter quantity of the difference correction module and the second model parameter quantity of the encoder is greater than a third threshold, and the first model parameter quantity is less than the second model parameter quantity. The difference correction module is used to correct the distribution difference between the robot's current real observation information and simulation observation information.

[0020] Obtain a first target latent variable, wherein the first target latent variable satisfies any of the following conditions: when the difference correction module is configured at the input end of the encoder, the first target latent variable is obtained by the encoder encoding the data after correcting the current observation information based on the difference correction module; when the difference correction module is configured at the output end of the encoder, the first target latent variable is obtained by the difference correction module correcting the latent variable encoded by the encoder based on the current observation information.

[0021] Obtain a second target latent variable, wherein the second target latent variable satisfies any of the following conditions: when the difference correction module is configured at the input end of the encoder, the second target latent variable is obtained by the encoder encoding data after correcting the observation information of the next state based on the difference correction module; when the difference correction module is configured at the output end of the encoder, the second target latent variable is obtained by the difference correction module correcting the latent variable after encoding the observation information of the next state based on the encoder.

[0022] The first target latent variable and the current first action parameter are input into the latent space dynamics model to predict the first intermediate latent variable of the next state.

[0023] Calculate the second consistency error between the first intermediate latent variable and the second target latent variable;

[0024] A first latent space anchoring constraint term is constructed based on the first target latent variable and the first reference latent variable. A first target loss value is calculated based on the second consistency error and the first latent space anchoring constraint term. The first latent space anchoring constraint term is used to constrain the difference between the first target latent variable and the first reference latent variable.

[0025] The model parameters of the difference correction module, the encoder, and the latent space dynamics model in the first world model are updated based on the first target loss value to obtain the second world model.

[0026] In some embodiments, the encoder includes a plurality of sequentially connected coding layers, the plurality of coding layers including a first coding layer to an nth coding layer, where n is a positive integer greater than 1. The first world model is calibrated based on the state deviation between the second reference latent variable and the third reference latent variable to obtain a second world model, including:

[0027] When the first consistency error is greater than or equal to the first threshold, or the difference between the first consistency error and the simulation verification error is greater than or equal to the second threshold, the parameters of the m-th coding layer to the n-th coding layer in the encoder are updated using the target learning rate to obtain a new encoder, where m is a positive integer greater than 1 and less than n;

[0028] The new encoder encodes the current observation information into a third target latent variable.

[0029] The new encoder encodes the observation information of the next state into a fourth target latent variable;

[0030] The third target latent variable and the current first action parameter are input into the latent space dynamics model to predict the second intermediate latent variable of the next state.

[0031] Calculate the third consistency error between the second intermediate latent variable and the fourth target latent variable;

[0032] A second latent space anchoring constraint term is constructed based on the third target latent variable and the first reference latent variable. A second target loss value is calculated based on the third consistency error and the second latent space anchoring constraint term. The second latent space anchoring constraint term is used to constrain the difference between the third target latent variable and the first reference latent variable.

[0033] The model parameters of the new encoder and the model parameters of the latent space dynamics model in the first world model are updated based on the second target loss value to obtain the second world model.

[0034] In some embodiments, a first latent space anchoring constraint term is constructed based on the first target latent variable and the first reference latent variable, and a first target loss value is calculated based on the second consistency error and the first latent space anchoring constraint term, including:

[0035] Calculate the first Euclidean distance between the first target latent variable and the first reference latent variable, and determine the square of the first Euclidean distance as the first latent space anchoring constraint term;

[0036] The first target loss value is obtained by weighted summing of the second consistency error and the first latent space anchoring constraint term.

[0037] In some embodiments, a second latent space anchoring constraint term is constructed based on the third target latent variable and the first reference latent variable, and a second target loss value is calculated based on the third consistency error and the second latent space anchoring constraint term, including:

[0038] Calculate the second Euclidean distance between the third target latent variable and the first reference latent variable, and determine the square of the second Euclidean distance as the second latent space anchoring constraint term;

[0039] The second target loss value is obtained by weighted summing of the third consistency error and the second latent space anchoring constraint term.

[0040] Secondly, embodiments of this application provide a control device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the world model-based robot motion control method as described in the first aspect.

[0041] Thirdly, embodiments of this application also provide an electronic device, including the control device of the second aspect.

[0042] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for performing the world-model-based robot motion control method as described in the first aspect.

[0043] This application provides a robot motion control method, device, equipment, and medium based on a world model. The method includes: acquiring real interaction data, wherein the real interaction data includes current observation information of the robot, current first action parameters, and observation information of the next state collected from a real physical environment, and any observation information includes state information of the robot body and an observation image of the robot collected at the same time; encoding the current observation information into a first reference latent variable through an encoder of a first world model, wherein the first world model is trained in a simulation environment; encoding the observation information of the next state into a second reference latent variable through the encoder; inputting the first reference latent variable and the current first action parameters into the latent space dynamics model of the first world model to predict a third reference latent variable for the next state; calibrating the first world model based on the state deviation between the second reference latent variable and the third reference latent variable to obtain a second world model; and deploying the second world model on the robot so that the robot determines a second action parameter based on the second world model and controls the robot's motion based on the second action parameter. According to the solution provided in the embodiments of this application, based on the robot's real machine state data, the state deviation between the robot's current real latent variables and the latent variables predicted by the world model obtained from simulation training is calculated. The world model is calibrated based on the state deviation. Compared with the existing application solution that is only based on the world model trained by simulation, the accuracy of the prediction results of the world model in the real machine environment can be improved, thereby improving the accuracy of the target action prediction model trained by the world model. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating the steps of a robot motion control method based on a world model, provided as an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of a control device provided in another embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0048] It is understandable that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0049] In the model-based reinforcement learning framework, the latent space world model, as an environment simulator within the reinforcement learning algorithm, can predict the latent variables, rewards, or values ​​at the next moment during the reinforcement learning update process, thereby assisting the action prediction model in policy updates and action evaluation.

[0050] Existing latent world models used in reinforcement learning are first trained in a simulation environment and then deployed to real robots. However, there are multi-dimensional differences between real robots and simulation environments, including differences in dynamics, control latency, joint execution errors, friction and contact model deviations, and camera timing differences. These differences lead to poor accuracy of the latent world models trained in simulation in real environments, which in turn affects the accuracy of the action prediction models trained by reinforcement learning.

[0051] To address the aforementioned problems, this application provides a robot motion control method, apparatus, device, and medium based on a world model. The method includes: acquiring real-world interaction data, wherein the real-world interaction data includes current observation information of the robot, current first action parameters, and observation information of the next state, all collected from a real physical environment. Each observation information includes state information of the robot body and an observation image of the robot collected at the same time; encoding the current observation information into a first reference latent variable using an encoder of a first world model, wherein the first world model is trained in a simulation environment; encoding the observation information of the next state into a second reference latent variable using the encoder; inputting the first reference latent variable and the current first action parameters into a latent space dynamics model of the first world model to predict a third reference latent variable for the next state; calibrating the first world model based on the state deviation between the second and third reference latent variables to obtain a second world model; and deploying the second world model on the robot so that the robot determines second action parameters based on the second world model and controls the robot's motion based on the second action parameters. According to the solution provided in the embodiments of this application, based on the robot's real machine state data, the state deviation between the robot's current real latent variables and the latent variables predicted by the world model obtained from simulation training is calculated. The world model is calibrated based on the state deviation. Compared with the existing application solution that is only based on the world model trained by simulation, the accuracy of the prediction results of the world model in the real machine environment can be improved, thereby improving the accuracy of the target action prediction model trained by the world model.

[0052] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0053] refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a robot motion control method based on a world model, as provided in one embodiment of this application. This embodiment of the application provides a robot motion control method based on a world model, which includes, but is not limited to, the following steps:

[0054] Step S10: Obtain real interaction data, wherein the real interaction data includes the robot's current observation information, current first action parameters and next state observation information collected from the real physical environment, and any observation information includes the robot's body state information and robot observation image collected at the same time.

[0055] It should be noted that the embodiments of this application do not limit the source of the real interaction data, which can be direct inference data from the real machine, teleoperation data, or human teaching data. Direct inference data from the real machine is used to indicate: deploying the action prediction model trained based on the first-world model obtained from simulation training onto the real robot for closed-loop inference, and recording the interaction data during the robot's actual task execution. Teleoperation or human teaching data is used to indicate: human intervention via teleoperation to control the real robot to complete actions such as approaching, grasping, carrying, or placing, and the system records the corresponding real state changes when the robot reproduces the human action. It is understood that when the real interaction data includes both direct inference data from the real machine and teleoperation or human teaching data, this embodiment unifies the two types of data into the same data format for calibrating the first-world model.

[0056] In addition, in some embodiments, after acquiring real interaction data, data cleaning is performed based on conditions such as timestamps, frame intervals, motion amplitudes, and observation validity to reduce the impact of camera desynchronization, abnormal motion, or invalid observations on the calibration operation of the first-world model. For critical mission phases or low success rate phases, this embodiment can also resample the real interaction data or increase the sampling weight to enhance the adaptability of the first-world model's calibration operation to critical operation processes.

[0057] Specifically, the observation information in this embodiment includes the state information of the robot body and the observation image of the robot (which may include RGB image, depth map or RGBD image) collected at the same time. The state information of the robot body may include the joint state of the robot, tactile signals, etc., and will not be limited here.

[0058] Understandably, obtaining real-world interaction data can provide an effective data foundation for subsequent calibration of the first-world model.

[0059] Step S20: The current observation information is encoded into a first reference latent variable by the encoder of the first-world model, wherein the first-world model is trained in a simulation environment.

[0060] It should be noted that in this embodiment, the current observation information is encoded into a first reference latent variable by the encoder of the first-world model, and is calculated according to the following formula:

[0061] ;

[0062] in, As the first reference latent variable, Based on current observation information, The encoder for the first-world model.

[0063] It should be noted that the embodiments of this application do not limit the specific architecture of the first world model, and can be a world model of the TD-MPC2, Dreamer, PlaNet or RSSM type.

[0064] Step S30: The observation information of the next state is encoded into a second reference latent variable by the encoder.

[0065] It should be noted that in this embodiment, the observation information of the next state is encoded into a second reference latent variable by an encoder, which is calculated according to the following formula:

[0066] ;

[0067] in, As the second reference latent variable, This provides observation information for the next state.

[0068] Step S40: Input the first reference latent variable and the current first action parameter into the latent space dynamics model of the first world model to predict the third reference latent variable for the next state.

[0069] It should be noted that in this embodiment, the first reference latent variable and the current first action parameter are input into the latent space dynamics model of the first world model to predict the third reference latent variable of the next state, which is calculated according to the following formula:

[0070] ;

[0071] in, As the third reference latent variable, This is the parameter for the current first action.

[0072] Understandably, based on steps S20 to S40, the robot's current real latent variables (i.e., the second reference latent variables) and the latent variables predicted by the world model obtained from simulation training (i.e., the third reference latent variables) are calculated, providing an effective data basis for subsequent calibration of the first world model.

[0073] Step S50: Calibrate the first-world model based on the state deviation between the second and third reference latent variables to obtain the second-world model.

[0074] It is understood that the specific method by which this embodiment uses the state deviation between the second and third reference latent variables to calibrate the first-world model depends on the actual situation of the state deviation. Specifically, the actual situation of the state deviation will affect the loss function used to calibrate the first-world model and / or affect specific modules in the first-world model (such as encoders, latent space dynamics models, etc.).

[0075] Specifically, in some embodiments, Figure 1 Step S50 includes, but is not limited to, the following steps:

[0076] Step S501: Calculate the first consistency error between the second reference latent variable and the third reference latent variable;

[0077] Step S502: When the first consistency error is less than the first threshold and the difference between the first consistency error and the simulation verification error is less than the second threshold, the first world model is updated based on the first consistency error to obtain the second world model. Here, the simulation verification error is the error value between the latent variables of the next state calculated by the world model based on historical simulation data in the simulation environment and the latent variables of the actual next state given by the simulation environment.

[0078] It is understood that in this embodiment, the first consistency error between the second reference latent variable and the third reference latent variable is calculated and used as the state deviation. When the first consistency error is less than the first threshold, or the difference between the first consistency error and the simulation verification error is less than the second threshold, it is determined that there is a small difference between the robot's current real observation information and the simulation prediction observation information. At this time, the first world model is updated based on the first consistency error to obtain the calibrated second world model.

[0079] It should be noted that this embodiment does not limit the specific modules of the first world model to be updated based on the first consistency error. It can update the model parameters of only a single module in the first world model, or it can update the model parameters of the entire first world model. For example, when the first world model is a TD-MPC2 type world model, the module to be calibrated is a latent space dynamics model; when the first world model is a Dreamer type or RSSM type world model, the posterior latent state obtained by encoding the observation information based on the actual next state is used as the supervision target to calibrate the action condition transition model or prior dynamics model.

[0080] It should be noted that, in this embodiment, the first consistency error between the second reference latent variable and the third reference latent variable is calculated according to the following formula:

[0081] ;

[0082] in, This is the first consistency error.

[0083] Specifically, step S502 includes, but is not limited to, the following steps:

[0084] Step S5021: Keep the model parameters of the remaining modules in the first-world model, except for the latent space dynamics model, unchanged.

[0085] Step S5022: Update the model parameters of the latent space dynamics model in the first world model based on the first consistency error to obtain the second world model.

[0086] It is understandable that the first-world model includes not only the encoder and latent space dynamics model, but also modules such as reward prediction model, value assessment model, and policy prior. Referring to the description of the above embodiments, when it is determined that there is a small difference between the robot's current real observation information and the simulation prediction observation information, it indicates that the encoder of the first-world model still has good applicability. Therefore, this embodiment only updates the model parameters of the latent space dynamics model in the first-world model based on the first consistency error, keeping the model parameters of the remaining modules (including reward prediction model, value assessment model, policy prior, etc.) in the first-world model unchanged. In this way, the real observation information can still be mapped to the same latent variable space formed in the simulation training stage, so that the encoder, reward model, value model, and policy prior can still correctly interpret the meaning of latent variables, avoiding the failure of the original task semantics and policy structure due to the drift of the latent space dynamics model or encoder.

[0087] Specifically, in some embodiments, Figure 1 Step S50 includes, but is not limited to, the following steps:

[0088] Step S503: When the first consistency error is greater than or equal to the first threshold, or the difference between the first consistency error and the simulation verification error is greater than or equal to the second threshold, a difference correction module is configured at the input or output of the encoder. The difference between the first model parameter quantity of the difference correction module and the second model parameter quantity of the encoder is greater than the third threshold, and the first model parameter quantity is less than the second model parameter quantity. The difference correction module is used to correct the distribution difference between the robot's current real observation information and the simulation observation information.

[0089] Step S504: Obtain the first target latent variable, wherein the first target latent variable satisfies any of the following conditions: when the difference correction module is configured at the input end of the encoder, the first target latent variable is the data encoded by the encoder after correcting the current observation information based on the difference correction module; when the difference correction module is configured at the output end of the encoder, the first target latent variable is the latent variable obtained by the difference correction module after correcting the latent variable encoded by the encoder based on the current observation information.

[0090] Step S505: Obtain the second target latent variable, wherein the second target latent variable satisfies any of the following conditions: when the difference correction module is configured at the input end of the encoder, the second target latent variable is obtained by encoding the data after the encoder corrects the observation information of the next state based on the difference correction module; when the difference correction module is configured at the output end of the encoder, the second target latent variable is obtained by correcting the latent variable after the difference correction module corrects the latent variable after encoding the observation information of the next state based on the encoder.

[0091] Step S506: Input the first target latent variable and the current first action parameter into the latent space dynamics model to predict the first intermediate latent variable of the next state;

[0092] Step S507: Calculate the second consistency error between the first intermediate latent variable and the second target latent variable;

[0093] Step S508: Construct a first latent space anchoring constraint term based on the first target latent variable and the first reference latent variable; calculate the first target loss value based on the second consistency error and the first latent space anchoring constraint term, wherein the first latent space anchoring constraint term is used to constrain the difference between the first target latent variable and the first reference latent variable;

[0094] Step S509: Update the model parameters of the difference correction module, the encoder, and the latent space dynamics model in the first world model based on the first target loss value to obtain the second world model.

[0095] It should be noted that this embodiment does not limit the specific structure of the difference correction module. It can include one or two layers of residual fully connected networks. The number of model parameters in the difference correction module is less than the number of model parameters in the coding layer. The difference correction module can be initialized in a manner close to identity mapping.

[0096] It should be noted that, in the process of updating the model parameters of the difference correction module, the encoder, and the latent space dynamics model, this embodiment does not limit which specific part of the model parameters of the difference correction module, the encoder, or the latent space dynamics model should be updated. Those skilled in the art can determine this according to the actual situation.

[0097] Understandably, by using the residual form, when the latent variables obtained after encoding the current observation information through the difference correction module are close to zero, the adapted latent variables (the first target latent variables obtained after configuring the difference correction module at the input or output of the encoder of the first world model) can be made to be close to zero. Latent variables close to the original encoder output This avoids causing significant changes to the original potential space during the initial training phase.

[0098] Understandably, when the first consistency error is greater than or equal to the first threshold, or the difference between the first consistency error and the simulation verification error is greater than or equal to the second threshold, it is determined that there is a significant difference between the robot's current real observation information and the simulation-predicted observation information. In this case, when the difference correction module is configured at the encoder input, the difference correction module converts the color, brightness, texture, and local visual features corresponding to the current observation information, outputs the corrected observation information, and then inputs the observation information back to the encoder, which outputs the first target latent variable. When the difference correction module is configured at the encoder output, the encoder first encodes the current observation information into an initial latent variable, and then inputs this initial latent variable into the difference correction module for correction to obtain the first target latent variable. Similarly, based on the different configurations of the difference correction module in the encoder, the second target latent variable corresponding to the observation information of the next state is obtained. In this way, obtaining the first and second target latent variables provides an effective data foundation for calculating the first target loss value used to update the first world model.

[0099] It should be noted that when the difference correction module is configured at the encoder output, the first target latent variable is calculated according to the following formula:

[0100] ;

[0101] When the difference correction module is configured at the encoder output, the second target latent variable is calculated according to the following formula:

[0102] ;

[0103] in, This is the difference correction module.

[0104] It should be noted that this embodiment will use the adjusted first target latent variable. Substitute the original latent variable As input to the latent space dynamics model of the first-world model, it predicts the corresponding latent variables for the next time step (i.e., the first intermediate latent variables). :

[0105] .

[0106] Understandably, when a significant difference is found between the robot's current real observation information and the simulation-predicted observation information, this embodiment calculates the second consistency error between the first intermediate latent variable and the second target latent variable, constructs a first latent space anchoring constraint term based on the first target latent variable and the first reference latent variable, calculates the first target loss value based on the second consistency error and the first latent space anchoring constraint term, and updates the model parameters of the difference correction module, the encoder, and the latent space dynamics model in the first world model based on the first target loss value to obtain the second world model. In other words, this embodiment updates the encoder and latent space dynamics model parameters in the first world model based on a loss function that incorporates latent space anchoring constraints, ensuring that the fine-tuned real observation latent variables (i.e., the first target latent variable and the second target latent variable) maintain a latent space distribution close to the original encoder output. This effectively avoids a full parameter update of the entire first world model, reducing computational costs while ensuring the stability of the model update.

[0107] In addition, in some embodiments, the first target latent variable and the second target latent variable can be provided to the reward prediction model, the value assessment model and the policy prior. The correction magnitude generated by the difference correction module is limited by the latent space anchoring constraint term, so that the difference correction module can reduce the difference between the real observation (i.e. the second target latent variable) and the simulated predicted observation (i.e. the first intermediate latent variable) while maintaining the latent space structure formed by the first world model in the simulation training stage as much as possible.

[0108] Specifically, in some embodiments, the encoder includes multiple sequentially connected coding layers, which include a first coding layer to an nth coding layer, where n is a positive integer greater than 1. Figure 1 Step S50 includes, but is not limited to, the following steps:

[0109] Step S510: When the first consistency error is greater than or equal to the first threshold, or the difference between the first consistency error and the simulation verification error is greater than or equal to the second threshold, the parameters of the m-th to n-th coding layers in the encoder are updated using the target learning rate to obtain a new encoder, where m is a positive integer greater than 1 and less than n;

[0110] Step S511: Encode the current observation information into a third target latent variable using a new encoder;

[0111] Step S512: Encode the observation information of the next state into the fourth target latent variable using a new encoder;

[0112] Step S513: Input the third target latent variable and the current first action parameter into the latent space dynamics model to predict the second intermediate latent variable of the next state;

[0113] Step S514: Calculate the third consistency error between the second intermediate latent variable and the fourth target latent variable;

[0114] Step S515: Construct a second latent space anchoring constraint term based on the third target latent variable and the first reference latent variable; calculate the second target loss value based on the third consistency error and the second latent space anchoring constraint term, wherein the second latent space anchoring constraint term is used to constrain the difference between the third target latent variable and the first reference latent variable;

[0115] Step S516: Update the model parameters of the new encoder and the model parameters of the latent space dynamics model in the first world model based on the second target loss value to obtain the second world model.

[0116] Specifically, in this embodiment, when the first consistency error is greater than or equal to the first threshold, or the difference between the first consistency error and the simulation verification error is greater than or equal to the second threshold, i.e., when it is determined that there is a significant difference between the robot's current real observation information and the simulation predicted observation information, the parameters of the m-th to n-th coding layers in the encoder are updated using the target learning rate to obtain a new encoder, where m is a positive integer greater than 1 and less than n, i.e., only the coding layers with lower layer numbers in the encoder of the first world model are fine-tuned with a low learning rate (corresponding to the target learning rate; this embodiment does not limit the specific value of the target learning rate, which can be determined by those skilled in the art according to the actual situation), to obtain a new encoder. Afterwards, the current observation information is encoded into the m-th coding layer using the new encoder. The model uses three target latent variables and encodes the observation information of the next state as a fourth target latent variable. Then, the third target latent variable and the current first action parameter are input into the latent space dynamics model to predict the second intermediate latent variable of the next state. The third consistency error between the second intermediate latent variable and the fourth target latent variable is calculated. A second latent space anchoring constraint term is constructed based on the third target latent variable and the first reference latent variable. The second target loss value is calculated based on the third consistency error and the second latent space anchoring constraint term. The second latent space anchoring constraint term is used to constrain the difference between the third target latent variable and the first reference latent variable. The model parameters of the new encoder and the model parameters of the latent space dynamics model in the first world model are updated based on the second target loss value to obtain the second world model. It is understood that, referring to the description of the above embodiments (steps S503 to S509), this embodiment is also based on the loss function of introducing latent space anchoring constraint terms to update the model parameters of the encoder and the model parameters of the latent space dynamics model in the first world model, so that the fine-tuned real observed latent variables (i.e., the third target latent variable and the fourth target latent variable) maintain a latent space distribution close to the original encoder output. The difference lies in the adjustment method of the encoder of the first world model. The embodiments corresponding to steps S510 to S516 are to perform low learning rate fine-tuning on the coding layers with lower layer numbers in the encoder, while the embodiments corresponding to steps S503 to S509 are to configure a difference correction module at the input or output end of the encoder.

[0117] It should be noted that this embodiment does not limit the specific values ​​of the first threshold, the second threshold, and the third threshold; those skilled in the art can determine them according to the actual situation.

[0118] Specifically, in some embodiments, step S508 includes, but is not limited to, the following steps:

[0119] Calculate the first Euclidean distance between the first target latent variable and the first reference latent variable, and determine the square of the first Euclidean distance as the first latent space anchoring constraint term;

[0120] The first target loss value is obtained by weighted summation of the second consistency error and the first latent space anchoring constraint term.

[0121] Specifically, in some embodiments, step S515 includes, but is not limited to, the following steps:

[0122] Calculate the second Euclidean distance between the third target latent variable and the first reference latent variable, and determine the square of the second Euclidean distance as the second latent space anchoring constraint term;

[0123] The second target loss value is obtained by weighted summation of the third consistency error and the second latent space anchoring constraint term.

[0124] It should be noted that the first target loss value in this embodiment is calculated according to the following formula:

[0125] ;

[0126] ;

[0127] in, for, This is the second consistency error. As weight, Anchor constraints for the first potential space. As the first target latent variable, It is the first reference latent variable.

[0128] It should be noted that the second target loss value in this embodiment is calculated according to the following formula:

[0129] ;

[0130] ;

[0131] in, for, This is the third consistency error. Anchoring constraints for the second potential space. This is the third latent variable for the objective.

[0132] Step S60: Deploy the second world model on the robot so that the robot can determine the second motion parameters based on the second world model and control the robot's movement based on the second motion parameters.

[0133] Understandably, the examples in this application can calculate the state deviation between the robot's current real latent variables and the latent variables predicted by the world model obtained from simulation training, based on the robot's real interaction data. The first world model is calibrated based on the state deviation, and a second world model is obtained after calibration. Furthermore, the second world model is deployed on the robot, enabling the robot to perform online motion planning directly based on the second world model to obtain second motion parameters for controlling the robot's movement (e.g., the implementation method corresponding to the TD-MPC2 type world model); or, a target model for robot control is trained based on the second world model, and the newly acquired state information and observation images of the robot body are input into the target model to obtain second motion parameters for controlling the robot's movement (e.g., the implementation method corresponding to the Dreamer type world model).

[0134] Thus, when the robot performs closed-loop control in a real environment, the latent space model's prediction of the consequences of real actions is closer to the actual robot's execution results. Therefore, model predictive control or policy planning can select more stable action sequences that are more in line with real dynamics, thus ensuring the accuracy of the target model.

[0135] Furthermore, compared to directly retraining a complete world model or policy on a real robot, the calibration method of this application embodiment does not rely on real dense reward reconstruction, does not require future image reconstruction, has lower data requirements and computational costs, and can retain the visual representations, task semantics and policy structure learned in the simulation stage, while improving the accuracy of latent space prediction, closed-loop control stability and task success rate in real deployment.

[0136] like Figure 2 As shown, Figure 2This is a structural diagram of a control device provided in one embodiment of this application. The present invention also provides a control device 200, including: a processor 210, which can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing relevant programs to implement the technical solutions provided in the embodiments of this application; and a memory 220, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 220 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 210 to implement the robot motion control method based on the world model of this application embodiment. The input / output interface 230 is used to realize information input and output. The communication interface 240 is used to realize communication interaction between this device and other devices. Communication can be realized by wired means (e.g., USB, network cable, etc.) or by wireless means (e.g., mobile network, WIFI, Bluetooth, etc.). The bus 250 transmits information between the various components of the device (e.g., processor 210, memory 220, input / output interface 230 and communication interface 240). The processor 210, memory 220, input / output interface 230 and communication interface 240 realize communication connection between each other within the device through the bus 250.

[0137] In addition, this application also provides an electronic device, including the control device 200 described in the above embodiments.

[0138] In addition, this application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described robot motion control method based on a world model.

[0139] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or temporary media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies. Digital multifunction discs (DVDs) or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0141] The above provides a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A robot motion control method based on a world model, characterized in that, The method includes: Acquire real interaction data, wherein the real interaction data includes the robot's current observation information, current first action parameters and next state observation information collected from the real physical environment, and any observation information includes the robot's own state information and the robot's observation image collected at the same time. The current observation information is encoded into a first reference latent variable by the encoder of the first world model, wherein the first world model is trained in a simulation environment; The encoder encodes the observation information of the next state into a second reference latent variable; The first reference latent variable and the current first action parameter are input into the latent space dynamics model of the first world model to predict the third reference latent variable of the next state; The first world model is calibrated based on the state deviation between the second reference latent variable and the third reference latent variable to obtain the second world model; The second world model is deployed on the robot so that the robot determines the second motion parameters based on the second world model and controls the robot's movement based on the second motion parameters.

2. The control method according to claim 1, characterized in that, The first world model is calibrated based on the state deviation between the second reference latent variable and the third reference latent variable to obtain the second world model, including: Calculate the first consistency error between the second reference latent variable and the third reference latent variable; When the first consistency error is less than the first threshold, and the difference between the first consistency error and the simulation verification error is less than the second threshold, the first world model is updated based on the first consistency error to obtain the second world model. The simulation verification error is the error value between the latent variables of the next state calculated by the world model based on historical simulation data in the simulation environment and the latent variables of the actual next state given by the simulation environment.

3. The control method according to claim 2, characterized in that, The first world model is updated based on the consistency error to obtain the second world model, including: The model parameters of the remaining modules in the first world model, except for the latent space dynamics model, remain fixed. The model parameters of the latent space dynamics model in the first world model are updated based on the first consistency error to obtain the second world model.

4. The control method according to claim 2, characterized in that, The first world model is calibrated based on the state deviation between the second reference latent variable and the third reference latent variable to obtain the second world model, including: When the first consistency error is greater than or equal to the first threshold, or the difference between the first consistency error and the simulation verification error is greater than or equal to the second threshold, a difference correction module is configured at the input or output end of the encoder. The difference between the first model parameter quantity of the difference correction module and the second model parameter quantity of the encoder is greater than a third threshold, and the first model parameter quantity is less than the second model parameter quantity. The difference correction module is used to correct the distribution difference between the robot's current real observation information and simulation observation information. Obtain a first target latent variable, wherein the first target latent variable satisfies any of the following conditions: when the difference correction module is configured at the input end of the encoder, the first target latent variable is obtained by the encoder encoding the data after correcting the current observation information based on the difference correction module; when the difference correction module is configured at the output end of the encoder, the first target latent variable is obtained by the difference correction module correcting the latent variable encoded by the encoder based on the current observation information. Obtain a second target latent variable, wherein the second target latent variable satisfies any of the following conditions: when the difference correction module is configured at the input end of the encoder, the second target latent variable is obtained by the encoder encoding data after correcting the observation information of the next state based on the difference correction module; when the difference correction module is configured at the output end of the encoder, the second target latent variable is obtained by the difference correction module correcting the latent variable after encoding the observation information of the next state based on the encoder. The first target latent variable and the current first action parameter are input into the latent space dynamics model to predict the first intermediate latent variable of the next state. Calculate the second consistency error between the first intermediate latent variable and the second target latent variable; A first latent space anchoring constraint term is constructed based on the first target latent variable and the first reference latent variable. A first target loss value is calculated based on the second consistency error and the first latent space anchoring constraint term. The first latent space anchoring constraint term is used to constrain the difference between the first target latent variable and the first reference latent variable. The model parameters of the difference correction module, the encoder, and the latent space dynamics model in the first world model are updated based on the first target loss value to obtain the second world model.

5. The control method according to claim 2, characterized in that, The encoder includes multiple sequentially connected coding layers, each coding layer comprising a first coding layer to an nth coding layer, where n is a positive integer greater than 1. The first world model is calibrated based on the state deviation between the second reference latent variable and the third reference latent variable to obtain a second world model, including: When the first consistency error is greater than or equal to the first threshold, or the difference between the first consistency error and the simulation verification error is greater than or equal to the second threshold, the parameters of the m-th coding layer to the n-th coding layer in the encoder are updated using the target learning rate to obtain a new encoder, where m is a positive integer greater than 1 and less than n; The new encoder encodes the current observation information into a third target latent variable. The new encoder encodes the observation information of the next state into a fourth target latent variable; The third target latent variable and the current first action parameter are input into the latent space dynamics model to predict the second intermediate latent variable of the next state. Calculate the third consistency error between the second intermediate latent variable and the fourth target latent variable; A second latent space anchoring constraint term is constructed based on the third target latent variable and the first reference latent variable. A second target loss value is calculated based on the third consistency error and the second latent space anchoring constraint term. The second latent space anchoring constraint term is used to constrain the difference between the third target latent variable and the first reference latent variable. The model parameters of the new encoder and the model parameters of the latent space dynamics model in the first world model are updated based on the second target loss value to obtain the second world model.

6. The control method according to claim 4, characterized in that, A first latent space anchoring constraint term is constructed based on the first target latent variable and the first reference latent variable. The first target loss value is calculated based on the second consistency error and the first latent space anchoring constraint term, including: Calculate the first Euclidean distance between the first target latent variable and the first reference latent variable, and determine the square of the first Euclidean distance as the first latent space anchoring constraint term; The first target loss value is obtained by weighted summing of the second consistency error and the first latent space anchoring constraint term.

7. The control method according to claim 5, characterized in that, A second latent space anchoring constraint term is constructed based on the third target latent variable and the first reference latent variable. The second target loss value is calculated based on the third consistency error and the second latent space anchoring constraint term, including: Calculate the second Euclidean distance between the third target latent variable and the first reference latent variable, and determine the square of the second Euclidean distance as the second latent space anchoring constraint term; The second target loss value is obtained by weighted summing of the third consistency error and the second latent space anchoring constraint term.

8. A control device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the world model-based robot motion control method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, Includes the control device as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the world-model-based robot motion control method as described in any one of claims 1 to 7.

Images