A Method and Related Device for Generating Robot Motion Sequences Based on Physics-Based Distillation
By using a physics-based distillation method, a latent space world state representation is generated and a robot action sequence is directly generated. This solves the problems of high computational overhead and poor physical consistency of existing robot world models in complex environments, and improves the real-time performance and robustness of robot operation tasks.
Patent Information
- Application Number
- CN202611516635.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-28
- Publication Date
- 2026-11-17
AI Technical Summary
Existing robot world models suffer from problems such as high computational overhead for future video generation in complex environments, low efficiency of future imagination during the testing phase, insufficient environmental dynamics modeling capabilities, and poor physical consistency in complex contact tasks, resulting in insufficient robustness and generalization ability in robot operation tasks.
A physical knowledge-based distillation method is adopted to generate a latent world state representation by acquiring current environmental observations, robot state, and task description information. Then, using a pre-trained latent world representation model and robot world action model, robot action sequences are directly generated, avoiding future video generation and state scrolling prediction, thus achieving physical consistency and real-time performance.
It enables the direct generation of robot action sequences without the need for future video generation and state scrolling prediction, improving the real-time performance, robustness, and generalization ability of robot operation tasks, and ensuring physical consistency and task executability.
Smart Images

Figure CN123103672A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot control technology, and relates to a method and related apparatus for generating robot motion sequences based on physical knowledge distillation. Background Technology
[0002] With the rapid development of embodied intelligence and autonomous robot operation technologies, robots need to possess the ability to understand, predict, and reason about dynamic changes in the environment in order to achieve autonomous decision-making and operation in complex environments. In recent years, world modeling technology has gradually become an important research direction in the field of robotics. Its core idea is to learn the robot's actions, changes in environmental states, and the interaction patterns between the robot and the environment, and to use the learned dynamic environmental model to assist the robot in completing planning, control, and decision-making tasks.
[0003] Existing robot world modeling methods mainly fall into two categories: video prediction-based world models and future-vision-based planning models. Video prediction-based methods typically model the future state of the environment by generating future video sequences and then use these future videos to plan actions and make decisions. Future-vision-based methods, on the other hand, usually require multi-step rolling predictions of the future state during the testing phase and combine this with model predictive control or search algorithms to generate control actions.
[0004] However, existing technologies suffer from at least the following problems: future video generation tasks require predicting a large amount of background information and visual details that are irrelevant to the task, resulting in high model training costs, low inference efficiency, and a tendency to accumulate errors; the future imagination process during the testing phase usually requires multiple state rolling predictions and online planning searches, which is difficult to meet the real-time control requirements of robots; existing world models usually focus mainly on visual change patterns, but lack the ability to model the contact evolution patterns, physical constraint patterns, and environmental dynamics patterns during robot operation; robot operation tasks generally involve complex physical interaction processes such as collisions, contact, sliding, insertion, and assembly, and traditional visual world models cannot guarantee the physical rationality of the prediction results and the executability of the task; existing world models and action models usually adopt independent training methods, lacking an effective transfer mechanism for environmental dynamic knowledge to action generation models, resulting in insufficient robustness and generalization ability of action generation. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and related apparatus for generating robot motion sequences based on physical knowledge distillation.
[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, this invention provides a method for generating robot action sequences based on physical knowledge distillation, comprising: acquiring current environmental observation information, current robot state information, and task description information; generating a latent space world state representation based on a pre-trained implicit world representation model, according to the current environmental observation information, current robot state information, and task description information; wherein the latent space world state representation is used to represent the task-related change relationship between the current environment and the target environment; and generating a robot action sequence based on a pre-trained robot world action model, according to the current environmental observation information, current robot state information, task description information, and latent space world state representation; wherein the pre-trained implicit world representation model... The model is obtained through the following steps: An initial physically consistent world simulation model and an initial implicit world representation model are constructed; wherein, the physically consistent world simulation model takes multimodal interaction data as input and future state prediction results as output; the implicit world representation model takes environmental observation information, robot state information, and task description information as input and latent space world state representation as output; the initial physically consistent world simulation model is trained based on multimodal interaction data of the robot's historical task execution, resulting in a pre-trained physically consistent world simulation model; the capabilities of the pre-trained physically consistent world simulation model are transferred to the initial implicit world representation model through knowledge distillation, resulting in a pre-trained implicit world representation model.
[0007] Optionally, the multimodal interaction data includes: environmental observation information, environmental depth information, robot state information, robot motion information, task description information, and interaction information between the robot and the environment; wherein, the interaction information between the robot and the environment includes: contact information, force information, torque information, slip information, friction state information, target object state information, and task stage information; the future state prediction result includes the environmental state change results, target object motion results, and interaction results between the robot and the environment at several future time points.
[0008] Optionally, the multimodal interaction data is collected in one or more states during robot autonomous execution, remote teaching, human-machine shared control, teaching learning, reinforcement learning exploration, and autonomous interaction in a real environment, and then processed through time synchronization.
[0009] Optionally, during the initial training of the physically consistent world simulation model, trajectory consistency constraints and relation consistency constraints are used for optimization; wherein, the trajectory consistency constraints are used to constrain the continuity and geometric consistency of the motion trajectories of the target object, the robot end effector, and environmental entities at consecutive moments; the relation consistency constraints are used to constrain the consistency of contact relationships, spatial topological relationships, and interactive semantic relationships among the robot, the target object, and environmental entities.
[0010] Optionally, the trajectory consistency constraint is constructed based on one or more of pixel-level trajectory supervision information, feature-level trajectory supervision information, or latent space trajectory supervision information; the relationship consistency constraint is constructed based on one or more of object relationship graphs, interaction relationship graphs, or scene relationship graphs.
[0011] Optionally, the initial implicit world representation model is constructed using an image editing-based implicit world representation mechanism; when generating the latent space world state representation model, the features of the task-related region, the interaction key region, and the contact-sensitive region are assigned a first weight, and the features of the background region and the task-irrelevant visual information region are assigned a second weight, wherein the first weight is greater than the second weight.
[0012] Optionally, the latent space world state representation includes one or more of the following: latent space dynamic features, attention features, cross-modal features, historical state memory features, contact state features, and task condition features; the robot action sequence includes one or more of the following: joint space action sequence, task space action sequence, end-effector pose action sequence, force control action sequence, impedance control action sequence, admittance control action sequence, or hybrid control action sequence.
[0013] Optionally, the pre-trained robot world action model is obtained by: constructing an initial robot world action model; generating a latent space world state representation corresponding to the multimodal interaction data based on the multimodal interaction data and the pre-trained implicit world representation model; and training the initial robot world action model based on the multimodal interaction data and the latent space world state representation corresponding to the multimodal interaction data to obtain the pre-trained robot world action model.
[0014] Optionally, it also includes: based on the latent space world state representation, predicting collision events, abnormal contact events, slip events, assembly failure events, grab failure events, or task deviation events based on a preset risk model, and generating compensation actions, recovery actions, search actions, replanning actions, or safety actions in conjunction with current environmental observation information.
[0015] In a second aspect, the present invention provides a robot action sequence generation system based on physical knowledge distillation, comprising: a data acquisition module for acquiring current environmental observation information, current robot state information, and task description information; an implicit world representation module for generating a latent world state representation based on a pre-trained implicit world representation model, according to the current environmental observation information, current robot state information, and task description information; wherein the latent world state representation represents the task-related change relationship between the current environment and the target environment; and an action generation module for generating a robot action sequence based on a pre-trained robot world action model, according to the current environmental observation information, current robot state information, task description information, and latent world state representation. The pre-trained implicit world representation model is obtained through the following steps: constructing an initial physically consistent world simulation model and an initial implicit world representation model; wherein, the physically consistent world simulation model takes multimodal interaction data as input and future state prediction results as output; the implicit world representation model takes environmental observation information, robot state information, and task description information as input and latent space world state representation as output; acquiring and training the initial physically consistent world simulation model based on the multimodal interaction data of the robot's historical task execution to obtain the pre-trained physically consistent world simulation model; and transferring the capabilities of the pre-trained physically consistent world simulation model to the initial implicit world representation model through knowledge distillation to obtain the pre-trained implicit world representation model.
[0016] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for generating robot motion sequences based on physical knowledge distillation.
[0017] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for generating robot motion sequences based on physical knowledge distillation.
[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention presents a robot action sequence generation method based on physical knowledge distillation. By acquiring current environmental observation information, current robot state information, and task description information, it provides a comprehensive and accurate multimodal input foundation for subsequent latent world state representation generation and robot action sequence generation. Based on the current environmental observation information, current robot state information, and task description information, a latent world state representation is generated using a pre-trained latent world representation model. This latent world state representation is used to represent the task-related changes between the current environment and the target environment, enabling the robot to understand these changes in the form of a latent world representation without explicitly generating future environmental states, future observation results, or complete future video sequences. Furthermore, based on the current environmental observation information, current robot state information, task description information, and latent world state representation, a robot action sequence is generated using a pre-trained robot world action model. This allows the robot world action model to directly generate robot action sequences under the conditions of the latent world state representation. In the pre-training process of the implicit world representation model, an initial physically consistent world simulation model and an initial implicit world representation model are constructed. Multimodal interaction data of the robot's historical task execution are acquired and used to train the initial physically consistent world simulation model, resulting in a pre-trained physically consistent world simulation model. This model enables the physical consistent world simulation model to learn the environmental dynamics, contact evolution, and task execution laws in robot operation tasks. Then, through knowledge distillation, the capabilities of the pre-trained physically consistent world simulation model are transferred to the initial implicit world representation model, resulting in a pre-trained implicit world representation model. This model enables the latent space world state representation to simultaneously possess environmental dynamic understanding and action generation capabilities, thereby realizing knowledge transfer from the physical world model to the action generation model. Through the training of the physically consistent world simulation model and the capability transfer through knowledge distillation, the direct generation of robot action sequences without the need for future video generation, future state scrolling prediction, or future imagination during the testing phase is achieved. This is suitable for complex task scenarios and can effectively improve the real-time performance, robustness, generalization ability, and physical consistency of robot operation tasks. Attached Figure Description
[0019] Figure 1 This is a flowchart of a robot motion sequence generation method based on physical knowledge distillation, according to an embodiment of the present invention.
[0020] Figure 2 This is a block diagram of a robot action sequence generation system based on physical knowledge distillation, according to an embodiment of the present invention. Detailed Implementation
[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0023] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, a robot action sequence generation method based on physical knowledge distillation is provided to solve the problems of high computational overhead for future video generation, low efficiency of future imagination during the testing phase, insufficient environmental dynamics modeling capability, and poor physical consistency of complex contact tasks in existing robot world models in complex real environments. This method enables the generation and autonomous control of actions for complex operation tasks without the need for future video generation, future state decoding, or future imagination during the testing phase.
[0024] Specifically, the robot action sequence generation method based on physical knowledge distillation of the present invention includes the following steps: S1: Obtain current environmental observation information, current robot status information, and task description information.
[0025] S2: Based on the current environmental observation information, the current robot state information, and the task description information, a latent world state representation is generated using a pre-trained implicit world representation model. This latent world state representation is used to represent the task-related changes between the current environment and the target environment.
[0026] S3: Based on the current environmental observation information, current robot state information, task description information, and latent space world state representation, generate robot action sequences based on the pre-trained robot world action model.
[0027] The pre-trained implicit world representation model is obtained through the following steps: constructing an initial physically consistent world simulation model and an initial implicit world representation model; wherein the physically consistent world simulation model takes multimodal interaction data as input and future state prediction results as output; the implicit world representation model takes environmental observation information, robot state information, and task description information as input and latent space world state representation as output; acquiring and training the initial physically consistent world simulation model based on the multimodal interaction data of the robot's historical task execution to obtain the pre-trained physically consistent world simulation model; and transferring the capabilities of the pre-trained physically consistent world simulation model to the initial implicit world representation model through knowledge distillation to obtain the pre-trained implicit world representation model.
[0028] Explanatoryly, step S1 provides a comprehensive and accurate multimodal input foundation for subsequent latent space world state representation generation and robot action sequence generation. Here, the current environment observation information refers to the perceived information in the robot's current operating environment, including but not limited to visual observation information and depth information, used to characterize the environmental scene currently faced by the robot. The current robot state information refers to the robot's own ontological state information, including but not limited to the angles, velocities, and torques of each joint, as well as the pose information of the robot's end effector, used to characterize the robot's current motion state. The task description information refers to information describing the task the robot needs to perform, including but not limited to natural language commands and task target information, used to characterize the operational task the robot needs to complete. It should be noted that the current environment observation information, current robot state information, and task description information can be obtained through sensors carried by the robot itself or through external sensing devices; this invention does not specifically limit this. For example, when the robot performs a plug-in assembly task, the current environmental observation information may include environmental image information including sockets and plugs collected by the robot's vision sensor, the current robot state information may include the angle and torque information of each joint of the robot, and the task description information may be a natural language instruction to insert the plug into the socket.
[0029] Explained, step S2 enables the robot to understand the task-related changes between the current and target environmental states in the form of a latent space representation, without explicitly generating future environmental states, future observations, or complete future video sequences. The latent world representation model learns the local changes between the current and target environmental states, focusing on modeling the visual changes in task-related regions, key interaction regions, and contact-sensitive regions during robot operation. The latent space world state representation refers to the latent space feature representation used to represent the task-related changes between the current and target environments, constituting the world context information during robot action generation. It should be noted that the latent space world state representation is not an explicit prediction of future environmental states, but rather an implicit encoding of the task-related changes between the current and target states, thus avoiding the computational overhead of a large amount of task-irrelevant visual information during future video generation. For example, when the robot performs a plug-in assembly task, the latent space world state representation can characterize the relative changes between the current plug position and the target socket position, as well as the required posture adjustment information.
[0030] For example, the implicit world representation model can employ a conditional coding network based on image editing feature extraction, including a visual coding layer, a robot state coding layer, a task coding layer, a region-weighted layer, a conditional interaction layer, and a latent space projection layer. The visual coding layer is connected to the region-weighted layer; the region-weighted layer, robot state coding layer, and task coding layer are each connected to the conditional interaction layer; the conditional interaction layer is connected to the latent space projection layer; and the output of the latent space projection layer serves as the conditional input to the robot world action model. These layers can be trained jointly, or the robot state coding layer and latent space projection layer can be added to a pre-trained image editing network before training.
[0031] Specifically, the visual encoding layer divides the current environment image into image blocks and uses convolutional networks or visual transformation networks to extract visual features with spatial location identifiers; when depth information exists, depth features aligned with the image are fused into the corresponding image block. The robot state encoding layer normalizes state variables such as joint angles, joint velocities, and end-effector poses, and maps them to state features through a fully connected network. The task encoding layer encodes the natural language task description into task features; when the task target is represented by structured information such as pose and target category, the corresponding information is projected to the same dimension as the task features. The region weighting layer weights the corresponding visual features based on the recognition results of task-related regions, key interaction regions, and contact-sensitive regions; the conditional interaction layer uses the weighted visual features as features to be edited, and the robot state features and task features as editing conditions, extracting the feature increments corresponding to the changes in the current environment towards the task target through cross-attention, residual connections, and feedforward transformation. The feature increments reflect changes in the relative pose of the target object, changes in the operating parts, and changes in contact relationships, and do not require the formation of a complete, decorable target image. The latent space projection layer pools, linearly projects, or aggregates the output of the conditional interaction layer to obtain a fixed number of latent space feature vectors of fixed dimensions, which are then used as the latent space world state representation. During training, subsequent observations from the same task trajectory can be used to form editing supervision, and features provided by the physically consistent world simulation model can be used for distillation. During online runtime, only the current observation, the current robot state, and the task description are used to compute the above representation, without calling the image decoding layer to generate the target image or future video. Optional historical states are provided by cached features up to the current moment, which are initialized to empty or a preset initial value at the start of the task.
[0032] Explained, step S3 enables the robot world action model to directly generate robot action sequences under the condition of the latent space world state representation, achieving joint modeling between the world state change law and the robot action generation law. Specifically, the robot world action model adopts a joint state-action modeling approach, learning the causal relationship between the environmental state evolution law and the robot action generation law by uniformly encoding the current environmental state, the latent space world state representation, and the task description information. It should be noted that during the inference phase, only the current environmental observation information, the current robot state information, and the task description information need to be input to directly generate the robot's future action sequence, without needing to perform future video generation, future state rolling prediction, model predictive control, or future imagination processes during the testing phase, thereby achieving real-time closed-loop control in complex robot operation tasks. For example, when the robot performs a plug-in assembly task, the robot world action model can directly output the end-effector pose sequence of the plug moving to the socket position and the force control sequence during the insertion process based on the current environmental observation information, the current robot state information, and the latent space world state representation.
[0033] For example, the robot world action model can employ a conditional action sequence decoding network, including an observation encoding layer, a state encoding layer, a task encoding layer, a world feature adaptation layer, a multimodal fusion layer, and an action sequence decoding layer. The observation encoding layer, state encoding layer, and task encoding layer encode the current environment observation information, the current robot state information, and the task description information, respectively; the world feature adaptation layer maps the latent space world state representation to the feature dimensions adopted by the fusion layer; the above four features, after adding modal identifiers and position identifiers, are input into the multimodal fusion layer to form the action generation context.
[0034] Specifically, the action sequence decoding layer sets learnable query vectors that correspond one-to-one with each future action moment. Each query vector exchanges sequence information through self-attention and reads the action generation context through cross-attention. The action output head then simultaneously outputs the action parameters within the time domain of the future preset action. Taking a hybrid action involving end-effector pose and force control as an example, the action output head outputs the end-effector translation increment, rotation increment, gripper control quantity, and desired contact force, respectively; the rotation increment can be represented by a rotation vector. When joint control is used, the action output head outputs the joint position or velocity command accordingly, preventing the motion quantities from different control modes from being mixed under the same supervision label.
[0035] During training, the current input and its subsequent actual action segments are extracted from multimodal interaction data in chronological order. The subsequent actual action segments serve as supervision labels, and the representation calculated by the pre-trained implicit world representation model is used as the world condition. Continuous action components are first standardized according to their respective dimensions, and then the mean absolute error or smoothed absolute error between the predicted and actual actions is calculated. Discrete components such as gripper opening and closing can use separate classification losses. To achieve joint state-action modeling, an auxiliary prediction head can be derived from the fusion features and used only during the training phase. The target object displacement and contact probability predicted by the teacher model are used as auxiliary supervision; this auxiliary prediction head is removed during inference. During online runtime, the action sequence decoding layer directly outputs an action sequence. The controller executes the first action or the first few actions of this sequence, and then updates the input and generates the next action sequence based on newly acquired environmental observations. This update uses actual observations to form a closed-loop feedback. The decoding process does not use the predicted future environmental state as the input for the next moment, nor does it select actions through candidate future state search. The scale recovery of the actions and the limiting of joints, velocities, and contact forces are executed by the corresponding control interface according to the robot configuration.
[0036] Explained, the training process of the pre-trained implicit world representation model includes two stages: the first stage is the training stage of the physically consistent world simulation model, which trains the initial physically consistent world simulation model by acquiring multimodal interaction data of the robot's historical tasks, enabling the physically consistent world simulation model to learn the environmental dynamics, contact evolution, and task execution laws in the robot's operation tasks, and to predict the environmental state changes, target object motion results, and robot-environment interaction results in the future time domain; the second stage is the knowledge distillation stage, which transfers the capabilities of the pre-trained physically consistent world simulation model to the initial implicit world representation model through knowledge distillation, so that the latent space world state representation has both environmental dynamic understanding and action generation capabilities.
[0037] For example, the physically consistent world simulation model learns the dynamic coupling law between robot actions and environmental state changes by jointly modeling the spatiotemporal correlation between historical environmental state sequences, robot state sequences, and action sequences. Based on the current world state, it predicts the environmental state changes, target object motion results, and robot-environment interaction results at multiple future moments. Simultaneously, it learns the laws of environmental dynamics, target object motion, robot action execution, and dynamic interaction between the robot and the environment, and uses the future state prediction results to assist in the dynamic understanding of the environment and the action generation process.
[0038] It should be noted that the physically consistent world simulation model serves as the teacher model, and the implicit world representation model serves as the student model. Through knowledge distillation, the laws of environmental dynamics, contact evolution, spatiotemporal interaction, and task-related visual changes are transferred to the implicit space world state representation, thereby realizing the knowledge transfer from the physically consistent world simulation model to the robot world action model.
[0039] For example, the physically consistent world simulation model can employ a network composed of a multimodal spatiotemporal encoder and a parallel future state decoder, including a multimodal input layer, a temporal encoding layer, an entity relationship interaction layer, and a future state prediction layer. The multimodal input layer receives the environmental image, depth, robot state, executed actions, task description, and interaction information up to the current moment. These are encoded, aligned according to timestamps, and then a time identifier, modality identifier, and valid data mask are added. Missing modalities are identified using a mask, and their padding values are not used as actual measured values.
[0040] Specifically, the temporal coding layer employs a temporal attention network with causal masks to extract the correlation between historical observations, states, and executed actions. The entity relationship interaction layer uses the features corresponding to the robot end effector, target object, and related environmental entities as nodes, and propagates the position, contact, and interaction relationship information between entities through graph attention or pairwise relationship transformation. The future state prediction layer uses query vectors corresponding to each future prediction time to read the above features, and outputs the geometric state change, target pose change, contact or slip probability, and available force and torque prediction quantities at the corresponding time through the environmental state change prediction head, target object motion prediction head, and interaction result prediction head.
[0041] During training, each synchronous interactive trajectory is divided into a historical input window and a future supervision window. The actual observations, target object states, and interaction measurements in the future window are only used to construct supervision labels and are not considered historical input. For continuous quantities, normalized regression errors are calculated; for discrete events such as contact and slippage, classification errors are calculated. The future state prediction loss is obtained by weighted summation of items with effective supervision, and is then optimized together with the trajectory consistency loss and relationship consistency loss.
[0042] in, The total training loss of the teacher model. Predicting losses for future states, For trajectory consistency loss, For relationship consistency loss, and Non-negative weights are used; errors are calculated after normalization based on the number of valid samples and the scale of the corresponding physical quantities. The specific construction of the trajectory and relational constraints is described in the following implementation method. After training, the teacher model parameters are fixed, and its spatiotemporal features, entity relation features, and future state prediction results are extracted for distillation. The future state prediction layer of the teacher model runs during offline training; it does not need to be called during online action generation.
[0043] For example, knowledge transfer is achieved sequentially through distillation from the physically consistent world simulation model to the implicit world representation model, and conditional transfer from the implicit world representation model to the robot's action model. First, a pre-trained teacher model is fixed. For the same interaction sample, the teacher model reads multimodal historical information up to the current moment, while the student model reads the current environmental observations, the current robot state, and the task description. When historical caching is enabled, the student model only uses the cached information up to the current moment. The teacher model's output serves as offline supervision and is not required input for the student model's online execution.
[0044] In latent space distillation, the outputs of the teacher model's entity relationship interaction layer or future query layer are selected as teacher model features, and the outputs of the student model's conditional interaction layer are selected as student model features. The two are aligned according to semantic correspondences of the same entities, image regions, or preset queries. A projection layer is used to transform them into feature vectors of the same number and dimension, and then normalization is performed vector by vector, denoted as follows: and The characteristic distillation loss is constructed as follows:
[0045] in, and These represent the projected and normalized features of the student and teacher models, respectively. A small positive constant is added to the normalization denominator to avoid dividing the zero vector by zero. `sg` indicates stopping gradient propagation, and `mean` indicates averaging over the effective aligned vectors. This represents the Euclidean norm. The projection at the teacher model end can be an existing projection of the teacher model or a fixed projection determined before distillation; during distillation, the teacher model and its projection at the teacher model end are fixed, and only the student model and its projection at the student model end are updated to prevent simultaneous contraction at both ends from causing the supervised target to fail.
[0046] When the teacher and student models have alignable spatial attention, their attention maps are resampled to the same spatial grid and normalized to non-negative weight distributions with a sum of one. The mean squared error of both models is used as the attention distillation loss. For attention layers whose structure or semantics do not correspond, element-by-element comparison is not mandatory; feature distillation can be used instead. Optional output space distillation uses a training-specific prediction head to regress continuous outputs such as target displacement and contact force from the student model representation, and fits the contact and slip probability outputs of the teacher model. The sum of the continuous regression error and the probability cross-entropy is denoted as... The training-specific prediction head is removed during online runtime.
[0047] The student model can also employ editing supervision independent of the teacher model's predictions. Specifically, within the same task trajectory, the current image and subsequent task progress images are selected to form an image editing pair. A fixed visual encoder is used to obtain the feature difference between the two in corresponding regions. This feature difference is predicted by the auxiliary projection of the student model's features, and the regression error is calculated within the effective corresponding region, denoted as . This supervision comes directly from actual observations and does not require the reconstruction of the complete image; when changes in viewpoint cause regions to become misaligned, the loss in the relevant regions is masked. The training objective of the student model can be expressed as:
[0048] in, For student model training loss, , , and For each non-negative weight, Greater than zero; disabled auxiliary items will have their corresponding weights reset to zero. The above embodiments primarily use latent space feature distillation, with output space distillation and editing supervision as optional auxiliary methods. Simultaneous use of image reconstruction loss and contrast loss is not required. Weights are determined based on the scale of each loss and the performance of downstream actions on independent validation data, without using unknown future observations as inference input.
[0049] After training the student model, the student model is fixed, and its latent world state representation, along with observations, robot states, and task descriptions, is used to jointly train the robot world action model. The action prediction loss is supervised by the actual robot action sequence. The state change knowledge provided by the teacher model is thus transferred to the action generation network through the student model representation. When joint fine-tuning is needed, the teacher model can be kept fixed, and the student model loss and action prediction loss can be weighted and summed to update both the student model and the action model; at this time, the action prediction loss can be backpropagated to the student model along the latent world state representation. After training, only the representation calculation branch of the student model and the robot world action model are retained.
[0050] In summary, the robot action sequence generation method based on physical knowledge distillation of this invention provides a comprehensive and accurate multimodal input foundation for subsequent latent space world state representation generation and robot action sequence generation by acquiring current environmental observation information, current robot state information, and task description information. It generates latent space world state representations based on a pre-trained latent world representation model, allowing the robot to focus only on the state change processes of task-related and key interaction areas, avoiding the computational overhead caused by a large amount of task-irrelevant visual information during future video generation. It generates robot action sequences based on a pre-trained robot world action model, achieving joint modeling between the world state change rules and the robot action generation rules. Through training a physically consistent world simulation model and capability transfer via knowledge distillation, it achieves knowledge transfer from the physical world model to the action generation model. The synergistic effect of these steps enables the direct generation of robot action sequences without future video generation, future state prediction, or future imagination during the testing phase, achieving real-time closed-loop control in complex robot operation tasks.
[0051] In one possible implementation, the multimodal interaction data includes: environmental observation information, environmental depth information, robot state information, robot motion information, task description information, and interaction information between the robot and the environment; wherein, the interaction information between the robot and the environment includes: contact information, force information, torque information, slippage information, friction state information, target object state information, and task stage information; the future state prediction result includes the environmental state change results, target object motion results, and interaction results between the robot and the environment at several future time points.
[0052] Specifically, the multimodal interaction data refers to the multimodal interaction data generated by the robot during task execution. Among these, the environmental observation information refers to environmental image information collected by the robot through a visual sensor; the environmental depth information refers to environmental depth information collected by the robot through a depth sensor; the robot state information refers to the robot's own body state information, including the angles, speeds, and torques of each joint; the robot motion information refers to the motion information executed by the robot, including control commands for each joint; the task description information refers to information describing the task the robot currently needs to perform; and the robot-environment interaction information refers to the interaction information generated between the robot and the environment during task execution, including one or more of the following: contact information, force information, torque information, slippage information, friction state information, target object state information, and task stage information. Wherein, the contact information refers to the contact state information between the robot end effector and the target object; the force information refers to the contact force information between the robot end effector and the target object; the torque information refers to the contact torque information between the robot end effector and the target object; the slip information refers to the information on relative slippage between the robot end effector and the target object; the friction state information refers to the friction state information between the robot end effector and the target object; the target object state information refers to the position, attitude, motion state, and other information of the target object; and the task stage information refers to the task stage information currently in which the robot is located.
[0053] Interpretive, the future state prediction results include environmental state changes at several future moments, target object motion results, and robot-environment interaction results, thereby providing comprehensive supervision signals for the training of the physically consistent world simulation model.
[0054] Optionally, the multimodal interaction data is collected in one or more states during robot autonomous execution, remote teaching, human-machine shared control, teaching learning, reinforcement learning exploration, and autonomous interaction in a real environment, and then processed through time synchronization.
[0055] Explanatoryly, the multimodal interaction data is processed through time synchronization to form a unified data sequence in order to ensure the time alignment between different modal data, so that the physically consistent world simulation model can accurately learn the dynamic coupling relationship between robot actions and changes in environmental state.
[0056] Explained in this embodiment, by acquiring multimodal interaction data including environmental observation information, environmental depth information, robot state information, robot action information, task description information, and interaction information between the robot and the environment, and by processing the data in a time-synchronized manner to form a unified data sequence, a comprehensive modeling of the multimodal interaction process in the robot operation task is achieved, providing a rich and accurate data foundation for training the physically consistent world simulation model.
[0057] In one possible implementation, during the training of the initial physically consistent world simulation model, trajectory consistency constraints and relation consistency constraints are used for optimization; wherein, the trajectory consistency constraints are used to constrain the continuity and geometric consistency of the motion trajectories of the target object, the robot end effector, and environmental entities between consecutive moments; the relation consistency constraints are used to constrain the consistency of contact relationships, spatial topological relationships, and interactive semantic relationships between the robot, the target object, and environmental entities.
[0058] Specifically, the physically consistent world simulation model introduces trajectory consistency constraints and relational consistency constraints during training to improve the physical rationality, dynamic stability, and task executability of the world simulation results. The trajectory consistency constraint constrains the continuity and geometric consistency of the motion trajectories of the target object, robot end effector, and environmental entities across consecutive time points. This means the motion trajectories of the target object, robot end effector, and environmental entities should remain continuous and meet geometric consistency requirements, avoiding abrupt changes in trajectory or contradictory geometric relationships. The relational consistency constraint constrains the consistency of contact relationships, spatial topological relationships, and interactive semantic relationships among the robot, target object, and environmental entities. This means the contact relationships, spatial topological relationships, and interactive semantic relationships among the robot, target object, and environmental entities should remain consistent throughout temporal evolution, avoiding abrupt changes in contact relationships or contradictory spatial topological relationships. It should be noted that the trajectory consistency constraints and relational consistency constraints jointly optimize the world model training process, thereby achieving unified modeling of the physical evolution process and the interaction relationship evolution process during robot operation.
[0059] Explanatoryly, due to the complex physical interactions such as collisions, contact, sliding, insertion, and assembly commonly involved in robot operation tasks, traditional visual world models struggle to guarantee the physical plausibility and task executability of prediction results. Therefore, trajectory consistency constraints and relation consistency constraints are needed to constrain the training process of the world model, enabling the physically consistent world simulation model to learn dynamic evolution laws that conform to physical principles. In this embodiment, by employing trajectory consistency constraints and relation consistency constraints to jointly optimize the world model training process, effective constraints are achieved on the continuity of the target object's motion trajectory, the consistency of the contact relationship between the robot and the target object, and the consistency of the spatiotemporal relationships among multiple interacting entities. This improves the physical plausibility, dynamic stability, and task executability of the world simulation results.
[0060] In one possible implementation, the trajectory consistency constraint is constructed based on one or more of pixel-level trajectory supervision information, feature-level trajectory supervision information, or latent space trajectory supervision information; the relationship consistency constraint is constructed based on one or more of object relationship graphs, interaction relationship graphs, or scene relationship graphs.
[0061] Specifically, trajectory consistency constraints first establish cross-frame entity correspondences, then calculate the error between the predicted trajectory and the measured trajectory at valid correspondence locations. Pixel-level trajectory supervision can be obtained through target segmentation, keypoint tracking, or optical flow tracking. Stable identifiers are assigned to target objects, robot end effectors, and related environmental entities. The coordinates of the same keypoint in consecutive images are recorded, and valid masks are determined through forward and backward tracking consistency and occlusion detection. Combining camera intrinsic parameters, depth, and camera extrinsic parameters, pixel coordinates are back-projected to a unified robot base coordinate system; when depth is invalid, the corresponding error is calculated only in pixel space, without directly adding the pixel distance to the 3D distance.
[0062] Feature-level trajectory supervision samples features from a fixed visual encoder at the corresponding positions mentioned above, obtaining a feature sequence of the same entity or keypoint changing over time. Latent space trajectory supervision inputs the observed entity features into a fixed reference encoder to obtain the corresponding latent variable sequence, which is then compared with the latent variable sequence output by the teacher model's prediction layer. The reference encoder is fixed or uses an independent update strategy during this round of training, and the gradient is stopped at the supervision end to avoid the prediction and the target collapsing together. The three types of supervision can be used separately or combined in a weighted manner after being normalized separately.
[0063] Specifically, taking location trajectory as an example, record and Let be the predicted position and the measured position of the i-th tracking point at the k-th prediction time, respectively. For a valid mask with values of zero or one, This represents the number of valid points; under the same coordinate system and the same length scale, the position term can be written as:
[0064] in, This means averaging only over points where the effective mask is one, i.e., the sum of the effective errors divided by . ;when When the value is zero, this term is not included in the current batch training. The predicted velocity and the measured velocity are obtained by dividing the difference between adjacent time points by the sampling interval, and the same mask regression loss is calculated on the difference between the two, denoted as . This velocity term matches actual motion changes and does not force all velocities to approach zero. For valid point pairs on the same rigid object, the predicted distance between points can be compared with the measured distance to calculate the geometric consistency loss. This does not apply to point pairs across hinges or deformable parts. The trajectory consistency constraint can be expressed as:
[0065] in, and They are respectively and Non-negative weights are applied. Position, velocity, and distance errors are normalized using their respective scales. When using feature-level or latent space trajectory supervision, position variables are replaced with the normalized corresponding features, and cross-frame feature increment errors are used; latent space distance is not interpreted as true physical length. When collisions, contact establishment, or contact dissolution occur, the changes are supervised according to the measured trajectory and event mask to avoid eliminating true physical events with unconditional smoothing.
[0066] Specifically, relationship consistency constraints employ a relationship graph with stable entity identifiers. The object relationship graph uses robot end effectors, target objects, and related environmental entities as nodes, and relationships such as relative position, relative orientation, inclusion, or support as edges. The interaction relationship graph uses the same nodes and uses interaction states such as contact, grasping, pushing, inserting, or sliding as edges. The scene relationship graph further incorporates entities such as workbenches, sockets, and obstacles, along with their spatial topological relationships. These graphs can share nodes, and each edge stores the relationship type, relationship value, and valid supervision identifier.
[0067] Relationship labels are constructed from synchronized data: relative pose and spatial proximity relationships are obtained from depth, target pose estimation, and robot forward kinematics; contact relationships are determined by integrating surface spacing, force, or tactile signals; sliding relationships are determined based on relative tangential motion or tactile changes in the contact state; semantic relationships such as grasping and insertion are labeled in conjunction with task stages, target pose, and actual execution records. For mechanical relationships that cannot be reliably judged visually alone, sensors or verified labels are used; edges with insufficient evidence are marked as unknown and masked from that supervision.
[0068] The teacher model predicts the relationship graph at future time points based on entity features and aligns it with the supervised graph according to entity identifiers. Binary cross-entropy is calculated for each valid binary relationship, and normalized regression error is calculated for continuous edge attributes such as relative displacement and contact force. The average of these factors yields the final result. For the same edge that is valid at both time points, the change in predicted relation is compared with the change in labeled relation to obtain... The relationship consistency loss can be expressed as:
[0069] in, The weights are non-negative. The constraint is that the evolution of relationships must be consistent with the measured changes, but it is not required that all relationships remain unchanged over time. For real-world events such as contact establishment, dissolution, and capture / transfer, predicted relationships are allowed to change according to the annotations. When there are no valid edges or valid temporal edge pairs, the corresponding loss is not included in the current batch training. Trajectory loss and relationship loss are backpropagated together through the shared encoding layer of the teacher model, making geometric motion prediction and interaction relationship prediction mutually constrained.
[0070] Interpretively, by employing one or more of pixel-level trajectory supervision information, feature-level trajectory supervision information, or latent space trajectory supervision information to construct trajectory consistency constraints, and by employing one or more of object relationship graphs, interaction relationship graphs, or scene relationship graphs to construct relationship consistency constraints, it is possible to achieve unified modeling of the physical evolution process and the interaction relationship evolution process during robot operation, thereby improving the prediction accuracy and physical rationality of the physically consistent world simulation model, and improving the physical rationality and task executability of the world simulation results.
[0071] In one possible implementation, the initial implicit world representation model is constructed using an image editing-based implicit world representation mechanism; when generating the latent space world state representation, the implicit world representation model assigns a first weight to the features of task-related regions, interaction-critical regions, and contact-sensitive regions, and assigns a second weight to the features of background regions and task-irrelevant visual information regions, wherein the first weight is greater than the second weight.
[0072] Specifically, the initial implicit world representation model is constructed using an image editing-based implicit world representation mechanism, which builds a latent space world state representation by learning the local change relationships between the current environmental state and the target environmental state. It should be noted that the task-related region refers to the region related to the task the robot currently needs to perform; the interaction-critical region refers to the critical region where the robot interacts with the target object during operation; the contact-sensitive region refers to the contact-sensitive region during robot operation; the background region refers to the background region unrelated to the task the robot currently needs to perform; and the task-independent visual information region refers to the visual information region unrelated to the task the robot currently needs to perform.
[0073] Specifically, for region recognition, the task description is semantically encoded to identify the operation object, operation part, and task target, and this information is matched with candidate regions of objects in the image. Candidate regions of objects are obtained by detection or segmentation networks. During matching, object semantics, spatial location, and the current task stage are considered simultaneously to obtain task-related regions. For example, in a plug-in assembly task, the areas where the plug and target socket are located are identified as task-related regions; when there is not a unique candidate target, multiple candidates and their confidence scores can be retained as input to subsequent conditional interaction layers. Based on the current robot state, the end effector pose is obtained through forward kinematics. Combined with camera calibration, the end effector position is projected onto the image, and the grasping point, operation part, or operable area of the target object is used to determine key interaction regions. The operation part can be obtained by a segmentation or key point detection branch trained with annotations. Based on the surface distance, depth boundary, assembly gap, and currently available force or tactile information between the target object and the end effector or mating part, contact-sensitive regions are further determined; for example, the plug end, socket edge, and their neighborhood are contact-sensitive regions. The union of the above three types of regions is mapped onto a visual feature grid to form a region mask corresponding to each image block. Regions located outside the union of the given set and not identified as operational objects, related obstacles, or potential interactive objects can be considered as background regions or task-independent visual information regions. Task relevance is determined based on the current task conditions; not all static regions or screen backgrounds are considered irrelevant. Obstacle regions that may affect end-effector motion should be retained as relevant regions. For regions with uncertain recognition results, their confidence level can be retained to form a soft mask. Let the region mask of the i-th image patch be denoted as... The value ranges from 0 to 1; the first weight is... The second weight is ,satisfy > and >0. The feature weights of image patches can be calculated using the following formula:
[0074] When using a binary mask, the weight of the relevant region is the first weight, and the weight of the remaining regions is the second weight; when using a soft mask, interpolation is performed between the two weights based on the region's confidence level. The weighted visual features retain their original location identifiers and are input into the conditional interaction layer along with the robot's state features and task features. This layer extracts feature changes under task conditions through cross-attention, and the latent space projection layer then converges them into a latent space world state representation. Region detection, task association, and feature extraction all utilize currently available information; subsequent task progress images used for training serve only as supervision targets and do not participate in online region recognition.
[0075] Explained, since the state changes of task-related regions, key interaction regions, and contact-sensitive regions are crucial for action generation in robot operation tasks, while the state changes of background regions and task-irrelevant visual information regions have a smaller impact on action generation, assigning a larger first weight to the features of task-related regions, key interaction regions, and contact-sensitive regions, and a smaller second weight to the features of background regions and task-irrelevant visual information regions, the implicit world representation model can focus on the state change processes of task-related regions and key interaction regions, reducing the impact of background regions and task-irrelevant visual information on the action generation process. This achieves a focus on task-related changes, avoids the computational overhead caused by a large amount of task-irrelevant visual information in future video generation, improves model training efficiency and online inference efficiency, and thus improves the robustness and generalization ability in the action generation process.
[0076] In one possible implementation, the latent space world state representation includes one or more of latent space dynamic features, attention features, cross-modal features, historical state memory features, contact state features, and task condition features; the robot action sequence includes one or more of joint space action sequences, task space action sequences, end-effector pose action sequences, force control action sequences, impedance control action sequences, admittance control action sequences, or hybrid control action sequences.
[0077] Specifically, the latent space dynamic features represent the required change trend of the current state under task conditions. These features can be obtained by linear projection of the difference between the visual features processed by the conditional interaction layer and the visual features before processing, or by reading the output of the conditional interaction layer using a dedicated dynamic query vector. These features encode information about the relative motion, end-effector adjustment, and contact evolution of the target object. Their values are learned latent space quantities and are not directly equivalent to actual displacement or force.
[0078] Specifically, the attention features represent the distribution of the model's attention to different image regions, entities, or task terms. After calculating the similarity between the query and key of the specified attention head in the conditional interaction layer and normalizing it, the resulting attention weight matrix is read and averaged head-by-head or compressed by projection to obtain the attention features. The cross-modal features represent the joint relationship between visual, robot state, and task information. Cross-attention is calculated using visual features as the query and robot state and task features as keys and values, and then obtained through residual connections and feedforward transformations. Alternatively, the query side and key-value side can be exchanged for bidirectional fusion.
[0079] Specifically, the historical state memory features represent the states and interactions that have occurred up to the current moment. World features generated from each actual observation can be written into a fixed-length cache, and then converged using temporal attention with causal masks, or updated via a gated recurrent unit. The cache is updated only by real historical inputs, initialized at the start of the task, and the oldest features are deleted when the cache exceeds its capacity. Predicted future states are not written to the cache. Historical state memory features are not required when historical branches are not enabled.
[0080] Specifically, the contact state features represent information such as the contact position, contact probability, contact force direction, or slippage tendency between the end effector, target object, and environment. When there is force or tactile input, the corresponding measurements are fused with visual geometric features and obtained through a contact encoding branch. When there is no corresponding sensor input, contact estimation features are formed from the current observation and the robot's state. This branch is supervised during training using offline contact labels, and the output is an estimated value. Training and deployment should employ matched input modalities or modality missing masks. The task condition features are obtained by encoding natural language instructions or structured targets by the task encoding layer, and may include the operation object, target position, action semantics, and task stage conditions. The task stage is estimated from existing execution records or current observations, without using actual future completion results as online input.
[0081] The above features are used in one or more of the selected model structures. After being adapted and projected to a unified dimension, they are used to form the latent space world state representation by word concatenation or query aggregation, and type identifiers are retained so that the action model can distinguish various information sources.
[0082] Optionally, the latent space world state representation may further include one or more of the following: intermediate layer features generated during image editing, attention mapping features, key-value cache features, or latent variable features generated during the diffusion process.
[0083] Specifically, the intermediate layer features are the activation tensors generated by the convolutional blocks, transform blocks, or conditional interaction blocks of the image editing network when processing the current image and task conditions. One or more layers can be pre-specified, and their outputs, after spatial pooling, scale alignment, and linear projection, can be used as components of the world state representation. When multiple layers are selected, they are projected separately and then fused to avoid directly stitching together tensors of inconsistent sizes. This feature is read during the network's forward computation, without requiring the edited image to be output first.
[0084] Specifically, the attention mapping feature is the normalized correlation weight between the query vector and the key vector of the specified attention layer. Let Q be the query matrix, K be the key matrix, and ... If the dimension is the key vector, then the attention mapping A can be represented as:
[0085] Where T represents matrix transpose, softmax represents normalization along the dimension of the key; in self-attention, the query and the key come from the same modality, while in cross-attention, they come from different modalities.
[0086] After reading A, attention maps can be aggregated and projected according to attention heads, image regions, or task terms to form fixed-dimensional features. When image patches in different layers are divided differently, spatial correspondence should be completed before comparison or fusion of attention maps.
[0087] Specifically, the key-value cache features are the key and value matrices calculated by the attention layer on the encoded input. The keys and values are obtained by multiplying the input features by the corresponding projection matrices, and can be read in the same forward computation and used for world state representation after pooling or projection. In structures employing a caching mechanism, keys and values corresponding to task conditions that remain unchanged within the same control cycle can be reused. When observations or task changes occur across control cycles, the affected caches should be refreshed. Bidirectional attention networks without a causal caching structure do not directly reuse expired visual keys.
[0088] Optionally, when the image editing backbone adopts a diffusion structure, the latent variable features in the diffusion process are latent variables processed by the structure at a specified noise level or intermediate activations of the denoising network. As a feature extraction implementation, the current image can be encoded into the latent space, noise generated by a fixed random seed can be added at a preset noise level, and this noise, along with the current image conditions, robot state, and task conditions, can be input into the denoising backbone. Only a preset feature extraction layer or a finite number of denoising calculations can be performed. The process stops after reading the latent variable features, without image decoding.
[0089] When using the above-described diffusion feature extraction path, the same noise level, layer selection, and truncation method should be used in the student model distillation training to ensure that the read features are supervised by the teacher model and that their correlation with robot actions is established through subsequent action training. The diffusion noise level is an internal feature calculation parameter and does not correspond to future physical moments of the robot; latent variables at each noise level are not considered as continuous future environmental states. The diffusion backbone is optional; non-diffusion conditional image editing coding networks can directly read intermediate layer or attention features to construct a latent space world state representation.
[0090] In one possible implementation, the pre-trained robot world action model is obtained by: constructing an initial robot world action model; generating a latent space world state representation corresponding to the multimodal interaction data based on the multimodal interaction data and the pre-trained latent world representation model; and training the initial robot world action model based on the multimodal interaction data and the latent space world state representation corresponding to the multimodal interaction data to obtain the pre-trained robot world action model.
[0091] Specifically, an initial robot world action model is constructed, taking current environmental observation information, current robot state information, task description information, and latent world state representation as inputs, and robot action sequence as output. Then, based on multimodal interaction data and a pre-trained latent world representation model, a latent world state representation corresponding to the multimodal interaction data is generated. When generating the latent world state representation, the environmental observation information, robot state information, and task description information from the multimodal interaction data are input into the pre-trained latent world representation model, which outputs the latent world state representation corresponding to the multimodal interaction data. Since the pre-trained latent world representation model is obtained by transferring the capabilities of a pre-trained physically consistent world simulation model to the initial latent world representation model through knowledge distillation, the output latent world state representation already possesses environmental dynamics laws, contact evolution laws, spatiotemporal interaction laws, and task-related visual change laws. Based on the multimodal interaction data and the latent world state representation corresponding to the multimodal interaction data, the initial robot world action model is trained to obtain a pre-trained robot world action model.
[0092] During training, the environmental observation information, robot state information, task description information, and generated latent space world state representation in the multimodal interaction data are used as inputs. The robot action sequence composed of robot action information in the multimodal interaction data is used as a supervision label. The loss between the predicted robot action sequence output by the initial robot world action model and the supervision label is calculated. The parameters of the initial robot world action model are updated based on the loss until the preset training convergence condition is met, and a pre-trained robot world action model is obtained.
[0093] Interpretively, since the latent space world state representation has inherited the capabilities of the pre-trained physically consistent world simulation model through knowledge distillation, it can provide world context information for action generation, including environmental dynamics, contact evolution, and task-related visual changes, when training the initial robot world action model.
[0094] Explained, the robot world action model employs a joint state-action modeling approach. By uniformly encoding the current environmental state, the latent space world state representation, and task description information, it learns the causal relationship between the evolution of environmental states and the generation of robot actions. During the training phase, the robot world action model is supervised through the future state information predicted by the physically consistent world simulation model and the task execution results, thereby learning the dynamic changes in latent space and the generation of actions. During the inference phase, only the current environmental observation information, the current robot state information, and the task description information need to be input to directly generate the robot's future action sequence, without needing to perform future video generation, future state scrolling prediction, model predictive control, or future imagination processes during the testing phase.
[0095] In one possible implementation, the robot action sequence generation method based on physical knowledge distillation further includes: predicting collision events, abnormal contact events, slip events, assembly failure events, grasping failure events, or task deviation events based on a preset risk model according to the latent space world state representation, and generating compensation actions, recovery actions, search actions, replanning actions, or safety actions in combination with current environmental observation information.
[0096] Explaining this, since robot operation tasks commonly involve complex physical interactions such as collisions, contact, sliding, insertion / removal, and assembly, robots may encounter various risk events during task execution. Therefore, it is necessary to predict potential risk events through risk models and generate corresponding compensation actions, recovery actions, search actions, replanning actions, or safety actions in advance to improve the robustness and success rate of robots performing complex contact operations. By predicting collision events, abnormal contact events, sliding events, assembly failure events, grasping failure events, or task deviation events based on a pre-set risk model according to the latent space world state representation, and combining this with current environmental observation information, compensation actions, recovery actions, search actions, replanning actions, or safety actions are generated. This enables the early prediction and proactive response to risk events during robot operation, improving the robot's robustness and success rate in complex contact operations.
[0097] For example, the risk model can employ a multi-label risk prediction network, including a world feature aggregation layer, a risk feature transformation layer, and multiple event prediction heads. The world feature aggregation layer receives the latent space world state representation and aggregates it through average pooling or learnable queries to obtain a risk input vector. The risk feature transformation layer forms shared features through fully connected layers, nonlinear activation, and normalization. Each event prediction head is connected to the shared features and provides the probability of collision, abnormal contact, slippage, assembly failure, grasping failure, or task deviation occurring within the future preset risk time domain through an independent Sigmoid probability output layer. Since multiple risks may occur simultaneously, the probabilities of each event are predicted independently, and their sum is not required to be 1.
[0098] Specifically, training data is constructed from actual interaction trajectories based on the current moment and subsequent risk windows. Collision labels are constructed from unexpected entity contact accompanied by collision measurements or verified collision records; abnormal contact labels are constructed from contact forces or torques outside the task's permissible range; and slip labels are constructed from continuous relative sliding during contact. Assembly or grasping failure labels are obtained from the target position, clamping and holding status, and execution results at the end of the corresponding operation phase; task deviation labels are obtained by comparing the target constraints of the current phase with the actual execution status. Normal expected contact is not marked as a collision, and tasks that are not yet completed and for which subsequent records are insufficient are not directly marked as failures.
[0099] Specifically, thresholds and durations are set based on task requirements, sensor calibration, and the robot's permissible working range. Samples with unclear labels are marked as invalid and masked. The current latent space world state is used as network input, and the occurrence of various events within the risk window is used as multi-label supervision. Weighted binary cross-entropy is calculated by event category and averaged over valid samples; category weights are determined based on event frequency in the training data to mitigate the impact of insufficient rare risk samples. Training and validation are divided by complete task trajectories to avoid adjacent segments of the same trajectory belonging to two groups; subsequently, trigger thresholds for each event are selected on independent validation data. Actual future events are only used for offline label construction.
[0100] During online operation, the risk model directly outputs event probabilities from the current latent space world state, without generating future images or rolling state sequences. For example, the event probabilities are compared with their respective thresholds, and the processing method is selected based on current environmental observations and the current robot state: if there is a risk of collision or abnormal contact, subsequent action segments can be stopped, movement speed reduced, or a safe action generated along a known separation direction; if there is a risk of slippage, clamping compensation or re-grasping actions can be generated within the allowable clamping force range; if there is a risk of assembly failure, small-range alignment, search, or exit recovery actions can be generated based on the relative poses of the current sockets and plugs; if there is a risk of task deviation, the action sequence is regenerated based on current observations and task conditions.
[0101] For example, the response action can be generated by a preset parameterized action template or a trained action generation module. The template parameters are determined by the currently measured relative pose, contact state, and robot constraints. For instance, when an abnormal contact risk is detected during insertion assembly, insertion is first stopped, a limited exit action is generated based on the actual contact direction and the known insertion axis, the plug and socket are re-identified, and then an alignment sequence is generated. When multiple risks are triggered simultaneously, safety actions such as stopping or exiting are executed first, followed by compensation and recovery; after completion, actual observations are re-acquired and the risks are updated. The replanning refers to regenerating actions based on the new observations.
[0102] This invention's method can be applied to various complex task scenarios, such as plug-in assembly, tool operation, door handle rotation, button operation, compliant assembly, complex contact operations, humanoid robot dual-arm collaborative operation, and industrial manufacturing. Compared with existing video-generated world models and planning methods based on future imagination during the testing phase, this invention's method eliminates the need to generate future video sequences, perform future state rolling predictions, or undergo future imagination during the testing phase. This significantly reduces model computational complexity and improves the robot's online reasoning efficiency, real-time control performance, and generalization ability and robustness in complex task scenarios.
[0103] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.
[0104] See Figure 2 In another embodiment of the present invention, a robot motion sequence generation system based on physical knowledge distillation is provided, which can be used to implement the above-mentioned robot motion sequence generation method based on physical knowledge distillation. Specifically, the robot motion sequence generation system based on physical knowledge distillation includes a data acquisition module, an implicit world representation module, and an motion generation module.
[0105] The data acquisition module is used to acquire current environmental observation information, current robot state information, and task description information; the implicit world representation module is used to generate a latent world state representation based on a pre-trained implicit world representation model, according to the current environmental observation information, current robot state information, and task description information; the latent world state representation is used to represent the task-related change relationship between the current environment and the target environment; the action generation module is used to generate a robot action sequence based on a pre-trained robot world action model, according to the current environmental observation information, current robot state information, task description information, and latent world state representation. The pre-trained implicit world representation model is obtained through the following steps: constructing an initial physically consistent world simulation model and an initial implicit world representation model; wherein the physically consistent world simulation model takes multimodal interaction data as input and future state prediction results as output; the implicit world representation model takes environmental observation information, robot state information, and task description information as input and latent space world state representation as output; acquiring and training the initial physically consistent world simulation model based on the multimodal interaction data of the robot's historical task execution to obtain the pre-trained physically consistent world simulation model; and transferring the capabilities of the pre-trained physically consistent world simulation model to the initial implicit world representation model through knowledge distillation to obtain the pre-trained implicit world representation model.
[0106] All relevant content of each step involved in the aforementioned embodiments of the robot motion sequence generation method based on physical knowledge distillation can be referenced to the functional description of the corresponding functional module of the robot motion sequence generation system based on physical knowledge distillation in the embodiments of the present invention, and will not be repeated here.
[0107] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0108] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a robot action sequence generation method based on physical knowledge distillation.
[0109] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the robot action sequence generation method based on physical knowledge distillation in the above embodiments.
[0110] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0111] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for generating robot motion sequences based on physical knowledge distillation, characterized in that, include: Acquire current environmental observation information, current robot status information, and task description information; Based on current environmental observation information, current robot state information, and task description information, a latent world state representation is generated using a pre-trained latent world representation model; the latent world state representation is used to represent the task-related change relationship between the current environment and the target environment. Based on current environmental observation information, current robot state information, task description information, and latent space world state representation, a robot action sequence is generated based on a pre-trained robot world action model. The pre-trained implicit world representation model is obtained in the following way: An initial physically consistent world simulation model and an initial implicit world representation model are constructed. The physically consistent world simulation model takes multimodal interaction data as input and future state prediction results as output. The implicit world representation model takes environmental observation information, robot state information, and task description information as input and latent space world state representation as output. Acquire and train an initial physical consistent world simulation model based on the multimodal interaction data of the robot's historical tasks, and obtain a pre-trained physical consistent world simulation model; The capabilities of a pre-trained physically consistent world simulation model are transferred to an initial implicit world representation model through knowledge distillation, resulting in a pre-trained implicit world representation model.
2. The robot motion sequence generation method based on physical knowledge distillation according to claim 1, characterized in that, The multimodal interaction data includes: environmental observation information, environmental depth information, robot state information, robot motion information, task description information, and interaction information between the robot and the environment; wherein, the interaction information between the robot and the environment includes: contact information, force information, torque information, slippage information, friction state information, target object state information, and task stage information. The future state prediction results include the environmental state changes at several future time points, the motion of the target object, and the interaction between the robot and the environment.
3. The robot motion sequence generation method based on physical knowledge distillation according to claim 1, characterized in that, The multimodal interaction data is collected in one or more states during robot autonomous execution, remote teaching, human-machine shared control, teaching learning, reinforcement learning exploration, and autonomous interaction in the real environment, and is obtained through time synchronization processing.
4. The robot motion sequence generation method based on physical knowledge distillation according to claim 1, characterized in that, During the initial training of the physically consistent world simulation model, trajectory consistency constraints and relation consistency constraints are used for optimization. The trajectory consistency constraints are used to constrain the continuity and geometric consistency of the motion trajectories of the target object, the robot end effector, and environmental entities at consecutive moments. The relation consistency constraints are used to constrain the consistency of contact relationships, spatial topological relationships, and interactive semantic relationships among the robot, the target object, and environmental entities.
5. The robot motion sequence generation method based on physical knowledge distillation according to claim 4, characterized in that, The trajectory consistency constraint is constructed based on one or more of pixel-level trajectory supervision information, feature-level trajectory supervision information, or latent space trajectory supervision information; the relationship consistency constraint is constructed based on one or more of object relationship graphs, interaction relationship graphs, or scene relationship graphs.
6. The robot motion sequence generation method based on physical knowledge distillation according to claim 1, characterized in that, The initial implicit world representation model is constructed using an image editing-based implicit world representation mechanism. When generating the latent space world state representation, the implicit world representation model assigns a first weight to the features of task-related regions, key interaction regions, and contact-sensitive regions, and assigns a second weight to the features of background regions and task-irrelevant visual information regions, wherein the first weight is greater than the second weight.
7. The robot motion sequence generation method based on physical knowledge distillation according to claim 1, characterized in that, The latent space world state representation includes one or more of the following: latent space dynamic features, attention features, cross-modal features, historical state memory features, contact state features, and task condition features. The robot motion sequence includes one or more of the following: joint space motion sequence, task space motion sequence, end-effector pose motion sequence, force control motion sequence, impedance control motion sequence, admittance control motion sequence, or hybrid control motion sequence.
8. The method for generating robot motion sequences based on physical knowledge distillation according to claim 1, characterized in that, The pre-trained robot world action model is obtained in the following way: Construct an initial robot world action model; generate a latent space world state representation corresponding to the multimodal interaction data based on the multimodal interaction data and the pre-trained implicit world representation model; train the initial robot world action model based on the multimodal interaction data and the latent space world state representation corresponding to the multimodal interaction data to obtain a pre-trained robot world action model.
9. The method for generating robot motion sequences based on physical knowledge distillation according to claim 1, characterized in that, Also includes: Based on the hidden space world state representation, collision events, abnormal contact events, slip events, assembly failure events, grab failure events, or task deviation events are predicted based on a preset risk model. Combined with current environmental observation information, compensation actions, recovery actions, search actions, replanning actions, or safety actions are generated.
10. A robot motion sequence generation system based on physical knowledge distillation, characterized in that, include: The data acquisition module is used to acquire current environmental observation information, current robot status information, and task description information; The implicit world representation module is used to generate a latent world state representation based on a pre-trained implicit world representation model, according to current environmental observation information, current robot state information, and task description information. The latent world state representation is used to represent the task-related change relationship between the current environment and the target environment. The action generation module is used to generate robot action sequences based on a pre-trained robot world action model, according to current environmental observation information, current robot state information, task description information, and latent space world state representation. The pre-trained implicit world representation model is obtained in the following way: An initial physically consistent world simulation model and an initial implicit world representation model are constructed. The physically consistent world simulation model takes multimodal interaction data as input and future state prediction results as output. The implicit world representation model takes environmental observation information, robot state information, and task description information as input and latent space world state representation as output. Acquire and train an initial physical consistent world simulation model based on the multimodal interaction data of the robot's historical tasks, and obtain a pre-trained physical consistent world simulation model; The capabilities of a pre-trained physically consistent world simulation model are transferred to an initial implicit world representation model through knowledge distillation, resulting in a pre-trained implicit world representation model.
11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the robot motion sequence generation method based on physical knowledge distillation as described in any one of claims 1 to 9.
12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the robot motion sequence generation method based on physical knowledge distillation as described in any one of claims 1 to 9.
