A physical information guided embodied world model construction method and system

Abstract

The application provides a physical information guided embodied world model construction method and system, the method comprising: performing data processing on collected robot operation videos to obtain corresponding RGB frame sequences, depth map sequences and motion trajectories; encoding the RGB frame sequences and the depth map sequences to obtain RGB token sequences and depth token sequences, and converting the motion trajectories into action embedding vectors compatible with the RGB token sequences; performing autoregressive prediction according to the RGB token sequences, the depth token sequences and the action embedding vectors to obtain predicted RGB tokens and predicted depth tokens; and performing joint optimization according to the predicted RGB tokens and the predicted depth tokens to obtain a trained embodied world model. The application implicitly learns and internalizes the three-dimensional geometric constraints of the scene and the motion rules of the objects while generating visual content, and can effectively improve the physical authenticity of the embodied world model generated video.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and system for constructing an embodied world model guided by physical information. Background Technology

[0002] With the development of artificial intelligence technology, building world models that can simulate the dynamics of the real world for embodied agents has become a key technology. World models, by predicting future states based on current observations and given actions, can act as efficient environment simulators, addressing the problem of high cost and long cycle of real-world data acquisition and data scarcity in the field of robot learning. However, existing embodied world models have significant technical bottlenecks: current mainstream video generation models such as CogVideoX, Sora, and Wan mainly focus on the generation and prediction of video pixels (RGB), with their training objectives concentrated on optimizing visual fidelity. This results in models that, while capable of generating seemingly reasonable 2D images, have weak modeling capabilities for key physical properties such as 3D geometry and motion dynamics, leading to defects in the physical realism of the generated videos, such as poor spatial consistency and illogical motion. Summary of the Invention

[0003] This application provides a method and system for constructing a physically-guided embodied world model, which addresses the problem of deficiencies in the physical realism of videos generated by embodied world models in related technologies.

[0004] The first aspect of this application provides a method for constructing a physically-informed embodied world model, the method comprising: The collected robot operation videos are processed to obtain the corresponding RGB frame sequences, depth map sequences, and motion trajectories; The RGB frame sequence and the depth map sequence are encoded to obtain an RGB word sequence and a depth word sequence, and the motion trajectory is converted into an action embedding vector compatible with the RGB word sequence; Autoregressive prediction is performed based on the RGB word sequence, the deep word sequence, and the action embedding vector to obtain the predicted RGB word and the predicted deep word. The trained embodied world model is obtained by jointly optimizing the multi-task objective function based on the predicted RGB words and the predicted deep words.

[0005] Optionally, in the first implementation of the first aspect of this application, the step of processing the acquired robot operation video to obtain the corresponding RGB frame sequence, depth map sequence, and motion trajectory includes: Semantic segmentation is performed on the robot operation video to obtain short video clips of the robot operation; The short video segment is decomposed into an RGB frame sequence, and each RGB frame of the RGB frame sequence is processed to obtain a depth map sequence that is strictly aligned with the RGB frame sequence in time. An initial set of key points is determined in the first frame of the short video segment, and the pixel coordinates of the initial set of key points are tracked in all frames of the short video segment to obtain a set of key point trajectories. The total motion amplitude of each trajectory in the set of key point trajectories is determined, and the trajectories are filtered according to the total motion amplitude to obtain the trajectories of the top K key points with the largest motion amplitude, which are then determined as the motion trajectories of the target key points.

[0006] Optionally, in a second implementation of the first aspect of this application, the step of encoding the RGB frame sequence and the depth map sequence to obtain an RGB token sequence and a depth token sequence, and converting the motion trajectory into an action embedding vector compatible with the RGB token sequence, includes: The RGB frame sequence is compressed from a high-dimensional image frame sequence into a low-dimensional first latent space word sequence by a preset visual encoder to obtain the RGB word sequence. The depth map sequence is compressed from a high-dimensional image frame sequence into a low-dimensional second latent space word sequence by a preset visual encoder to obtain a depth word sequence. By inputting the motion trajectory into the action embedding module for linear transformation, each action vector in the motion trajectory is mapped to the same feature dimension as the RGB word sequence to obtain the action embedding vector.

[0007] Optionally, in a third implementation of the first aspect of this application, the step of performing autoregressive prediction based on the RGB word sequence, the deep word sequence, and the action embedding vector to obtain predicted RGB words and predicted deep words includes: The RGB word sequence, depth word sequence, action embedding vector, and position embedding vector prior to the current prediction time step are obtained, and information is fused by adding them element by element to obtain the fused input features. The fused input features are respectively input into the RGB prediction branch and the depth prediction branch of the autoregressive prediction network; The RGB prediction branch outputs the RGB features of the fused input features at the current prediction time step, and the depth prediction branch outputs the depth features of the fused input features at the current prediction time step. Determine the probability distribution of the RGB features and the depth features on a predetermined vocabulary, and determine the predicted RGB words and predicted depth words based on the probability distribution.

[0008] Optionally, in the fourth implementation of the first aspect of this application, the method further includes: The depth intermediate features output by the depth prediction branch at each level are transformed into the same dimension as the RGB intermediate features output by the RGB prediction branch at the corresponding level through a learnable linear projection. The transformed depth intermediate features are fused with the RGB intermediate features output at the same level by the RGB prediction branch to obtain the RGB features output by the RGB prediction branch.

[0009] Optionally, in the fifth implementation of the first aspect of this application, the step of jointly optimizing a multi-task objective function based on the predicted RGB symbols and the predicted deep symbols to obtain the trained embodied world model includes: The first cross-entropy loss between the predicted RGB word sequence and the actual RGB word sequence, and the second cross-entropy loss between the predicted depth word sequence and the actual depth word sequence are determined respectively. The Euclidean distance between the feature vector of the predicted RGB word at the corresponding word position on the motion trajectory and the initial frame is determined, and the average is calculated for all target key points and all time steps to obtain the temporal consistency loss. The first cross-entropy loss is weighted and summed according to the preset spatiotemporal attention map to obtain the key point-guided attention loss; The multi-task objective function is constructed based on the weighted sum of the first cross-entropy loss, the second cross-entropy loss, the temporal consistency loss, and the key point-guided attention loss. The parameters of the autoregressive prediction network are optimized according to the multi-task objective function to obtain the trained embodied world model.

[0010] Optionally, in the sixth implementation of the first aspect of this application, after the step of jointly optimizing the multi-task objective function based on the predicted RGB symbols and the predicted deep symbols to obtain the trained embodied world model, the method further includes: Obtain the initial frame and preset action sequence; The initial frame is encoded into an initial word sequence, and a corresponding target action embedding vector is generated based on the preset action sequence; Using the initial RGB word sequence, deep word sequence, and action embedding vector sequence as input, the trained embodied world model is used to autoregress frame by frame to generate future word sequences. The future word sequence is mapped back to pixel space through a pre-trained decoder to obtain a robot operation video.

[0011] A second aspect of this application provides a physically-guided embodied world model construction system, which is used to implement a physically-guided embodied world model construction method. The physically-guided embodied world model construction system includes: The acquisition module is used to process the acquired robot operation video to obtain the corresponding RGB frame sequence, depth map sequence, and motion trajectory; The encoding module is used to encode the RGB frame sequence and the depth map sequence to obtain an RGB word sequence and a depth word sequence, and to convert the motion trajectory into an action embedding vector compatible with the RGB word sequence; The prediction module is used to perform autoregressive prediction based on the RGB word sequence, the deep word sequence and the action embedding vector to obtain predicted RGB words and predicted deep words; The optimization module is used to jointly optimize the multi-task objective function based on the predicted RGB symbols and the predicted deep symbols to obtain the trained embodied world model.

[0012] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps in the physical information-guided embodied world model construction method provided in the first aspect of this application.

[0013] The fourth aspect of this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the physical information-guided embodied world model construction method provided in the first aspect of this application.

[0014] In summary, the method and system for constructing a physically-guided embodied world model provided in this application process the collected robot operation video for data processing, obtaining the corresponding RGB frame sequence, depth map sequence, and motion trajectory; encoding the RGB frame sequence and depth map sequence to obtain RGB word sequence and depth word sequence, and converting the motion trajectory into action embedding vectors compatible with the RGB word sequence; performing autoregressive prediction based on the RGB word sequence, depth word sequence, and action embedding vector to obtain predicted RGB words and predicted depth words; and jointly optimizing a multi-task objective function based on the predicted RGB words and predicted depth words to obtain the trained embodied world model. This application implicitly learns and internalizes the three-dimensional geometric constraints of the scene and the motion laws of objects while generating visual content, effectively improving the physical realism of the videos generated by the embodied world model. Attached Figure Description

[0015] Figure 1 A flowchart illustrating the physical information-guided embodied world model construction method provided in this application embodiment; Figure 2 A schematic diagram of the physical information-guided embodied world model architecture provided for embodiments of this application; Figure 3 This is a schematic diagram illustrating the generation of robot operation video based on the embodied world model, provided in an embodiment of this application. Figure 4 A flowchart of a robot strategy evaluation application based on a world model is provided for embodiments of this application; Figure 5 A schematic diagram of the program modules of the physically-guided embodied world model construction system provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] To address the issue of physical realism deficiencies in videos generated from embodied world models in related technologies, this application provides a method for constructing embodied world models guided by physical information. Figure 1 This is a flowchart illustrating the physical information-guided embodied world model construction method provided in this embodiment. The physical information-guided embodied world model construction method includes the following steps: Step 110: Process the collected robot operation video to obtain the corresponding RGB frame sequence, depth map sequence, and motion trajectory.

[0018] Specifically, the acquired robot operation videos undergo data processing to transform the raw, continuous video data into structured information suitable for model learning. By parsing the video content, the video is broken down into a sequence of RGB image frames arranged in chronological order, thus fully preserving the visual information of the robot's interaction with the environment. Simultaneously, to address the limitation of relying solely on two-dimensional images to reflect spatial structure, depth information acquisition technology is introduced. A time-aligned depth map is generated for each frame of the RGB image to characterize the three-dimensional geometric relationships of objects in the scene. Furthermore, key points are densely sampled from the first frame of the robot operation video, and their motion trajectories are tracked throughout the video sequence, establishing a clear correlation between visual changes and robot behavior.

[0019] Step 120: Encode the RGB frame sequence and the depth map sequence to obtain the RGB token sequence and the depth token sequence, and convert the motion trajectory into an action embedding vector compatible with the RGB token sequence.

[0020] Specifically, the obtained RGB frame sequences and depth map sequences are encoded separately to achieve efficient representation and unified modeling. The encoding process uses a visual encoder to compress high-dimensional, continuous image data into low-dimensional, discrete word sequences, transforming complex pixel information into a compact and semantically expressive latent space representation, thereby reducing computational complexity and improving modeling efficiency. Simultaneously, robot actions or motion trajectories are input into the action embedding module. Through linear mapping or embedding transformation, the original action vectors are converted into action embedding representations compatible with visual words in terms of feature dimensions. This allows action information to be fused with visual information in the same feature space, laying the foundation for jointly modeling the relationship between visual changes and action-driven processes.

[0021] Step 130: Perform autoregressive prediction based on the RGB word sequence, deep word sequence, and action embedding vector to obtain the predicted RGB word and the predicted deep word.

[0022] Specifically, an autoregressive prediction process is performed based on RGB word sequences, depth word sequences, and action embedding vectors to learn the temporal evolution of the robot's operational scenarios. In this process, the model uses multimodal word and action embeddings from historical time steps as conditional inputs, and progressively predicts the visual and depth representations for future moments through an autoregressive mechanism. This prediction method ensures that the model relies only on known historical information when generating results for the current moment, thus conforming to real-world temporal causality. By simultaneously predicting RGB and depth words, the model implicitly constrains spatial structure changes while generating visual appearance, making the prediction results more reasonable in terms of temporal continuity and spatial consistency.

[0023] Step 140: Perform joint optimization of the multi-task objective function based on the predicted RGB symbols and predicted deep symbols to obtain the trained embodied world model.

[0024] Specifically, a multi-task objective function is constructed based on the predicted RGB and depth symbols and jointly optimized to train the embodied world model. This joint optimization simultaneously measures visual prediction error, depth prediction error, and motion-related temporal consistency constraints, enabling the model to learn image generation capabilities while maintaining physical plausibility and dynamic continuity. The synergistic effect of multiple loss terms prompts the model to gradually internalize the geometric structure and motion patterns of the scene during parameter updates, thereby obtaining an embodied world model that possesses both visual realism and physical logic. This effectively improves the accuracy of robot operation video generation and provides a reliable foundation for robot policy learning.

[0025] In one optional implementation of this embodiment, the steps of processing the acquired robot operation video to obtain the corresponding RGB frame sequence, depth map sequence, and motion trajectory include: semantically segmenting the robot operation video to obtain short video clips of robot operation; decomposing the short video clips into RGB frame sequences, and processing each RGB frame of the RGB frame sequence to obtain a depth map sequence that is strictly aligned with the RGB frame sequence in time; determining an initial keypoint set in the first frame of the short video clip, and tracking the pixel coordinates of the initial keypoint set in all frames of the short video clip to obtain a keypoint trajectory set; determining the total motion amplitude of each trajectory in the keypoint trajectory set, and filtering the trajectories according to the total motion amplitude to obtain the trajectories of the top K keypoints with the largest motion amplitude, which are then determined as the motion trajectories of the target keypoints.

[0026] In this embodiment, semantic segmentation is achieved by analyzing the correspondence between changes in the video frame and changes in robot behavior. When there are significant changes in the motion pattern, contact object, or interaction state of the robot's end effector in the frame, it can be determined as the boundary point of different operational semantics. This method divides the original video into several short video segments, each corresponding to a complete and semantically consistent robot operation process. For example, in a desktop operation scenario, the movement of the robotic arm from the start of the gripper to the completion of dragging the cloth can be identified as an independent operation segment, thus avoiding the introduction of noise by mixing multiple unrelated or weakly related actions in the model. After obtaining the semantically consistent short video segments, frame-level parsing is performed to extract the RGB frame sequence, converting the continuous video signal into a two-dimensional image representation arranged in temporal order, thereby completely preserving visual appearance information. Subsequently, depth processing is performed on each frame in the RGB frame sequence to generate a depth map sequence that is strictly aligned with that frame in time. A depth map is an image representation that describes the distance from each point in the scene to the camera in pixels, directly reflecting the three-dimensional spatial structure and relative positional relationships of objects. By generating a depth map for each RGB frame separately and maintaining frame-level synchronization, it is ensured that at any given time, RGB information corresponds one-to-one with the corresponding spatial geometry. For example, in a cloth dragging scene, RGB frames depict the color and texture changes of the cloth, while the depth map reflects the spatial shape changes of the cloth as it is pulled up, drooping, or close to the table; together, they describe the realistic physical state. Figure 2 As shown, Figure 2 The process embodies keypoint trajectory tracking and bi-branch collaborative autoregressive prediction. It uses an existing keypoint sampling model (such as SpatialTracker) to densely sample N keypoints in the first frame of the video and tracks them across the entire frame. Keypoints are pixel coordinate trajectories within a frame. Keypoints refer to stable pixel locations in an image that can be consistently identified and located in subsequent frames, typically corresponding to object edges, areas of abrupt texture changes, or regions of significant motion. By densely selecting keypoints in the first frame and tracking their pixel coordinate changes in all subsequent frames, a set of keypoint trajectories can be obtained. Keypoint trajectories represent the projection changes of the same spatial location over time, used to characterize the motion of objects or robot parts. After obtaining the set of keypoint trajectories, they are filtered by calculating the total motion amplitude of each trajectory. The total motion amplitude measures the cumulative displacement of a single keypoint throughout the entire video clip, reflecting the dynamic activity of the area corresponding to that keypoint. Trajectories with smaller motion amplitudes typically correspond to background areas or static objects, contributing little to modeling physical interactions, while trajectories with larger motion amplitudes are often concentrated on robot end effectors or manipulated objects. By sorting all trajectories according to their total motion amplitude and selecting the top K trajectories with the largest motion amplitudes as the motion trajectories of the target keypoints, the model learning process focuses on the areas with the most concentrated physical interactions.

[0027] In one optional implementation of this embodiment, the steps of encoding the RGB frame sequence and the depth map sequence to obtain the RGB symbol sequence and the depth symbol sequence, and converting the motion trajectory into an action embedding vector compatible with the RGB symbol sequence, include: compressing the RGB frame sequence from a high-dimensional image frame sequence into a low-dimensional first latent space symbol sequence using a preset visual encoder to obtain the RGB symbol sequence; compressing the depth map sequence from a high-dimensional image frame sequence into a low-dimensional second latent space symbol sequence using a preset visual encoder to obtain the depth sequence; and linearly transforming the motion trajectory by inputting it into the action embedding module to map each action vector in the motion trajectory to the same feature dimension as the RGB symbol sequence to obtain the action embedding vector.

[0028] In this embodiment, processing the RGB frame sequence using a pre-defined visual encoder is a key technical step in achieving efficient modeling and unified representation. The RGB frame sequence consists of continuous color images, each containing a large number of pixels. Its data dimensionality is high and its redundancy is significant, making direct use for temporal modeling a substantial computational burden. A visual encoder (such as MAGVIT-v2) is an encoding model that maps images to a discrete latent space representation. Its core function is to divide the original image into several semantically consistent local regions and map each region to a discrete token. This token corresponds to an index in a predefined vocabulary, representing the appearance features and semantic information of the local region. In this way, the original RGB frames are compressed into a sequence of multiple tokens, i.e., the RGB token sequence, thereby significantly reducing data dimensionality while preserving key visual structures and semantic information. The same visual encoder is used to encode the depth map sequence, ensuring that the depth information and RGB information maintain consistency in representation. The depth map records the spatial distance information of each point in the scene in pixel form. Its numerical distribution and physical meaning differ from the RGB image, but it corresponds one-to-one with the RGB frames in spatial structure. By sharing a visual encoder structure and performing the same local partitioning and discrete mapping processing on the depth map, continuous depth values ​​can be converted into discrete depth symbol sequences. Depth symbols not only reflect the relative spatial position of local regions but also implicitly encode the surface undulations, occlusion relationships, and spatial continuity of the object. Since the RGB symbol sequences and depth symbol sequences are strictly aligned in time and space, the model can simultaneously process appearance and geometric information at the same latent space level during subsequent prediction. Simultaneously, motion trajectory information is input into the action embedding module for linear transformation to achieve unified modeling of robot actions and visual representations. The motion trajectory consists of a series of action vectors arranged in temporal order. Each action vector describes the robot's control command or state change at a corresponding time point, and its numerical dimension is not consistent with the feature dimension of the visual symbols. The action embedding module projects each action vector onto the same feature dimension as the RGB symbol sequence through linear mapping, enabling the fusion of action information and visual symbols in the same feature space. Linear transformation refers to weighted combination of the original action vectors using a learnable parameter matrix to obtain a new vector representation that retains the key control meaning of the action and adapts to the requirements of visual modeling. For example, during the process of the fabric being pulled up, RGB symbols capture the changes in the fabric's color and shape, while the corresponding depth symbols characterize the changes in the height of the fabric from the table. The combination of the two helps to constrain the spatial consistency of the generated results. The direction and speed of the gripper's movement are mapped into an action embedding vector of the same dimension as the visual symbols after embedding. In the latent space, the action change of "the gripper moves to the left" is directly associated with the visual change of "the edge of the fabric shifts to the left", thus establishing a clear action-driven relationship for autoregressive prediction.

[0029] In one optional implementation of this embodiment, the step of performing autoregressive prediction based on RGB word sequences, deep word sequences, and action embedding vectors to obtain predicted RGB words and predicted deep words includes: acquiring the RGB word sequence, deep word sequence, action embedding vector, and position embedding vector before the current prediction time step, and fusing information by adding them element-wise to obtain fused input features; inputting the fused input features into the RGB prediction branch and the deep prediction branch of the autoregressive prediction network respectively; outputting the RGB features of the fused input features at the current prediction time step through the RGB prediction branch, and outputting the deep features of the fused input features at the current prediction time step through the deep prediction branch; determining the probability distribution of the RGB features and deep features on a predetermined vocabulary, and determining the predicted RGB words and predicted deep words based on the probability distribution.

[0030] In this embodiment, during the autoregressive prediction process, known information is extracted from the historical time range prior to the current prediction time step as the conditional input for the current prediction. This includes RGB word sequences, depth word sequences, action embedding vectors, and position embedding vectors prior to the current prediction time step. The RGB word sequences and depth word sequences represent the historical evolution of the scene at the appearance and spatial geometry levels, respectively. The action embedding vectors characterize the control information applied to the environment by the robot within the corresponding time period, while the position embedding vectors are used to explicitly identify the relative positional relationships of words in the temporal and spatial dimensions, enabling the model to distinguish the semantic meanings of different time steps and different spatial regions. By fusing the above multi-source features element-wise, various types of information form a unified representation on the same feature dimension, thereby achieving tight coupling of visual, geometric, and action information without introducing additional structural complexity. This fusion result constitutes the fused input features required for prediction. The fused input features are then fed in parallel into the RGB prediction branch and the depth prediction branch in the autoregressive prediction network. The autoregressive prediction network uses a Transformer structure based on spatiotemporal modeling as the network backbone to perform unified temporal feature extraction and evolutionary modeling of multimodal word sequences. The network backbone consists of multiple spatiotemporal Transformer blocks stacked hierarchically. Each spatiotemporal Transformer block includes a spatial attention layer, a temporal attention layer, and a feedforward neural network. These layers model the relationships between tokens in both spatial and temporal dimensions. The spatial attention layer performs bidirectional correlation modeling of the token sequence within the same time step, enabling each token to integrate contextual information from different spatial locations within the current frame, thereby enhancing the ability to express the relationship between local regions and the overall structure. The temporal attention layer performs causal modeling of the token sequence in the temporal dimension, restricting the features of the current time step to rely only on token information from historical time steps, ensuring that the model conforms to the temporal causal relationships of the real physical world during prediction. In each spatiotemporal Transformer block, the fused input features are processed sequentially by the spatial and temporal attention layers before being input to the feedforward neural network for nonlinear feature transformation. Residual connections and normalization operations maintain feature stability. Through the cascading of multiple spatiotemporal Transformer blocks, the network backbone can abstract and aggregate the temporal features of multimodal tokens layer by layer, enabling the model to form a dynamic representation of the robot's operation process in a high-level semantic space. In the dual-branch collaborative autoregressive structure, the RGB prediction branch and the depth prediction branch share the same network backbone structure, but are independent in terms of parameters, thus modeling visual appearance information and three-dimensional geometric information respectively.Meanwhile, the network backbone provides an injection interface for intermediate features output by the deep prediction branch within its hierarchical structure. This allows the geometric features generated by the deep branch to guide the feature evolution of the RGB prediction branch at each level, thereby improving the performance of the generated results in terms of spatial consistency and physical plausibility. The RGB prediction branch focuses on modeling the patterns of visual appearance changes over time, while the deep prediction branch focuses on modeling the trends of the scene's 3D structure changes over time. Both branches perform feature transformations separately under the premise of receiving the same fused input features, enabling them to simultaneously infer changes in appearance and spatial structure under the same action conditions and historical state constraints.

[0031] In the RGB prediction branch, the fused input features are mapped to the RGB features of the current prediction time step, representing the prediction of the future visual appearance in the latent space. In the depth prediction branch, the fused input features are mapped to the depth features of the corresponding time step, representing the prediction of the future spatial geometric state. Both RGB and depth features are continuous latent space representations and do not yet correspond to specific discrete symbols; they need to be further matched with a predefined vocabulary. The vocabulary is constructed by the visual encoder during training and contains a finite number of discrete symbols, each representing a typical local visual or geometric pattern. By calculating the similarity between the predicted RGB and depth features and the symbols in the vocabulary, a probability distribution along the vocabulary dimension can be obtained. This probability distribution reflects the likelihood of different symbols being the prediction results. Based on the probability distribution, the predicted RGB symbols and predicted depth symbols are determined, thus completing the autoregressive prediction output for the current time step.

[0032] It should be noted that the autoregressive prediction formula can be expressed as: in, To predict RGB tokens, To predict depth symbols, For historical RGB symbols, For the depth of history, For action embedding vectors, This represents the embedding vector after the robot's continuous motion vectors have been mapped by a motion encoder (such as an MLP). Position embedding vector, Used to indicate the temporal and spatial positions of words in a sequence. To add element by element, and These are neural network functions for the RGB prediction branch and the depth prediction branch, respectively, and are composed of multiple spatiotemporal Transformer blocks stacked together.

[0033] In one optional implementation of this embodiment, the intermediate depth features output by the depth prediction branch at each level are transformed to the same dimension as the intermediate RGB features output by the RGB prediction branch at the corresponding level through a learnable linear projection; the transformed intermediate depth features are then fused with the intermediate RGB features output by the RGB prediction branch at the same level to obtain the RGB features output by the RGB prediction branch.

[0034] In this embodiment, in the dual-branch collaborative autoregressive structure, the intermediate depth features output by the depth prediction branch at each level are used to characterize the 3D geometric state of the scene at the corresponding abstract level. Intermediate depth features refer to the continuous feature vectors in the latent space by which the depth prediction branch expresses the spatial structure of a local region. These features not only contain relative distance information of object surfaces but also implicitly reflect changes in object shape and spatial continuity. Since the depth prediction branch and the RGB prediction branch may differ in network structure and feature channel settings, the feature dimensions of the intermediate depth features are not consistent with the RGB intermediate features output by the RGB prediction branch at the corresponding level, thus preventing direct fusion. To address this issue, a learnable linear projection mechanism is introduced. Through parameterized linear mapping, the intermediate depth features are transformed to the same feature dimensions as the RGB intermediate features, making the two types of features additive and fusionable in the numerical space. The linear projection automatically learns the importance of different depth feature channels in visual generation during the training process, thereby achieving effective alignment of geometric information to the visual feature space. After dimensional alignment, the depth intermediate features are fused with the RGB intermediate features output by the RGB prediction branch at the same level. The fused result serves as the output feature of the RGB prediction branch at that level, used for feature evolution in subsequent levels. The fusion process injects geometric information directly into the visual representation at the feature level, ensuring that the RGB prediction branch is continuously guided by the spatial constraints provided by the depth branch when generating the visual appearance. The RGB intermediate features primarily describe the color, texture, and appearance of local areas, while the depth intermediate features, after linear projection, supplement the structural information of the corresponding areas in three-dimensional space. The fusion of these two features allows the visual features to not only express appearance consistency but also implicitly encode spatial rationality. In applications where robots manipulate cloth, when the gripper moves forward and lifts the cloth, the depth prediction branch captures the geometric features of the cloth's height change at a lower level. After linear projection, these features are injected into the RGB prediction branch, ensuring that the RGB features are constrained by the spatial change of "the cloth being lifted" at the same level. This prevents inconsistencies between the cloth's shape and its spatial position during generation. As the network layers deepen, the fusion mechanism repeatedly applies geometric guidance at multiple abstraction levels, enabling the RGB prediction branch to continuously follow the real three-dimensional structural change rules during the overall generation process, ultimately resulting in an RGB feature representation that conforms to physical intuition in terms of both appearance and spatial consistency.

[0035] In one optional implementation of this embodiment, the step of jointly optimizing a multi-task objective function based on predicted RGB symbols and predicted deep symbols to obtain a trained embodied world model includes: determining the first cross-entropy loss between the predicted RGB symbols and the real RGB symbol sequences, and the second cross-entropy loss between the predicted deep symbols and the real deep symbol sequences; determining the Euclidean distance between the feature vectors of the predicted RGB symbols at the corresponding symbol positions on the motion trajectory at the current time step and the initial frame, and averaging over all target keypoints and all time steps to obtain the temporal consistency loss; weighting and summing the first cross-entropy loss according to a preset spatiotemporal attention map to obtain the keypoint-guided attention loss; constructing a multi-task objective function based on the weighted sum of the first cross-entropy loss, the second cross-entropy loss, the temporal consistency loss, and the keypoint-guided attention loss; and optimizing the parameters of the autoregressive prediction network according to the multi-task objective function to obtain the trained embodied world model.

[0036] Specifically, during model training, to ensure the reasonableness of the generated results in terms of visual appearance, spatial structure, and physical dynamics, a multi-task objective function composed of multiple loss terms is introduced to jointly optimize the autoregressive prediction network. First, the first cross-entropy loss is determined by comparing the difference between the predicted RGB symbols and the real RGB symbol sequences. Cross-entropy loss is a measure of the difference between the predicted probability distribution and the real discrete label distribution, used to characterize the model's accuracy in visual appearance prediction. When the model assigns higher probabilities to real RGB symbols in the vocabulary, this loss value decreases accordingly, thereby driving the model to gradually learn visual generation capabilities that conform to the real video distribution. Simultaneously, the second cross-entropy loss is determined by comparing the difference between the predicted depth symbols and the real depth symbol sequences, used to measure the model's accuracy in spatial geometry prediction. Depth symbols correspond to the relative spatial positions and structural states of various regions in the scene. This loss term prompts the model to learn the real three-dimensional geometric distribution during prediction, ensuring the reasonableness of the generated spatial structure over time. To further characterize the dynamic consistency of objects during interaction, a temporal consistency loss is introduced to constrain the feature stability of key regions. This loss is achieved by extracting feature vectors of predicted RGB symbols at the symbol positions corresponding to the motion trajectory and calculating the Euclidean distance between the current time step and the initial frame. Euclidean distance measures the difference between two feature vectors in the feature space; a smaller value indicates closer features. The distances across all target keypoints and all time steps are averaged to ensure the loss reflects the stability of key region features over time during the overall dynamic process. Furthermore, to enhance the model's learning of the core physical interaction regions, a keypoint-guided attention loss is introduced. Using a pre-defined spatiotemporal attention map, different weights are assigned to the first cross-entropy loss at different spatial locations and time steps, giving higher weight to symbol positions within the target keypoint trajectory coverage area in the loss calculation. The spatiotemporal attention map characterizes the importance of key regions in both spatial and temporal dimensions, guiding the model to allocate more learning power to the vicinity of the gripper and the manipulated object region. Finally, the first cross-entropy loss, the second cross-entropy loss, the temporal consistency loss, and the keypoint-guided attention loss are weighted and summed according to pre-defined weights to form a unified multi-task objective function. By optimizing the objective function, the parameters of the autoregressive prediction network are continuously updated during training, enabling the model to achieve a balance in multiple dimensions such as visual generation, spatial modeling, and dynamic consistency, ultimately resulting in an embodied world model that can generate physically reasonable and temporally coherent embodied worlds.

[0037] It should be noted that, in order to enable the model to learn the motion continuity and intrinsic physical characteristics of the object parts represented by the key points, this application designs two cooperative loss functions.

[0038] Temporal consistency loss This loss function requires a selected keypoint to be at different time steps. The corresponding predicted visual symbols It should be in the initial frame Visual symbols of time Maintain consistency. Its loss function is expressed as: , in, The first frame reference anchor point is defined as the coordinates of the initial frame at the sampling keypoint. The extracted visual symbols are considered as the intrinsic property benchmark of the object. Physically, they represent the original state characteristics of the object before it is disturbed or deformed, such as initial texture and material information. The representation of subsequent frame prediction is defined as the first frame. Frame tracking coordinates The predicted lexical features represent the real-time state of a particle after undergoing motion or deformation. The model needs to predict the visual appearance of the particle during the dynamic process. Used to measure the Euclidean distance between the predicted word and the initial word in the feature space. Here, T is the spatiotemporal normalization term, T is the total number of frames, and K is the number of keypoint samples. Since the first frame serves as the reference, effective physical constraints occur in subsequent frames. Within the frame, this fractional term distributes the total error evenly across the average deviation of each particle at each time step. This ensures that the magnitude of the loss function does not fluctuate drastically as the video length or sampling density increases, guaranteeing the training stability of the algorithm under different tasks.

[0039] Attention loss guided by key points The loss function expression is: , in, Represented as a spatiotemporal attention map, this is a weight matrix with the same size as the feature map. For word positions located on keypoint trajectories, the loss weights are amplified by a hyperparameter. times (e.g.) The weights of the weighted regions are 1, while the weights of other regions are 1. ⊙ represents element-wise multiplication, where the weight matrix is... By applying it to the original cross-entropy loss function, it achieves a local amplification of the prediction loss. Through The increased weighting results in a more severe penalty for prediction errors in key physical regions during model training, forcing the model to focus on the core areas of physical interaction. The cross-entropy loss term measures the difference in distribution between the predicted values ​​and the actual visual symbols, representing the model's accuracy in predicting future visual evolution.

[0040] Ultimately, the entire embodied world model is optimized end-to-end through a unified multi-task objective function, which is a weighted sum of the losses mentioned above: , in, For the first cross-entropy loss, The second cross-entropy loss is used to optimize the prediction accuracy of RGB and depth words. These are adjustable hyperparameters used to balance the importance of different tasks. The formula comprehensively constrains the model from pixel representation to deep physical logic through four complementary loss terms.

[0041] In one optional implementation of this embodiment, after the step of jointly optimizing the multi-task objective function based on the predicted RGB symbols and predicted deep symbols to obtain the trained embodied world model, the method further includes: acquiring an initial frame and a preset action sequence; encoding the initial frame into an initial symbol sequence and generating a corresponding target action embedding vector based on the preset action sequence; using the initial RGB symbol sequence, deep symbols, and action embedding vector sequence as input, generating a future symbol sequence through frame-by-frame autoregression using the trained embodied world model; and mapping the future symbol sequence back to the pixel space through a pre-trained decoder to obtain the robot operation video.

[0042] In this embodiment, as Figure 3As shown, after training the embodied world model, it can be used to generate robot operation videos that conform to physical laws. The process unfolds with an initial frame and a preset action sequence as input conditions. The initial frame depicts the initial environmental state at the start of robot operation, containing visual information about the robot itself, the manipulated object, and the surrounding scene. This initial state provides a realistic and stable starting point for subsequent generation. The preset action sequence consists of a series of action vectors arranged in chronological order, describing the robot's control commands or motion planning over a future period. Its content can originate from a predetermined control strategy or manual planning, thus explicitly specifying the robot's behavioral trajectory. Before generation begins, the initial frame is first input to a visual encoder consistent with the training phase for encoding, converting the high-dimensional pixel representation into a low-dimensional, discrete initial RGB symbol sequence. This symbol sequence fully preserves the appearance and structural features of the initial scene in the latent space and serves as the starting condition for autoregressive generation. Simultaneously, the preset action sequence is input to the action embedding module, which generates a target action embedding vector sequence through the same linear mapping method as the training phase, ensuring that the action information at each time step is represented in a feature form compatible with RGB symbols. Through this process, the initial visual state and future action planning are uniformly mapped to a latent space representation that the model can directly process. Based on this, the trained embodied world model takes the initial RGB word sequence, depth words, and action embedding vector sequence as input, and generates the future word sequence through a frame-by-frame autoregressive approach. At each time step, it predicts the word for the next time step based on existing words and corresponding action embeddings, and feeds back the prediction result as the input condition for the next time step, thus gradually unfolding the complete time series. This generation method follows temporal causality, allowing the model to infer using only previously generated information at each moment. After the future word sequence is generated, the latent space representation needs to be restored to a visualized video format. For this purpose, a pre-trained decoder corresponding to the visual encoder is introduced to map the discrete word sequence back to pixel space. The decoder reconstructs the latent space features corresponding to the words, generating image frames corresponding to the word sequence frame by frame, thereby recovering the scene's color, texture, and spatial structure information. Because the generation process is implicitly constrained by both depth and action information, the decoded video exhibits stability in terms of visual continuity and spatial consistency. Through the above process, the embodied world model can automatically generate physically plausible robot operation videos starting from the real initial state and preset actions.

[0043] Optional, such as Figure 4As shown, during robot execution and evaluation, the embodied world model and the robot strategy model form a closed-loop interactive relationship with the observed image and robot actions as the core information carriers. The input to the robot strategy model is the observed image at the current moment, which can be an image frame from the real environment or generated by the embodied world model, used to characterize the environmental state of the robot and the interaction between the robot and the environment. The strategy model analyzes the spatial structure, target position, and interaction state contained in the observed image and outputs the robot action corresponding to the time step. This robot action describes the control command or motion decision that the robot should execute in the next moment. The embodied world model uses the robot action and the current observed image as input to predict the evolution of the environmental state in the time dimension, thereby generating a predicted image frame for the next time step. This predicted image frame visually depicts the possible state changes of the environment after the robot executes the action, including changes in the robot's own pose and changes in the shape and spatial position of the manipulated object. Because the embodied world model has internalized the correlation between visual appearance, three-dimensional geometry, and physical dynamics during training, the generated predicted image is consistent with the actual execution result in terms of temporal continuity and physical consistency. The generated predicted image frames are then fed back as the observation input for the robot's policy model in the next time step, enabling the policy model to continue generating subsequent robot actions based on the predicted environment state without actual execution. By repeatedly alternating between action generation and image prediction processes between the policy model and the embodied world model, a time-series deduction based on an autoregressive mechanism is formed, which can gradually unfold the state evolution across multiple time steps, ultimately obtaining a multi-frame image sequence or a complete robot operation video composed of consecutive predicted image frames. In the policy evaluation phase, the multi-frame predicted image sequence generated by the embodied world model can be aligned and compared with the observation image sequence collected when the robot executes the same policy in the real environment. By analyzing the differences in visual state between the two at key time points, the consistency of the policy performance in the predicted and real environments can be determined, thereby evaluating the effectiveness and stability of the policy model.

[0044] According to the physical information-guided embodied world model construction method provided in this application, data processing is performed on the acquired robot operation video to obtain the corresponding RGB frame sequence, depth map sequence, and motion trajectory; the RGB frame sequence and depth map sequence are encoded to obtain RGB word sequence and depth word sequence, and the motion trajectory is converted into action embedding vectors compatible with the RGB word sequence; autoregressive prediction is performed based on the RGB word sequence, depth word sequence, and action embedding vector to obtain predicted RGB words and predicted depth words; joint optimization of a multi-task objective function is performed based on the predicted RGB words and predicted depth words to obtain the trained embodied world model. This application implicitly learns and internalizes the three-dimensional geometric constraints of the scene and the motion laws of objects while generating visual content, which can effectively improve the physical realism of the videos generated by the embodied world model.

[0045] Figure 5 This application provides a physical information-guided embodied world model construction system, which can be used to implement the physical information-guided embodied world model construction method in the foregoing embodiments. For example... Figure 5 As shown, this physical information-guided embodied world model construction system mainly includes: The acquisition module 10 is used to process the acquired robot operation video to obtain the corresponding RGB frame sequence, depth map sequence and motion trajectory; Encoding module 20 is used to encode the RGB frame sequence and the depth map sequence to obtain the RGB token sequence and the depth token sequence, and convert the motion trajectory into an action embedding vector compatible with the RGB token sequence; Prediction module 30 is used to perform autoregressive prediction based on RGB word sequence, deep word sequence and action embedding vector to obtain predicted RGB word and predicted deep word; The optimization module 40 is used to jointly optimize the multi-task objective function based on the predicted RGB characters and the predicted deep characters to obtain the trained embodied world model.

[0046] In one optional implementation of this embodiment, the acquisition module is specifically used for: semantically segmenting the robot operation video to acquire short video clips of robot operation; decomposing the short video clips into an RGB frame sequence, and processing each RGB frame of the RGB frame sequence to acquire a depth map sequence that is strictly aligned with the RGB frame sequence in time; determining an initial keypoint set in the first frame of the short video clip, and tracking the pixel coordinates of the initial keypoint set in all frames of the short video clip to acquire a keypoint trajectory set; determining the total motion amplitude of each trajectory in the keypoint trajectory set, and filtering the trajectories according to the total motion amplitude to acquire the trajectories of the top K keypoints with the largest motion amplitude, and determining them as the motion trajectories of the target keypoints.

[0047] In one optional implementation of this embodiment, the encoding module is specifically used to: compress the RGB frame sequence from a high-dimensional image frame sequence into a low-dimensional first latent space word sequence using a preset visual encoder, to obtain an RGB word sequence; compress the depth map sequence from a high-dimensional image frame sequence into a low-dimensional second latent space word sequence using a preset visual encoder, to obtain a depth word sequence; and perform a linear transformation by inputting the motion trajectory into the action embedding module, mapping each action vector in the motion trajectory to the same feature dimension as the RGB word sequence, to obtain an action embedding vector.

[0048] In an optional implementation of this embodiment, the prediction module is specifically used to: acquire the RGB word sequence, deep word sequence, action embedding vector, and position embedding vector before the current prediction time step, and perform information fusion by adding them element by element to obtain fused input features; input the fused input features to the RGB prediction branch and the deep prediction branch of the autoregressive prediction network respectively; output the RGB features of the fused input features at the current prediction time step through the RGB prediction branch, and output the deep features of the fused input features at the current prediction time step through the deep prediction branch; determine the probability distribution of the RGB features and the deep features on a predetermined vocabulary, and determine the predicted RGB words and predicted deep words based on the probability distribution.

[0049] In an optional implementation of this embodiment, the prediction module is further configured to: transform the intermediate depth features output by the depth prediction branch at each level to the same dimension as the intermediate RGB features output by the RGB prediction branch at the corresponding level through a learnable linear projection; and fuse the transformed intermediate depth features with the intermediate RGB features output by the RGB prediction branch at the same level to obtain the RGB features output by the RGB prediction branch.

[0050] In an optional implementation of this embodiment, the optimization module is specifically used to: determine the first cross-entropy loss between the predicted RGB word sequence and the real RGB word sequence, and the second cross-entropy loss between the predicted deep word sequence and the real deep word sequence; determine the Euclidean distance between the feature vector of the predicted RGB word at the corresponding word position on the motion trajectory at the current time step and the initial frame, and average it over all target keypoints and all time steps to obtain the temporal consistency loss; perform a weighted summation of the first cross-entropy loss according to a preset spatiotemporal attention map to obtain the keypoint-guided attention loss; construct a multi-task objective function based on the weighted sum of the first cross-entropy loss, the second cross-entropy loss, the temporal consistency loss, and the keypoint-guided attention loss; and optimize the parameters of the autoregressive prediction network according to the multi-task objective function to obtain the trained embodied world model.

[0051] In an optional implementation of this embodiment, the physically-guided embodied world model construction system further includes a generation module. The generation module is used to: acquire an initial frame and a preset action sequence; encode the initial frame into an initial word sequence and generate a corresponding target action embedding vector based on the preset action sequence; use the initial RGB word sequence, depth word sequence, and action embedding vector sequence as input, and generate a future word sequence through frame-by-frame autoregression using the trained embodied world model; and map the future word sequence back to the pixel space using a pre-trained decoder to obtain a robot operation video.

[0052] According to the physical information-guided embodied world model construction system provided in this application, data processing is performed on the acquired robot operation video to obtain the corresponding RGB frame sequence, depth map sequence, and motion trajectory; the RGB frame sequence and depth map sequence are encoded to obtain RGB word sequence and depth word sequence, and the motion trajectory is converted into action embedding vectors compatible with the RGB word sequence; autoregressive prediction is performed based on the RGB word sequence, depth sequence, and action embedding vector to obtain predicted RGB words and predicted depth words; joint optimization of a multi-task objective function is performed based on the predicted RGB words and predicted depth words to obtain the trained embodied world model. This application implicitly learns and internalizes the three-dimensional geometric constraints of the scene and the motion laws of objects while generating visual content, which can effectively improve the physical realism of the videos generated by the embodied world model.

[0053] According to the scheme provided in this application Figure 6 An electronic device is provided as an embodiment of this application. This electronic device can be used to implement the physically-guided embodied world model construction method in the foregoing embodiments, and mainly includes: The system includes a memory 601, a processor 602, and a computer program 603 stored on the memory 601 and executable on the processor 602. The memory 601 and the processor 602 are connected via communication. When the processor 602 executes the computer program 603, it implements the physical information-guided embodied world model construction method described in the foregoing embodiments. The number of processors can be one or more.

[0054] The memory 601 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 601 is used to store executable program code, and the processor 602 is coupled to the memory 601.

[0055] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 6 The memory in the illustrated embodiment.

[0056] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the physical information-guided embodied world model construction method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk.

[0057] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0058] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0059] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing an embodied world model guided by physical information, characterized in that, include: The collected robot operation videos are processed to obtain the corresponding RGB frame sequences, depth map sequences, and motion trajectories; The RGB frame sequence and the depth map sequence are encoded to obtain an RGB word sequence and a depth word sequence, and the motion trajectory is converted into an action embedding vector compatible with the RGB word sequence; Autoregressive prediction is performed based on the RGB word sequence, the deep word sequence, and the action embedding vector to obtain the predicted RGB word and the predicted deep word. The trained embodied world model is obtained by jointly optimizing the multi-task objective function based on the predicted RGB words and the predicted deep words.

2. The method for constructing a physically-informed embodied world model according to claim 1, characterized in that, The steps of processing the acquired robot operation video to obtain the corresponding RGB frame sequence, depth map sequence, and motion trajectory include: Semantic segmentation is performed on the robot operation video to obtain short video clips of the robot operation; The short video segment is decomposed into an RGB frame sequence, and each RGB frame of the RGB frame sequence is processed to obtain a depth map sequence that is strictly aligned with the RGB frame sequence in time. An initial set of key points is determined in the first frame of the short video segment, and the pixel coordinates of the initial set of key points are tracked in all frames of the short video segment to obtain a set of key point trajectories. The total motion amplitude of each trajectory in the set of key point trajectories is determined, and the trajectories are filtered according to the total motion amplitude to obtain the trajectories of the top K key points with the largest motion amplitude, which are then determined as the motion trajectories of the target key points.

3. The method for constructing a physically-informed embodied world model according to claim 2, characterized in that, The step of encoding the RGB frame sequence and the depth map sequence to obtain an RGB word sequence and a depth word sequence, and converting the motion trajectory into an action embedding vector compatible with the RGB word sequence, includes: The RGB frame sequence is compressed from a high-dimensional image frame sequence into a low-dimensional first latent space word sequence by a preset visual encoder to obtain the RGB word sequence. The depth map sequence is compressed from a high-dimensional image frame sequence into a low-dimensional second latent space word sequence by a preset visual encoder to obtain a depth word sequence. By inputting the motion trajectory into the action embedding module for linear transformation, each action vector in the motion trajectory is mapped to the same feature dimension as the RGB word sequence to obtain the action embedding vector.

4. The method for constructing a physically-informed embodied world model according to claim 3, characterized in that, The step of performing autoregressive prediction based on the RGB word sequence, the deep word sequence, and the action embedding vector to obtain predicted RGB words and predicted deep words includes: The RGB word sequence, depth word sequence, action embedding vector, and position embedding vector prior to the current prediction time step are obtained, and information is fused by adding them element by element to obtain the fused input features. The fused input features are respectively input into the RGB prediction branch and the depth prediction branch of the autoregressive prediction network; The RGB prediction branch outputs the RGB features of the fused input features at the current prediction time step, and the depth prediction branch outputs the depth features of the fused input features at the current prediction time step. Determine the probability distribution of the RGB features and the depth features on a predetermined vocabulary, and determine the predicted RGB words and predicted depth words based on the probability distribution.

5. The method for constructing a physically-guided embodied world model according to claim 4, characterized in that, The method further includes: The depth intermediate features output by the depth prediction branch at each level are transformed into the same dimension as the RGB intermediate features output by the RGB prediction branch at the corresponding level through a learnable linear projection. The transformed depth intermediate features are fused with the RGB intermediate features output at the same level by the RGB prediction branch to obtain the RGB features output by the RGB prediction branch.

6. The method for constructing a physically-informed embodied world model according to claim 4, characterized in that, The step of jointly optimizing the multi-task objective function based on the predicted RGB words and the predicted deep words to obtain the trained embodied world model includes: The first cross-entropy loss between the predicted RGB word sequence and the actual RGB word sequence, and the second cross-entropy loss between the predicted depth word sequence and the actual depth word sequence are determined respectively. The Euclidean distance between the feature vector of the predicted RGB word at the corresponding word position on the motion trajectory and the initial frame is determined, and the average is calculated for all target key points and all time steps to obtain the temporal consistency loss. The first cross-entropy loss is weighted and summed according to the preset spatiotemporal attention map to obtain the key point-guided attention loss; The multi-task objective function is constructed based on the weighted sum of the first cross-entropy loss, the second cross-entropy loss, the temporal consistency loss, and the key point-guided attention loss. The parameters of the autoregressive prediction network are optimized according to the multi-task objective function to obtain the trained embodied world model.

7. The method for constructing a physically-informed embodied world model according to claim 1, characterized in that, After the step of jointly optimizing the multi-task objective function based on the predicted RGB words and the predicted deep words to obtain the trained embodied world model, the method further includes: Obtain the initial frame and preset action sequence; The initial frame is encoded into an initial word sequence, and a corresponding target action embedding vector is generated based on the preset action sequence; Using the initial RGB word sequence, deep word sequence, and action embedding vector sequence as input, the trained embodied world model is used to autoregress frame by frame to generate future word sequences. The future word sequence is mapped back to pixel space through a pre-trained decoder to obtain a robot operation video.

8. A physical information-guided embodied world model construction system, characterized in that, The physical information-guided embodied world model construction system is used to implement the physical information-guided embodied world model construction method of claim 1, wherein the physical information-guided embodied world model construction system comprises: The acquisition module is used to process the acquired robot operation video to obtain the corresponding RGB frame sequence, depth map sequence, and motion trajectory; The encoding module is used to encode the RGB frame sequence and the depth map sequence to obtain an RGB word sequence and a depth word sequence, and to convert the motion trajectory into an action embedding vector compatible with the RGB word sequence; The prediction module is used to perform autoregressive prediction based on the RGB word sequence, the deep word sequence and the action embedding vector to obtain predicted RGB words and predicted deep words; The optimization module is used to jointly optimize the multi-task objective function based on the predicted RGB symbols and the predicted deep symbols to obtain the trained embodied world model.

9. An electronic device, characterized in that, Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps in the physical information-guided embodied world model construction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the physical information-guided embodied world model construction method according to any one of claims 1 to 7.

Images