A method and system for constructing a body-aware intelligent world model
Patent Information
- Application Number
- CN202611108246.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-24
- Publication Date
- 2026-10-30
AI Technical Summary
[0005]为了克服现有具身智能世界模型难以在同一框架下兼顾物体级因果交互建模、长序列高效推演与环境自适应能力，本发明提供一种具身智能世界模型的构建方法及系统
[0039]1、本发明通过对象提取与离散环境码本检索，构建了场景内各物体对应的对象表征及全局环境表征，并在多层稀疏因果状态空间模型中设置因果交互层，以稀疏注意力机制计算物体间的关联权重生成单层因果邻接矩阵，进而将各层矩阵叠加单位矩阵后逐层连乘，构建累积多层因果传递路径的全局因果矩阵，最终输出物体与环境之间的显式因果结构图，使模型具备可解释的因果推理能力，为机器人在分布外场景中的行为决策提供了因果层面的可追溯依据；
Smart Images
Figure CN123014026A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of embodied intelligence technology, and more specifically, to a method and system for constructing an embodied intelligence world model. Background Technology
[0002] Embodied intelligence aims to endow intelligent agents with the ability to perceive, understand, and act in the physical environment, and is a key path for artificial intelligence to move towards real-world applications. World models, as a core component of embodied intelligent systems, construct dynamic models of the environment within the intelligent agent, enabling the agent to perform forward reasoning and behavioral planning without directly interacting with the physical world. This improves decision-making efficiency and safety, and has become a core research paradigm in fields such as robot cognition, autonomous driving, and general embodied agents.
[0003] The core task of world models is to learn environmental dynamics from high-dimensional visual observations and predict future states. The Dreamer series, by encoding visual input into latent states and learning environmental dynamics through a cyclic state-space model, has achieved excellent performance in multiple tasks. However, its method of encoding the entire scene into a unified latent vector fails to distinguish between different objects and their relationships, lacking refined modeling of physical interactions between objects. Research such as SPARTAN has begun to focus on object-level causal structure learning, modeling local causal relationships between entities through Transformer-based object decomposition and sparse regularization. This has initially verified the value of causal modeling in improving generalization ability. However, the computational complexity of attention in the Transformer architecture increases quadratically with sequence length, facing efficiency bottlenecks when handling long sequences. The Mamba architecture, based on a state-space model, has been introduced into world models to achieve linear complexity modeling of long sequences. However, existing work mainly focuses on global sequence prediction and does not explicitly model causal interactions between objects. Furthermore, existing models couple environmental physical rules with object states in the same representation space, requiring retraining when the environment changes, making rapid domain adaptation difficult.
[0004] In summary, existing embodied intelligent world models have significant shortcomings in object-level causal interaction modeling, long-sequence computation efficiency, and environmental adaptability. Therefore, there is an urgent need to provide a solution to address these issues. Summary of the Invention
[0005] To overcome the difficulty of existing embodied intelligent world models in simultaneously achieving object-level causal interaction modeling, efficient long-sequence inference, and environmental adaptation capabilities within the same framework, this invention provides a method and system for constructing embodied intelligent world models.
[0006] The technical solution of this invention is as follows:
[0007] A method for constructing an embodied intelligent world model includes the following steps:
[0008] The system acquires input video frames of the robot's working scene, extracts objects from the input video frames to obtain object representations of each object in the scene, generates an environment query vector based on continuous multi-frame observations of the input video, and performs similarity matching retrieval between the environment query vector and the discrete environment codebook to obtain a global environment representation.
[0009] The object representation and the global environment representation are input into a multi-layer sparse causal state space model. Each layer of the sparse causal state space model includes an independent evolution layer and a causal interaction layer. Based on the independent evolution layer and the causal interaction layer, the current layer's fused state representation is output, and a single-layer causal adjacency matrix of the current layer is generated. The fused state representation is used as the input of the next layer's sparse causal state space model, and the hierarchical deduction is completed layer by layer.
[0010] The single-layer causal adjacency matrices of each layer are superimposed with the identity matrix and then multiplied layer by layer to generate a global causal matrix that accumulates multiple causal propagation paths. A joint loss function containing video reconstruction loss and sparse regularization loss is constructed, and a sparse causal state space model and an image decoder are trained based on the joint loss function.
[0011] The fused state representation, which is the final output of the sparse causal state space model, is input into the image decoder to generate future video prediction frames of the scene, and an explicit causal structure graph between objects and the environment is output based on the global causal matrix.
[0012] Preferably, the process of extracting objects from the input video frames to obtain object representations of each object in the scene includes:
[0013] A pre-trained instance segmentation network or unsupervised object discovery module is used to obtain image regions of each object, and the image regions are encoded into object representations corresponding to the objects based on a variational autoencoder.
[0014] Preferably, the environment query vector is obtained based on the motion trajectory feature encoding of each object in the continuous multi-frame sequence.
[0015] Preferably, the process of outputting the current layer fusion state representation based on the independent evolution layer and the causal interaction layer includes:
[0016] The independent evolution layer performs parallel temporal deduction of the object representation and the global environment representation based on the state space model to obtain the independent evolution state corresponding to each object and the global environment; the causal interaction layer calculates the correlation weight between each object based on all the independent evolution states, generates a single-layer causal adjacency matrix, and weights and aggregates each of the independent evolution states based on the single-layer causal adjacency matrix to output the current layer fusion state representation.
[0017] Preferably, the process of generating a single-layer causal adjacency matrix by calculating the association weights between objects based on all the independent evolution states in the causal interaction layer includes:
[0018] All independent evolutionary states are mapped to query vectors, key vectors, and value vectors through linear projection. The attention score between any two independent evolutionary states is calculated as the association weight to obtain a single-layer causal adjacency matrix. The calculation formula is as follows:
[0019] ;
[0020] in, Indicates from object To the object The strength of causal influence The Sigmoid activation function is used to map attention scores to the [0, 1] interval to characterize the strength of causal influence; For the first The query vector corresponding to each independent evolutionary state. For the first The key vector corresponding to each independent evolutionary state For vector dimensions.
[0021] Preferably, in the process of weighted aggregation of each independent evolutionary state based on the single-layer causal adjacency matrix to output the current layer fusion state representation, the first... The updated state after weighted aggregation of individual evolutionary states is represented as follows:
[0022] ;
[0023] in, For the first An object at time... The representation vector, The output of the independent evolution layer The independent evolutionary state of an object For the single-layer causal adjacency matrix, from the object To the object The strength of causal influence For the first The value vector corresponding to each independent evolutionary state This is a multilayer perceptron used for nonlinear transformations of value vectors, with a summation term. Represents all objects The sum of the weighted effects of neighboring objects that have a causal influence on it.
[0024] Preferably, the expression for calculating the global causal matrix is:
[0025] ;
[0026] Among them, matrix elements in Indicates from object To the object The total number of existing causal paths The total number of layers in the sparse causal state-space model. For the first A single-layer causal adjacency matrix of layers. It is the identity matrix. Indicates the first Self-loops are introduced into the layers to preserve the state information of each object.
[0027] Preferably, the expression for the joint loss function is:
[0028] ;
[0029] in, It is the image generated by the decoder based on the prediction slot. For the actual next moment's video frame, These are sparse regularization weights, used to balance the relative importance of reconstruction loss and sparse regularization loss. is the L1 norm of the matrix, used to constrain the sparsity of the global causal matrix.
[0030] Preferably, during the training of the sparse causal state-space model and the image decoder based on the joint loss function, the sparse regularization weights are dynamically adjusted, specifically including:
[0031] In the early stages of training, set relatively small sparse regularization weight values so that the model learns basic motion patterns primarily through reconstruction loss.
[0032] As the number of training rounds increases, the sparse regularization weight value is gradually increased according to the decreasing trend of the reconstruction error to enhance the sparsity constraint strength, so that the model can gradually eliminate redundant causal connections and automatically discover the real physical interaction structure.
[0033] A second aspect of the present invention also provides a system for constructing an embodied intelligent world model, comprising a representation extraction module, a representation fusion module, a model optimization module, and a prediction output module, wherein:
[0034] The representation extraction module is used to acquire input video frames of the robot's working scene, extract objects from the input video frames to obtain object representations corresponding to each object in the scene, generate environment query vectors based on continuous multi-frame observations of the input video, and perform similarity matching retrieval between the environment query vectors and the discrete environment codebook to obtain global environment representations.
[0035] The representation fusion module is used to input the object representation and the global environment representation into a multi-layer sparse causal state space model. Each layer of the sparse causal state space model includes an independent evolution layer and a causal interaction layer. Based on the independent evolution layer and the causal interaction layer, the current layer's fused state representation is output. The fused state representation is used as the input to the next layer's sparse causal state space model, and the hierarchical deduction is completed layer by layer.
[0036] The model optimization module is used to multiply the single-layer causal adjacency matrix of each layer by the identity matrix and generate a global causal matrix that accumulates multiple causal propagation paths. It constructs a joint loss function that includes video reconstruction loss and sparse regularization loss, and trains a sparse causal state space model and an image decoder based on the joint loss function.
[0037] The prediction output module is used to input the fused state representation of the sparse causal state space model into the image decoder, decode it to generate a future video prediction frame of the scene, and output an explicit causal structure graph between the object and the environment based on the global causal matrix.
[0038] According to the above-described solution, the beneficial effects of this invention are as follows:
[0039] 1. This invention constructs object representations and global environment representations corresponding to each object in a scene through object extraction and discrete environment codebook retrieval. In a multi-layer sparse causal state space model, a causal interaction layer is set up. The association weights between objects are calculated using a sparse attention mechanism to generate a single-layer causal adjacency matrix. Then, the matrices of each layer are superimposed with an identity matrix and multiplied layer by layer to construct a global causal matrix that accumulates multiple causal transmission paths. Finally, an explicit causal structure graph between objects and the environment is output, which enables the model to have interpretable causal reasoning ability and provides a traceable causal basis for the robot's behavioral decisions in distributed scenes.
[0040] 2. This invention uses a state-space model as the core temporal inference unit, combined with a sparse attention mechanism and a multi-layer cascade structure, to achieve refined modeling of object-level causal interactions while maintaining linear computational complexity, thus balancing long sequence prediction efficiency and object-level interaction modeling capabilities.
[0041] 3. This invention constructs a global environment representation through a similarity matching retrieval mechanism of discrete environment codebooks, decouples environmental physical rules from object states and participates in the whole process of temporal inference and causal interaction, realizes environmental domain adaptation without retraining, and has the ability to adapt to the environment with zero or few samples. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 A schematic diagram illustrating the construction method of the embodied intelligent world model provided by the present invention. Detailed Implementation
[0044] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0045] It should be noted that when a component is referred to as "fixed," "set," or "connected" to another component, it may be located directly or indirectly on that other component. The terms "upper," "lower," "left," "right," "front," "rear," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or position based on the accompanying drawings, and are for ease of description only, and should not be construed as limiting the technical solution. The terms "first," "second," etc., are used for ease of description only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features. "Many" means two or more, unless otherwise explicitly specified. "Several" means one or more, unless otherwise explicitly specified.
[0046] This invention provides a method for constructing an embodied intelligent world model, see [link to relevant documentation]. Figure 1 This includes the following steps:
[0047] S1. Obtain the input video frames of the robot's working scene, extract objects from the input video frames to obtain the object representations of each object in the scene, generate an environment query vector based on continuous multi-frame observations of the input video, and perform similarity matching retrieval between the environment query vector and the discrete environment codebook to obtain the global environment representation.
[0048] S2. Input the object representation and the global environment representation into a multi-layer sparse causal state space model. Each layer of the sparse causal state space model includes an independent evolution layer and a causal interaction layer. Output the current layer fusion state representation based on the independent evolution layer and the causal interaction layer, and generate a single-layer causal adjacency matrix for the current layer. Use the fusion state representation as the input to the next layer of the sparse causal state space model, and complete the hierarchical deduction layer by layer.
[0049] S3. The single-layer causal adjacency matrix of each layer is superimposed with the identity matrix and then multiplied layer by layer to generate a global causal matrix that accumulates multiple causal propagation paths. A joint loss function including video reconstruction loss and sparse regularization loss is constructed. A sparse causal state space model and an image decoder are trained based on the joint loss function.
[0050] S4. Input the fused state representation, which is the final output of the sparse causal state space model, into the image decoder, decode it to generate a future video prediction frame of the scene, and output an explicit causal structure graph between the object and the environment based on the global causal matrix.
[0051] In some embodiments, the process of performing step S1 includes:
[0052] S1.1, Obtain the input video frame;
[0053] S1.2 Extract objects from the input video frames to obtain the image regions of each object;
[0054] S1.3 Encode each object image region into an object representation;
[0055] S1.4. Generate environment query vectors based on continuous multi-frame observations;
[0056] S1.5. Perform similarity matching retrieval between the environment query vector and the discrete environment codebook to obtain the global environment representation.
[0057] Specifically, during the execution of step S1.1, the following steps are included: real-time acquisition of video streams of the work scene using an RGB camera or depth camera mounted on the robot, with a frame rate of 30Hz or 60Hz, and each frame serving as input to the world model.
[0058] Further, in step S1.2, a pre-trained instance segmentation network or unsupervised object detection module is used to process the input video frames. The instance segmentation network can employ Mask R-CNN or Segment Anything Model (SAM) to output image region masks for each object; the unsupervised object detection module can use Slot Attention to automatically detect object regions in the scene without pre-annotation. The masks are multiplied by the original image to obtain independent image patches for each object.
[0059] Specifically, step S1.3 includes: inputting independent image patches of each object into a variational autoencoder (VAE) with shared weights; outputting the mean and variance of the encoder; and obtaining a low-dimensional vector through reparameterization techniques, which serves as the object representation for each object. The number of object representations is consistent with the number of objects in the scene, and each object representation represents the visual appearance and spatial location information of that object.
[0060] Further, step S1.4 is executed to extract the motion trajectory features of each object in a continuous multi-frame sequence, including the temporal motion information such as the displacement direction, velocity change, and acceleration of each object. These motion trajectory features are then concatenated and mapped through a linear layer to a fixed-dimensional environment query vector. The dimension of the environment query vector is consistent with the dimension of the object representation.
[0061] Furthermore, in step S1.5, during the training phase, a discrete environment codebook is pre-constructed using clustered environment dynamics patterns. This codebook contains K codebook entries representing different types of environmental physical rules (e.g., K=16), with each codebook entry being a vector of the same dimension as the object representation. The similarity (e.g., cosine similarity or Euclidean distance) between the environment query vector and each codebook entry is calculated, and the codebook entry with the highest similarity is selected as the global environment representation for the current time step.
[0062] In some embodiments, the process of performing step S2 includes:
[0063] S2.1 Input the first-layer sparse causal state space model;
[0064] S2.2, Parallel temporal deduction is performed on the independent evolution layer;
[0065] S2.3, the causal interaction layer calculates the association weights and generates a single-layer causal adjacency matrix;
[0066] S2.4, Add a causal mask;
[0067] S2.5, Weighted aggregation output fusion state representation;
[0068] S2.6 Complete the hierarchical deduction layer by layer.
[0069] Specifically, in step S2.1, the object representation and the global environment representation are jointly input into the first layer of the multi-layer sparse causal state space model. Each layer of the sparse causal state space model contains two sub-layers: an independent evolution layer and a causal interaction layer.
[0070] Further, in step S2.2, the independent evolution layer performs parallel temporal deduction of each representation based on the state space model to obtain the independent evolution state corresponding to each object and the global environment.
[0071] In practice, the state-space model adopts the Mamba architecture, leveraging its linear recursive properties to treat each representation as an independent time series for parallel temporal evolution prediction. Independently evolving state representations, without considering the mutual influence between objects, allow each object to independently evolve to the next state based on its own motion inertia and the global environment according to its own physical rules.
[0072] Furthermore, in step S2.3, the causal interaction layer calculates the correlation weights between objects based on all independent evolution states, generating a single-layer causal adjacency matrix.
[0073] Specifically, all independent evolutionary states are mapped to query vectors, key vectors, and value vectors using three independent linear projection matrices. The dot product of the query vector and key vector is calculated and scaled by the square root of the vector dimension. Then, the attention score is mapped to the [0,1] interval using a sigmoid activation function to obtain the association weights between any two independent evolutionary states, forming a single-layer causal adjacency matrix. The calculation formula is as follows:
[0074] ;
[0075] in, Indicates from object To the object The strength of causal influence The Sigmoid activation function is used to map attention scores to the [0, 1] interval to characterize the strength of causal influence; For the first The query vector corresponding to each independent evolutionary state. For the first The key vector corresponding to each independent evolutionary state For vector dimensions.
[0076] Furthermore, in step S2.4, a causal mask is added to the causal interaction layer to prevent future state information after the current moment from participating in the correlation weight calculation of the current moment, so as to prevent the leakage of future information.
[0077] Specifically, the execution of step S2.5 includes: weighted aggregation of each independent evolutionary state based on a single-layer causal adjacency matrix, and outputting the current layer's fusion state representation. The updated state after weighted aggregation of individual evolutionary states is represented as follows:
[0078] ;
[0079] in, For the first An object at time... The representation vector, The output of the independent evolution layer The independent evolutionary state of an object For the single-layer causal adjacency matrix, from the object To the object The strength of causal influence For the first The value vector corresponding to each independent evolutionary state This is a multilayer perceptron used for nonlinear transformations of value vectors, with a summation term. Represents all objects The sum of the weighted effects of neighboring objects that have a causal influence on it.
[0080] Furthermore, step S2.6 is executed, where the fused state representation output from the current layer is used as the input to the sparse causal state space model of the next layer, and the hierarchical deduction is completed layer by layer. As the number of layers increases, causal information is passed layer by layer along the object chain, enabling the model to capture multi-step causal relationships.
[0081] In some embodiments, the process of performing step S3 includes:
[0082] S3.1 Generate a global causal matrix by successive multiplication at each level;
[0083] S3.2 Construct the joint loss function;
[0084] S3.3, Dynamically adjust sparse regularization weights;
[0085] S3.4, End-to-end training and early stop.
[0086] Specifically, in the execution of step S3.1, the following steps are included: superimposing the single-layer causal adjacency matrices of each layer onto the identity matrix and then multiplying them layer by layer to generate a global causal matrix that accumulates multiple layers of causal propagation paths. The calculation formula is as follows:
[0087] ;
[0088] Among them, matrix elements in Indicates from object To the object The total number of existing causal paths The total number of layers in the sparse causal state-space model. For the first A single-layer causal adjacency matrix of layers. It is the identity matrix. Indicates the first Self-loops are introduced into the layers to preserve the state information of each object.
[0089] Further, in step S3.2, a joint loss function is constructed, which includes video reconstruction loss and sparse regularization loss. The calculation formula is as follows:
[0090] ;
[0091] in, It is the image generated by the decoder based on the prediction slot. For the actual next moment's video frame, These are sparse regularization weights, used to balance the relative importance of reconstruction loss and sparse regularization loss. is the L1 norm of the matrix, used to constrain the sparsity of the global causal matrix.
[0092] Specifically, step S3.3 includes: training a sparse causal state-space model and an image decoder based on the joint loss function. During training, the sparse regularization weights are dynamically adjusted: a smaller sparse regularization weight value is set in the early stages of training (e.g., λ = 1 × 10⁻⁻⁴). 4 The model prioritizes learning basic motion patterns through reconstruction loss. As the training rounds increase, the value of λ is gradually increased based on the decreasing trend of reconstruction error to enhance the sparsity constraint strength, forcing the model to gradually eliminate redundant causal connections and automatically discover the real physical interaction structure.
[0093] Further, step S3.4 is executed, using the AdamW optimizer for end-to-end training. During training, the loss value on the validation set is monitored. Training is terminated when the validation set loss does not decrease for multiple consecutive rounds (e.g., 10 consecutive rounds), and the trained sparse causal state space model and image decoder are exported.
[0094] In some embodiments, the process of performing step S4 includes:
[0095] S4.1 Input image decoder;
[0096] S4.2 Decode and generate future video prediction frames;
[0097] S4.3 Output an explicit cause-effect diagram.
[0098] Specifically, in step S4.1, the fused state representation output by the sparse causal state space model is input into the image decoder. The image decoder consists of deconvolutional layers or anti-attention layers, which map the latent state representation back to the pixel space.
[0099] Further, in step S4.2, the image decoder outputs the reconstructed images of each object and the background at the next moment, and stitches or overlays the reconstruction results of all objects according to their spatial positions to generate a complete future video prediction frame of the scene.
[0100] Furthermore, step S4.3 is executed to output an explicit causal structure graph between the object and the environment based on the global causal matrix. This causal structure graph is a directed weighted graph, where each node corresponds to an object in the scene or the global environment. The direction of the directed edges represents the direction of the causal influence, and the weight of the directed edges represents the strength of the corresponding causal influence. This causal structure graph can be output and visualized in real time, providing the robot decision-making system with information for behavior interpretation, fault diagnosis, and safety monitoring.
[0101] A second aspect of the invention also provides a system for constructing an embodied intelligent world model, comprising a representation extraction module, a representation fusion module, a model optimization module, and a prediction output module, wherein:
[0102] The representation extraction module is used to acquire input video frames of the robot's working scene, extract objects from the input video frames to obtain object representations corresponding to each object in the scene, generate environment query vectors based on continuous multi-frame observations of the input video, and perform similarity matching retrieval between the environment query vectors and the discrete environment codebook to obtain global environment representations.
[0103] The representation fusion module is used to input the object representation and the global environment representation into a multi-layer sparse causal state space model. Each layer of the sparse causal state space model includes an independent evolution layer and a causal interaction layer. Based on the independent evolution layer and the causal interaction layer, the current layer's fused state representation is output. The fused state representation is used as the input to the next layer's sparse causal state space model, and the hierarchical deduction is completed layer by layer.
[0104] The model optimization module is used to multiply the single-layer causal adjacency matrix of each layer by the identity matrix and generate a global causal matrix that accumulates multiple causal propagation paths. It constructs a joint loss function that includes video reconstruction loss and sparse regularization loss, and trains a sparse causal state space model and an image decoder based on the joint loss function.
[0105] The prediction output module is used to input the fused state representation of the sparse causal state space model into the image decoder, decode it to generate a future video prediction frame of the scene, and output an explicit causal structure graph between the object and the environment based on the global causal matrix.
[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for constructing an embodied intelligent world model, characterized in that, Includes the following steps: The system acquires input video frames of the robot's working scene, extracts objects from the input video frames to obtain object representations of each object in the scene, generates an environment query vector based on continuous multi-frame observations of the input video, and performs similarity matching retrieval between the environment query vector and the discrete environment codebook to obtain a global environment representation. The object representation and the global environment representation are input into a multi-layer sparse causal state space model. Each layer of the sparse causal state space model includes an independent evolution layer and a causal interaction layer. Based on the independent evolution layer and the causal interaction layer, the current layer's fused state representation is output, and a single-layer causal adjacency matrix of the current layer is generated. The fused state representation is used as the input of the next layer's sparse causal state space model, and the hierarchical deduction is completed layer by layer. The single-layer causal adjacency matrices of each layer are superimposed with the identity matrix and then multiplied layer by layer to generate a global causal matrix that accumulates multiple causal propagation paths. A joint loss function containing video reconstruction loss and sparse regularization loss is constructed, and a sparse causal state space model and an image decoder are trained based on the joint loss function. The fused state representation, which is the final output of the sparse causal state space model, is input into the image decoder to generate future video prediction frames of the scene, and an explicit causal structure graph between objects and the environment is output based on the global causal matrix.
2. The method for constructing an embodied intelligent world model according to claim 1, characterized in that, The process of extracting objects from input video frames to obtain object representations of each object in the scene includes: A pre-trained instance segmentation network or unsupervised object discovery module is used to obtain image regions of each object, and the image regions are encoded into object representations corresponding to the objects based on a variational autoencoder.
3. The method for constructing an embodied intelligent world model according to claim 1, characterized in that, The environment query vector is obtained by encoding the motion trajectory features of each object in the continuous multi-frame sequence.
4. The method for constructing an embodied intelligent world model according to claim 1, characterized in that, The process of outputting the current layer's fused state representation based on the independent evolution layer and the causal interaction layer includes: The independent evolution layer performs parallel temporal deduction of the object representation and the global environment representation based on the state space model to obtain the independent evolution state corresponding to each object and the global environment; the causal interaction layer calculates the correlation weight between each object based on all the independent evolution states, generates a single-layer causal adjacency matrix, and weights and aggregates each of the independent evolution states based on the single-layer causal adjacency matrix to output the current layer fusion state representation.
5. The method for constructing an embodied intelligent world model according to claim 4, characterized in that, The process of calculating the association weights between objects based on all the independent evolution states and generating a single-layer causal adjacency matrix in the causal interaction layer includes: All independent evolutionary states are mapped to query vectors, key vectors, and value vectors through linear projection. The attention score between any two independent evolutionary states is calculated as the association weight to obtain a single-layer causal adjacency matrix. The calculation formula is as follows: ； in, Indicates from object To the object The strength of causal influence The Sigmoid activation function is used to map attention scores to the [0, 1] interval to characterize the strength of causal influence; For the first The query vector corresponding to each independent evolutionary state For the first The key vector corresponding to each independent evolutionary state For vector dimensions.
6. The method for constructing an embodied intelligent world model according to claim 5, characterized in that, In the process of weighted aggregation of each independent evolutionary state based on the single-layer causal adjacency matrix and outputting the current layer fusion state representation, the first... The updated state after weighted aggregation of individual evolutionary states is represented as follows: ； in, For the first An object at time... The representation vector, The first output of the independent evolution layer The independent evolutionary state of an object For the single-layer causal adjacency matrix, from the object To the object The strength of causal influence For the first The value vector corresponding to each independent evolutionary state This is a multilayer perceptron used for nonlinear transformations of value vectors, with a summation term. Represents all objects The sum of the weighted effects of neighboring objects that have a causal influence on it.
7. The method for constructing an embodied intelligent world model according to claim 6, characterized in that, The expression for calculating the global causal matrix is as follows: ； Among them, matrix elements in Indicates from object To the object The total number of existing causal paths The total number of layers in the sparse causal state-space model. For the first A single-layer causal adjacency matrix of layers. It is the identity matrix. Indicates the first Self-loops are introduced into the layers to preserve the state information of each object.
8. The method for constructing an embodied intelligent world model according to claim 7, characterized in that, The expression for the joint loss function is: ； in, It is the image generated by the decoder based on the prediction slot. For the actual next moment's video frame, These are sparse regularization weights, used to balance the relative importance of reconstruction loss and sparse regularization loss. is the L1 norm of the matrix, used to constrain the sparsity of the global causal matrix.
9. The method for constructing an embodied intelligent world model according to claim 8, characterized in that, During the training of the sparse causal state-space model and image decoder based on the aforementioned joint loss function, the sparse regularization weights are dynamically adjusted, specifically including: In the early stages of training, set relatively small sparse regularization weight values so that the model learns basic motion patterns primarily through reconstruction loss. As the number of training rounds increases, the sparse regularization weight value is gradually increased according to the decreasing trend of the reconstruction error to enhance the sparsity constraint strength, so that the model can gradually eliminate redundant causal connections and automatically discover the real physical interaction structure.
10. A system for constructing an embodied intelligent world model, characterized in that, It includes a representation extraction module, a representation fusion module, a model optimization module, and a prediction output module, among which: The representation extraction module is used to acquire input video frames of the robot's working scene, extract objects from the input video frames to obtain object representations corresponding to each object in the scene, generate environment query vectors based on continuous multi-frame observations of the input video, and perform similarity matching retrieval between the environment query vectors and the discrete environment codebook to obtain global environment representations. The representation fusion module is used to input the object representation and the global environment representation into a multi-layer sparse causal state space model. Each layer of the sparse causal state space model includes an independent evolution layer and a causal interaction layer. Based on the independent evolution layer and the causal interaction layer, the current layer's fused state representation is output. The fused state representation is used as the input to the next layer's sparse causal state space model, and the hierarchical deduction is completed layer by layer. The model optimization module is used to multiply the single-layer causal adjacency matrix of each layer by the identity matrix and generate a global causal matrix that accumulates multiple causal propagation paths. It constructs a joint loss function that includes video reconstruction loss and sparse regularization loss, and trains a sparse causal state space model and an image decoder based on the joint loss function. The prediction output module is used to input the fused state representation of the sparse causal state space model into the image decoder, decode it to generate a future video prediction frame of the scene, and output an explicit causal structure graph between the object and the environment based on the global causal matrix.