Causal alignment-based embodied intelligent robot arm world model learning method and system

Abstract

The application provides a somatic intelligent mechanical arm world model learning method and system based on causal alignment, through multi-modal alignment, information that can be perceived, executed and described in the operation process of the mechanical arm is uniformly mapped to the same hidden space; a directed acyclic graph constraint is explicitly introduced in the hidden space, and causal dependency relationships between objects, contacts, force feedback and pose changes are learned, so that the world model has intervention and counterfactual reasoning (i.e. on the basis of observed facts, answering the hypothetical question of "how will the result be if a certain condition is changed at that time") ability; finally, by using stage-by-stage training and multi-step rolling prediction, the model not only retains the fact prediction accuracy, but also improves the generalization ability under the conditions of distribution shift, task switching and disturbance. The application effectively solves the problems of poor generalization ability, unexplainability and lack of causal reasoning ability of the existing world model, and can significantly improve the success rate and robustness of the real mechanical arm in complex operation tasks.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to an embodied intelligent robotic arm world model learning method and system, electronic device and computer-readable storage medium based on causal alignment. Background Technology

[0002] With the rapid development of embodied intelligence technology, world models have become the core foundation for robots to achieve autonomous decision-making, environmental interaction, and task planning. By learning the dynamic laws of the environment, world models can predict future state changes, enabling robots to perform pre-planning and decision-making without extensive real-world interaction, significantly reducing the trial-and-error costs in the real world. However, existing world models are generally based on pure statistical correlation, which has the following fundamental flaws: Lacking causal structure and with poor generalization ability: Most existing world models predict the future by fitting statistical patterns from large amounts of observational data, failing to distinguish between causal relationships and spurious correlations. When environmental distribution shifts, task switching, or unseen disturbances occur, the model's predictive performance drops sharply, making it unable to adapt to the dynamic changes in real-world open scenarios.

[0003] Multimodal representations are misaligned, and semantic entanglement is severe: Existing methods typically encode multimodal data such as vision, touch, and action separately, lacking an effective cross-modal semantic alignment mechanism. Representations of different modalities are entangled in the latent space, failing to form a unified and interpretable representation of the environment state, making it difficult for the model to understand the intrinsic connections between information from different modalities.

[0004] Modeling context-independent elements leads to computational inefficiency: Existing world models attempt to model all pixels and states in the environment, including a large amount of static background and redundant dynamics that are irrelevant to agent control. This not only increases the computational burden but also introduces unnecessary noise, reducing the accuracy of predictions relevant to core tasks.

[0005] Unable to intervene or perform counterfactual reasoning: Pure statistical models lack causal reasoning capabilities, cannot answer counterfactual questions such as "What will happen if a certain action is changed?", and cannot predict changes in the state of the environment after human intervention. This prevents robots from making effective decision adjustments when faced with abnormal situations, limiting their application in complex tasks.

[0006] The prediction results are uninterpretable and difficult to debug and optimize: the prediction process of existing world models is a black box, making it impossible to trace the causes of the prediction results. When the model makes incorrect predictions, engineers find it difficult to locate the problem and optimize the model accordingly, which seriously affects the reliability and maintainability of the system.

[0007] Unstable training and accumulation of prediction errors over long time: Existing world models typically employ end-to-end joint training, which is prone to gradient vanishing or exploding problems, resulting in poor training stability. Furthermore, during multi-step rolling prediction, errors accumulate rapidly with the number of prediction steps, leading to a sharp decline in long-term prediction accuracy.

[0008] These shortcomings severely restrict the practical application of world models in real-world robotic arm scenarios, especially in tasks requiring high precision and robustness, such as grasping, pushing and pulling, insertion, and precision assembly. Therefore, there is an urgent need for a causal world model that can learn causal structures, achieve multimodal alignment, focus on controllable factors, and possess intervention and counterfactual reasoning capabilities to enhance the environmental understanding and decision-making abilities of embodied intelligent robots. Summary of the Invention

[0009] To address the technical problems existing in the prior art, the present invention provides the following technical solution: On the one hand, a world model learning system for embodied intelligent robotic arms based on causal alignment is provided, including: The multimodal encoder module is used to encode visual images, tactile signals, control actions, and task language descriptions respectively, generating a unified latent space state representation. The causal branching module, connected to the multimodal encoder module, is used to apply structural causal regularization to the original latent representation, learn the causal dependencies between variables, and output the causally aligned structured latent representation. An action condition transition module, connected to the causal branch module, is used to predict the hidden state at the next moment based on the current causal hidden state and the action input. The training control unit is connected to the three modules mentioned above and is used to execute the three-stage training process, gradually introducing causal constraints and optimizing model parameters.

[0010] Preferably, the multimodal encoder module includes a visual encoder, a tactile encoder, a motion encoder, and a speech encoder, as well as a feature fusion unit; the feature fusion unit integrates the feature vectors of the four modalities into an object-centered latent space representation. ,in For the number of object slots, The feature dimension for each object slot.

[0011] Preferably, the multimodal encoder module is jointly trained using cross-modal contrastive loss, temporal consistency loss, and state alignment loss; the cross-modal contrastive loss is used to narrow the distance between different modal representations at the same time, the temporal consistency loss is used to ensure the smoothness of latent representations at adjacent time points, and the state alignment loss is used to anchor latent variables to interpretable physical states.

[0012] Preferably, the system further includes a controllable factor screening unit, used to calculate the comprehensive controllable score for each object slot and screen out the top-ranked factors. Controllable factors are incorporated into the causal graph learning; the comprehensive controllable score is a weighted sum of motion sensitivity, contact change, language matching degree, and pose change.

[0013] Preferably, the causal branching module internally includes a structural causal matrix learning unit, a DAG constraint unit, and an alignment mapping head; the structural causal matrix Used to represent the strength of causal influence between variables, the DAG constraint unit ensures that the causal graph is a directed acyclic graph through acyclic constraint loss. The DAG acyclic constraint loss is:

[0014] in For matrix trace operations, For matrix exponentiation operations, For element-wise multiplication, For causal variables.

[0015] Preferably, the three-stage training process executed by the training control unit includes: Phase 1: Freeze causal branches, train the multimodal encoder and action conditional transition module, and optimize the one-step prediction loss; Phase 2: Freeze the multimodal encoder and action conditional transition module, train the causal branch, and optimize the joint loss function including reconstruction loss, KL divergence, state alignment loss, DAG constraint loss, and mask loss; Phase 3: Freeze the multimodal encoder and causal branches, fine-tune the action conditional transfer module, and optimize the multi-step rolling loss.

[0016] Preferably, in Phase 3 training, an Alpha-gated fusion mechanism is used to weightedly mix the original latent representation with the causal regularized latent representation as the input to the transition module: in The mixing coefficients are determined using an exponential decay strategy. The output of the causal branch, This refers to the fused vector features output by the multimodal encoder.

[0017] On the other hand, a world model learning method for embodied intelligent robotic arms based on causal alignment is provided, including the following steps: S1: Collect multimodal data during the operation of the robotic arm, including visual images, tactile signals, control actions, and task language descriptions; S2: Encode and fuse multimodal data to generate a unified latent space state representation; S3: Screen controllable factors, establish a structural causal matrix, and learn the causal dependencies between variables; S4: Perform a three-stage training process to optimize model parameters; S5: Verify the fact prediction performance and counterfactual prediction performance of the model; S6: Connect the trained world model to the robotic arm control stack to achieve closed-loop control.

[0018] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above method.

[0020] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. Significantly improved counterfactual prediction capabilities: By introducing causal structure constraints, the model's predictive performance in intervention and counterfactual scenarios has achieved a qualitative leap. On the Physics benchmark, counterfactual retrieval performance has been improved by an average of 102.5%, solving the problem of poor generalization ability of existing pure statistical models.

[0021] 2. Achieve multimodal semantic alignment: By using cross-modal contrastive loss and state alignment loss, visual, tactile, action and language information are uniformly mapped to the same latent space, forming a structured representation with clear physical semantics, thus solving the problem of multimodal representation entanglement.

[0022] 3. Focus on controllable factors to improve modeling efficiency: Through a controllable score screening mechanism, static backgrounds and redundant dynamics that are irrelevant to control are automatically filtered out, and modeling resources are concentrated on the core factors that the robotic arm can intervene in, improving computational efficiency by more than 30% and significantly improving the prediction accuracy of core tasks.

[0023] 4. Interpretability and ease of debugging and optimization: The causal graph learned by the model can intuitively show the dependencies between variables, and the prediction results can be traced back to specific causal factors. When the model encounters errors, engineers can quickly locate the problem by analyzing the causal graph, greatly improving the maintainability of the system.

[0024] 5. Pluggable design, highly versatile: The causal branch, as an independent module, can be flexibly connected to various mainstream encoder-transfer architectures without redesigning the backbone network. This allows the invention to quickly adapt to different application scenarios and hardware platforms, exhibiting strong scalability.

[0025] 6. Three-stage training for high stability: The phased training strategy gradually introduces causal constraints, effectively avoiding gradient conflicts and instability issues during joint training. Model training convergence speed is improved by 2 times, resulting in more stable and reliable final performance.

[0026] 7. Supports closed-loop control for real robotic arms: The causal implicit representations output by the model can be directly decoded into the state variables required for robotic arm control, possessing intervention and counterfactual reasoning capabilities, and providing reliable support for decision-making and planning. In real robotic arm tasks such as grasping and insertion, the success rate is improved by more than 20%, and the robustness to disturbances is significantly enhanced. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a model architecture diagram provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a three-stage supervised training process provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating a method implementation provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0030] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0031] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0032] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0033] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0034] I. Purpose of this technology This invention proposes a world model learning method and system for embodied intelligent robotic arms based on causal alignment. First, through multimodal alignment of vision, touch, action, and language, perceptible, executable, and describable information during robotic arm operation is uniformly mapped to the same latent space (i.e., a low-dimensional continuous vector space obtained by compressing high-dimensional observation data, where each vector corresponds to a compact representation of an environmental state). Second, a Directed Acyclic Graph (DAG) constraint is explicitly introduced into the latent space to learn the causal dependencies between objects, contact, force feedback, and pose changes, enabling the world model to possess intervention (i.e., artificially changing the value of a variable to observe its impact on downstream variables) and counterfactual reasoning (i.e., answering hypothetical questions such as "what would have happened if a certain condition had been changed?" based on observed facts). Finally, by utilizing phased training and multi-step rolling prediction, the model retains factual prediction accuracy while improving generalization ability under distribution shifts, task switching, and perturbation conditions. This invention effectively solves the problems of poor generalization ability, lack of interpretability, and lack of causal reasoning ability in existing world models, and can significantly improve the success rate and robustness of real robotic arms in complex operation tasks.

[0035] II. Innovative Technical Path of the Invention This invention proposes a method for constructing a causal world model oriented towards embodied intelligence. The core approach no longer involves averaging all pixels or states of the environment, but rather revolves around three main lines: "controllable factors – causal structure – future state prediction." Compared with existing world models, it has the following four core innovations: 1. Focus on controllable factors and filter out irrelevant redundancy: Instead of modeling all elements of the environment indiscriminately, it automatically filters out environmental factors that the agent can control and intervene in (such as object pose, contact state, force, etc.) through multimodal feature analysis, and filters out static background and uncontrollable noise, which greatly improves modeling efficiency and prediction accuracy of core tasks.

[0036] 2. Pluggable causal module, adaptable to multiple architectures: The causal representation learning module is designed as an independent, pluggable structure, located between the encoder output and the transfer module input, without changing the basic structure of the backbone network. It can flexibly adapt to various mainstream encoder-transfer architectures such as graph neural networks (GNN), object slots, and variational autoencoders (VAE), and has extremely strong versatility and scalability.

[0037] 3. Three-level training path to ensure temporal stability: A phased training strategy of "multimodal alignment - causal structure learning - multi-step fine-tuning" is adopted to gradually introduce causal constraints. First, basic predictive capabilities are learned, then the causal structure in the data is mined, and finally the long-term prediction performance is fine-tuned, effectively solving the problem of instability in joint training.

[0038] 4. Real-world application-oriented design: The causal latent representations output by the model have clear physical semantics and can be directly decoded into the state variables required for robotic arm control. Simultaneously, the model possesses intervention and counterfactual reasoning capabilities, providing reliable predictive support for the robotic arm's decision-making and planning. Its effectiveness has been verified in real-world tasks such as grasping, pushing and pulling, insertion, and precision assembly.

[0039] III. Core Content of the Invention - Multimodal Alignment Algorithm and Feature Factor Capture Method This invention transforms high-dimensional, heterogeneous multimodal observation data into a unified, compact, and controllable latent space representation by three steps: multimodal coding, cross-modal alignment, and controllable factor screening.

[0040] 1) Multimodal coding.

[0041] At every moment The system processes visual images separately. tactile signals Controlling actions and task language description Feature extraction is performed using their respective encoders: Visual encoder Convolutional Neural Networks (CNNs) are used to map images into visual feature vectors. ; tactile encoder Mapping tactile array signals (such as contact pressure distribution and slip vector) into tactile feature vectors ; Motion encoder Map control actions (such as end-effector pose increments and gripper opening / closing amounts) to action feature vectors. ; Language encoder Mapping task instruction text to language feature vectors .

[0042] Subsequently, the four feature vectors mentioned above are integrated into a unified latent space state representation through a fusion module (such as by concatenating them and then passing them through a multilayer perceptron, or by using a cross-attention mechanism).

[0043] in This refers to the number of object slots (i.e., the number of objects in the scene that can be tracked independently). The feature dimension for each object slot.

[0044] 2) Alignment of Targets. A joint training approach using cross-modal contrastive loss and temporal consistency loss is employed to bring visual, tactile, motor, and linguistic representations of the same operational segment closer together in the latent space, while separating representations of different operational segments. Specifically, the cross-modal contrastive loss is defined as:

[0045] in The cosine similarity function is used. For the temperature hyperparameter, the denominator is summed over all samples in the same batch. This loss function aims to shorten the distance between representations of different modalities at the same time step and widen the distance between representations at different time steps, thereby achieving cross-modal semantic alignment. Temporal consistency loss ensures smooth changes in latent representations between adjacent time steps.

[0046] In addition, a state alignment loss is introduced to anchor latent variables to interpretable physical states (such as pose coordinates and contact forces), defined as:

[0047] in To align the mapping header (implemented by two layers of fully connected networks). Provides the simulator with a real physical state vector (including the position, velocity, contact force, etc. of each object). This is the latent representation after causal branching. The loss is anchored to the latent space coordinate system during training, but not required during inference. .

[0048] 3) Controllable Factor Capture. To focus on the environmental factors controllable by the robotic arm, the system calculates the comprehensive controllability score for each object slot and selects the top-ranked factors. Factors are incorporated into the cause-and-effect diagram learning: For each latent variable or object slot Define the action sensitivity score:

[0049] In the formula: That is, the implicit representation of the Jacobian norm with respect to the action input measures the degree of responsiveness of the object slot to changes in the control action, and combines the contact change index. (Amplitude of tactile signal change), language target word matching degree (Semantic similarity between language instructions and the object slot representation) and visual pose change indicators (The pose offset detected by vision) is used to calculate the overall controllability score:

[0050] in to Preset weights.

[0051] The system selects the top performers based on their overall controllable scores. By incorporating factors into the causal graph learning, attention is focused on "the objects, contact points, and constraints that the robotic arm can truly change," while filtering out static backgrounds and uncontrollable environmental noise.

[0052] 4) Causal abstraction.

[0053] Establish a structural causal matrix for the screened controllable factors. ,in Indicates the first The causal variable affects the first The influence strength of each causal variable, with non-zero elements corresponding to directed edges in the causal graph.

[0054] To ensure the rationality of the causal structure, the following loss function is introduced: Add acyclic constraint loss to ensure The corresponding graph is a directed acyclic graph (DAG):

[0055] in For matrix trace operations, For matrix exponentiation operations, This is the element-wise product (Hadamard product). For causal variables, the dimension is [the causal variable dimension]. When... Time indicates Strictly satisfies the acyclicity requirement.

[0056] Simultaneously, masked loss is utilized (to verify the necessity of causal edges through masking operations):

[0057] Compared to reconstruction losses (ensuring that key information is not lost in causal branches):

[0058] in This is the KL divergence (an asymmetric measure of the difference between two probability distributions). The first one after DAG and mask transformation Approximate posterior distribution of each concept component For the corresponding state condition prior distribution, For masked branch predictor, These are conceptual-level supervision signals derived from available state labels. Together, these losses enable the model to express patterns such as "changes in gripping force lead to changes in object posture" and "changes in end-effector trajectory lead to switching of contact states" as interpretable causal edges. These losses collectively enable the model to learn causal dependencies that conform to physical laws rather than purely statistical correlations.

[0059] IV. Model Architecture and Supervised Training Process like Figure 1 As shown, the model architecture of this invention consists of three core modules working together: encoder module Causal branching module and action condition transfer module These three elements form a complete "observation → causal representation → future prediction" link in the latent space:

[0060] 1. Encoder Module Responsible for multimodal observation Mapped to object-centric hidden state It can adopt various architectures such as CNN, GNN, VAE or object slot, and has strong versatility.

[0061] 2. Causal Branching Module right Apply structural causal regularization and output the causally aligned latent representation. ; 3. Action Condition Transfer Module Predict the hidden state at the next time step based on the current causal hidden state and the action input. .

[0062] The transition module uses residual dynamic parameterization to predict state increments rather than absolute states, which is beneficial for long-term stability. .

[0063] Among them, the causal branch, as a pluggable module, is located between the encoder output and the transfer module input. It does not change the basic structure of the backbone network, so it can flexibly adapt to different types of encoder-transfer architectures (such as GNN backbone based on graph network, Modular backbone based on modular decomposition, AE backbone based on autoencoder or VAE backbone based on variational autoencoder).

[0064] The interaction process between the three modules is as follows: the encoder receives multimodal observations and generates raw latent representations; the causal branch performs causal structuring on the raw latent representations, removing spurious correlations; and the transition module predicts future states based on causal representations and actions. This architecture decouples perception, causal reasoning, and prediction, facilitating the individual optimization of each module.

[0065] (II) Three-stage supervised training process This invention employs a phased training strategy, gradually introducing causal constraints, which effectively solves the problem of instability in joint training. Each phase has a clearly defined optimization target and loss function.

[0066] like Figure 2 As shown, the training process is divided into three stages, each with a clearly defined optimization target and loss function: 1. Phase One (Main Intervention Training) Optimization target: Encoder and transfer module ; Frozen Module: Causal Branches (Not yet activated); Training encoder and transfer module Freeze the causal branches (at this point, the causal branches are not yet activated). Optimize the one-step prediction loss using visual image streams, image-action streams, or image-haptic-action streams as input:

[0067] in Encoding for the observation at the next time step, For model prediction.

[0068] loss function It can be instantiated in one of the following two forms: Comparative loss:

[0069] in These are mismatched future hidden states (negative samples) randomly sampled from the same batch. The interval boundary is defined. This loss narrows the gap between correct predictions and pushes away incorrect matches.

[0070] Negative log-likelihood loss:

[0071] Assume that the isotropic Gaussian decoder in the latent space is equivalent to the scaled mean square error.

[0072] The goal of this stage is to obtain stable one-step prediction capabilities, providing a reliable latent space representation foundation for subsequent causal structure learning. The connection between this stage and the latent space lies in the fact that after the encoder compresses high-dimensional observations into the latent space, the transfer module learns dynamic prediction patterns within this latent space, ensuring that the latent space representation has sufficient information preservation and predictive distinguishability.

[0073] Phase Two (Learning Causal Structures) Freeze encoder and transfer module (That is, the parameters learned in phase one are no longer updated), and only the causal branches are trained. and its related parameters (including DAG matrix) and alignment mapping header ).

[0074] Optimization target: causal branches and its related parameters (including DAG matrix) and alignment mapping header ); Freeze module: Encoder and transfer module ; The optimized joint loss function is:

[0075] The definitions of each item are as follows: (a) Reconstruction losses : Requires the output of causal branches Faithful to the input in terms of information This prevents causal regularization from excessively distorting the original representation. Its operating mechanism is to minimize the Euclidean distance between the latent vectors before and after the causal transformation, ensuring that the causal branches do not lose the crucial information needed for prediction.

[0076] (b) KL divergence regularization term: , in: Causal branching from encoder hidden representation Induced approximate posterior distribution (i.e., the probability estimate of the model for the values ​​of latent variables). For the standard normal prior, The divergence measure is the difference between two probability distributions. The first term normalizes the distribution of global latent variables, and the second term aligns the conceptual components after the DAG transformation with the prior state conditions. represents the KL weighting coefficient. This term helps prevent posterior distribution collapse and ensures that each dimension of the latent variable has interpretable semantic meaning.

[0077] (c) State alignment loss As described in Part II, latent space coordinates are anchored to physically interpretable state variables. When the simulator state... When unavailable, set Skip this item.

[0078] (d) Acyclic constraint of DAG : Utilizing the trace operation of matrix exponents provides a penalty for continuous differentiability and acyclicity. When When the corresponding graph contains directed cycles, If and only if When it is a directed acyclic graph, This constraint ensures that the learned causal structure mathematically satisfies the basic requirements of a causal model.

[0079] (e) Masking loss By applying a mask to some causal paths (i.e., temporarily setting certain edges in a DAG to zero), we can verify whether the remaining structure can still recover the correct concept prediction, thereby reinforcing the necessity rather than the redundancy of causal edges.

[0080] Weighting coefficients to The selection is made through a grid search on the validation set. The connection between this stage and the latent space is that, based on the latent space established in stage one, the causal branch applies structural constraints within the latent space, reorganizing the originally entangled predictive representations into structured representations that satisfy causal semantics, while fixing the latent space coordinate system to a physically interpretable reference frame through alignment loss.

[0081] 3. Phase Three (Multi-step Rolling Fine-tuning) Optimization target: Transfer module ; Freeze module: Encoder and causal branches ; Freeze encoder and causal branches Only fine-tuning the transfer module .

[0082] To balance causal regularization and prediction accuracy, an Alpha-gated fusion mechanism is employed to weightedly mix the original latent representation with the causal regularized latent representation as input to the transition module.

[0083] in For the mixing coefficient, an exponential decay strategy is adopted:

[0084] For the initial fusion weights, The decay rate. Data dependency gating can be optionally applied:

[0085] in For the sigmoid function, and This is a gated hyperparameter that automatically reduces the fusion ratio to protect prediction stability when the causal bias is too large.

[0086] The transfer module uses residual dynamic parameterization:

[0087] Predicting state increments rather than absolute states is beneficial for long-term stability. Multi-step rolling prediction is achieved through recursive expansion:

[0088] The optimization objective is to reduce the multi-step rolling loss.

[0089] in: To predict the time domain length during rolling. A course learning strategy is used during training (initially using shorter prediction steps, gradually increasing to the target number of steps). A hybrid strategy (randomly sampling different prediction steps in different batches) can be used to improve long-term prediction stability. The goal of this stage is to adapt the transition module to the latent representation input after causal regularization, thereby enhancing prediction robustness in counterfactual scenarios while maintaining factual prediction accuracy.

[0090] V. Model Validation, Testing, and Deployment 1. Validation Phase: Using simulated or offline datasets, and dividing the dataset into training / validation / test sets, evaluate performance on one-step prediction, multi-step prediction (assessed in the time domain at 1, 5, and 10 steps respectively), contact recognition, operation success rate, and counterfactual retrieval. Evaluation metrics include: (Hit rate @1): , That is, the proportion of the model that prioritizes the correct future.

[0091] (Average last ranking): That is, the average of the reciprocals of the correct future ranking.

[0092] and : Defined the same as factual metrics but applied to counterfactual future retrieval tasks, where Replace with counterfactual ranking .

[0093] 2. Testing Phase: Apply intervention operations to the selected state variables:

[0094] That is, artificially making the first The first object offset of each state component Quantities (e.g., changing the grasping pose by 3cm, rotating the force direction by 15°, increasing the end effector velocity by 0.1m / s, or translating the initial position of the target object by 5cm) are used to obtain the true future state after intervention. As a baseline, the consistency between the counterfactual future states predicted by the model and the actual outcomes is compared to verify the causal generalization ability. Intervention tests cover single-objective interventions and multi-objective combined interventions, and robustness is assessed at small, medium, and large intervention magnitudes.

[0095] 3. Deployment Phase: The trained world model is connected to the real robotic arm control stack to form a complete closed-loop control system of "perception-prediction-decision-execution".

[0096] like Figure 3 As shown, the specific application control process is as follows: Step D1 (Multimodal Sensing Acquisition): In each control cycle (typical frequency 10–30Hz), the system synchronously reads the visual images from the RGB-D camera mounted above the robotic arm's workspace. Contact distribution signals of the tactile sensor array mounted on the fingertips of the grippers The current end-effector pose and joint status information are provided by the robotic arm joint encoder and torque sensor.

[0097] Step D2 (Latent Space Encoding and Causal Representation Inference): Input the collected multimodal data into the trained encoder. and causal branches The object-level hidden representation is obtained sequentially. Structured latent representation aligned with causality .at this time Each dimension in the model already possesses interpretable causal semantics (such as the pose, contact state, or motion trend of a specific object).

[0098] Step D3 (Candidate Action Generation and Future State Prediction in Parallel): Planner or Policy Network generates... Candidate action plan (e.g., different gripping angles, approach paths, or gripper force), for each candidate action Using the transfer module implement Step-by-step rolling prediction:

[0099] Obtain the future corresponding to each action plan Hidden trajectory .

[0100] Step D4 (Risk Assessment and Action Selection): For the predicted trajectory of each candidate action, the system decodes the physical state prediction from the causal latent representation (by aligning the mapping head). The following safety and quality metrics were evaluated: (a) Grasping stability metrics: whether the predicted tactile contact distribution is within the effective grasping area and whether the contact force is within the safety threshold; (b) Collision risk metrics: whether the minimum distance between the predicted object trajectory and environmental obstacles meets the safety margin; (c) Goal achievement metrics: whether the error between the predicted final object pose and the target pose is within the allowable range; (d) Causality consistency metrics: utilizing the learned causal graph. The changes in each causal variable in the predicted trajectory are examined to ensure they conform to known causal dependencies (e.g., if no force is applied at the end point, the object should not undergo non-inertial motion). Based on these indicators, the action plan with the lowest risk and highest probability of success is selected for execution.

[0101] Step D5 (Execution and Feedback Correction): The selected motion command is sent to the robotic arm controller for execution. During execution, the system continuously collects new multimodal observations and compares them with the model predictions in real time. If the actual observation deviates from the prediction by more than a preset threshold (e.g., sudden change in actual contact force, accidental slippage of the object), online replanning is triggered: based on the latest observations, the system re-encodes and predicts, updates the subsequent motion sequence, and achieves adaptive closed-loop correction control.

[0102] The above deployment process can significantly improve grasping stability, insertion success rate, and adaptability to new objects in complex scenarios. In particular, because causal branches provide an interpretable causal structure, the system can retrospectively trace "which object and which contact factor caused the predicted change," facilitating fault diagnosis and strategy optimization.

[0103] VI. Model Validation Experiment This invention has been comprehensively tested on multiple benchmark datasets such as Physics, Chemistry, 2D Shapes, and 3D Cubes. Experimental results show that after introducing the causal structure module, while maintaining the basic stability of fact prediction performance, the model has a more significant improvement in counterfactual / intervention evaluation.

[0104] Taking the Physics benchmark as an example, the average CF-H@1 of the 8 paired baselines improved by about 102.5%; among them, GNN-NLL improved from 11.0 to 41.0 (an improvement of +272.7%), and AE-NLL improved from 10.5 to 41.0 (an improvement of +290.5%), indicating that the scheme has a significant improvement on future state retrieval and causal consistency under intervention.

[0105] The fact prediction performance remained relatively stable, and some methods even saw improvements (such as AE_Contrastive's 1-step H@1 improving from 89.00 to 98.00), indicating that causal constraints did not impair the model's basic predictive ability.

[0106] Ablation experiments further demonstrate that three-stage training outperforms joint training without stages. On the Physics benchmark, the three-stage scheme achieves CF-H@1 of 31.00, higher than the joint training's 28.00, and significantly better than the base GNN_Contrastive model's 11.50. Removing causal branches, state alignment, or gating fusion reduces both the model's factual prediction and counterfactual robustness. These results support the claim that the algorithm described in this invention has clear technical effectiveness, and is not merely a conceptual description.

[0107] The following table provides detailed validation data, showing a quantitative comparison of fact prediction performance and counterfactual prediction performance.

[0108] Table 1: Performance Comparison of Fact Prediction (H@1, format: "baseline / baseline+CausalVAE") Physics 3-body AE_Contrastive 89.00 / 98.00 10.00 / 9.00 1.00 / 3.00 AE_NLL 92.00 / 94.00 4.00 / 4.00 0.00 / 1.00 VAE_Contrastive 70.00 / 91.00 1.00 / 3.00 2.00 / 2.00 VAE_NLL 92.00 / 94.00 1.00 / 2.00 0.00 / 0.00 Modular_Contrastive 90.00 / 86.00 20.00 / 8.00 3.00 / 2.00 Modular_NLL 96.00 / 95.00 11.00 / 11.00 1.00 / 3.00 GNN_Contrastive 88.00 / 99.00 25.00 / 36.00 3.00 / 7.00 GNN_NLL 98.00 / 99.00 9.00 / 17.00 7.00 / 5.00 Table 2: Counterfactual Prediction Performance Comparison (Format: "Baseline / Baseline + CausalVAE") Physics 3-body AE_Contrastive 10.00 / 24.00 +14.00 49.33 / 51.77 AE_NLL 10.50 / 41.00 +30.50 52.25 / 66.92 VAE_Contrastive 8.50 / 8.50 – 46.00 / 45.40 VAE_NLL 10.50 / 13.00 +2.50 51.75 / 51.00 Modular_Contrastive 22.50 / 25.00 +2.50 59.83 / 50.62 Modular_NLL 14.50 / 22.00 +7.50 55.83 / 57.50 GNN_Contrastive 11.50 / 15.00 +3.50 52.96 / 53.17 GNN_NLL 11.00 / 41.00 +30.00 53.50 / 68.88 Chemistry VAE_Contrastive 15.38 / 24.61 +9.23 33.53 / 48.24 VAE_NLL 15.50 / 25.26 +9.76 33.62 / 45.80 Modular_Contrastive 13.05 / 27.49 +14.44 33.22 / 48.67 GNN_Contrastive 13.54 / 25.76 +12.22 32.96 / 47.66 2D Shapes AE_NLL 65.75 / 67.82 +2.07 80.22 / 81.64 Modular_NLL 62.80 / 68.35 +5.55 78.46 / 81.88 GNN_NLL 33.00 / 54.75 +21.75 58.81 / 74.00 3D Cubes AE_NLL 39.90 / 41.65 +1.75 60.41 / 63.11 Modular_Contrastive 7.90 / 16.75 +8.85 19.07 / 33.80 Table 3: Ablation Experiment Results (Physics Benchmark) Three-stage training (this invention) 91.00 95.16 31.00 52.15 Joint training (without phase division) 90.33 94.75 28.00 51.05 Baseline (GNN_Contrastive) 88.00 93.37 11.50 52.96 Remove CausalVAE branch 80.00 89.55 27.67 50.06 Remove state alignment loss 9.00 28.02 26.67 50.57 Single-step scrolling only 90.67 94.91 30.67 52.31 Remove contrast loss 89.33 94.36 29.67 51.79 Close gating fusion 1.33 5.71 13.67 28.09 Phase division: s1_8_s2_40 91.00 95.08 28.67 51.45 Rolling Strategy: Course Learning 91.00 95.06 31.33 52.32 Rolling Strategy: Hybrid 90.00 94.75 29.67 51.88 Based on Tables 1-3, the present invention has at least the following technical effects: 1) Reduce reliance on full-scale supervised data and scene traversal, and reduce the need for pure statistical fitting by using causal structure constraints; 2) Improve the model's generalization ability under conditions of task switching, object change, and contact perturbation. Experiments show that the counterfactual retrieval performance is improved by an average of 102.5% on the Physics benchmark. 3) Improve the interpretability of world model predictions by tracing "which object / contact factor caused the predicted change" through the learned causal graph, and the causal discovery results are consistent with the interaction template of the first-order linearization of the physical system; 4) It supports closed-loop decision-making for real robotic arms in tasks such as grasping, pushing, pulling, inserting, and sorting, and has engineering value that can be deployed, verified, and iteratively optimized.

[0109] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0110] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0111] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0112] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0113] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0115] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0118] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A world model learning system for an embodied intelligent robotic arm based on causal alignment, characterized in that, include: The multimodal encoder module is used to encode visual images, tactile signals, control actions, and task language descriptions respectively, generating a unified latent space state representation. The causal branching module, connected to the multimodal encoder module, is used to apply structural causal regularization to the original latent representation, learn the causal dependencies between variables, and output the causally aligned structured latent representation. An action condition transition module, connected to the causal branch module, is used to predict the hidden state at the next moment based on the current causal hidden state and the action input. The training control unit is connected to the three modules mentioned above and is used to execute the three-stage training process, gradually introducing causal constraints and optimizing model parameters.

2. The system according to claim 1, characterized in that, The multimodal encoder module includes a visual encoder, a tactile encoder, a motion encoder, and a speech encoder, as well as a feature fusion unit; The feature fusion unit integrates the feature vectors of the four modalities into an object-centered latent space representation. ,in For the number of object slots, The feature dimension for each object slot.

3. The system according to claim 2, characterized in that, The multimodal encoder module is jointly trained using cross-modal contrastive loss, temporal consistency loss, and state alignment loss. The cross-modal contrastive loss is used to narrow the distance between different modal representations at the same time, the temporal consistency loss is used to ensure the smoothness of latent representations at adjacent time points, and the state alignment loss is used to anchor latent variables to interpretable physical states.

4. The system according to claim 1, characterized in that, The system also includes a controllable factor screening unit, used to calculate the comprehensive controllability score for each object slot and screen out the top-ranked factors. Controllable factors are incorporated into the causal graph learning; the comprehensive controllable score is a weighted sum of motion sensitivity, contact change, language matching degree, and pose change.

5. The system according to claim 1, characterized in that, The causal branching module internally includes a structural causal matrix learning unit, a DAG constraint unit, and an alignment mapping head; the structural causal matrix Used to represent the strength of causal influence between variables, the DAG constraint unit ensures that the causal graph is a directed acyclic graph through acyclic constraint loss. The DAG acyclic constraint loss is: in For matrix trace operations, For matrix exponentiation operations, For element-wise multiplication, For causal variables.

6. The system according to claim 1, characterized in that, The three-stage training process executed by the training control unit includes: Phase 1: Freeze causal branches, train the multimodal encoder and action conditional transition module, and optimize the one-step prediction loss; Phase 2: Freeze the multimodal encoder and action conditional transition module, train the causal branch, and optimize the joint loss function including reconstruction loss, KL divergence, state alignment loss, DAG constraint loss, and mask loss; Phase 3: Freeze the multimodal encoder and causal branches, fine-tune the action conditional transfer module, and optimize the multi-step rolling loss.

7. The system according to claim 6, characterized in that, In Phase 3 training, an Alpha-gated fusion mechanism is used to weightedly mix the original latent representation with the causal regularized latent representation as the input to the transition module: in The mixing coefficients are determined using an exponential decay strategy. The output of the causal branch, This refers to the fused vector features output by the multimodal encoder.

8. A method for learning the world model of an embodied intelligent robotic arm based on causal alignment, characterized in that, Includes the following steps: S1: Collect multimodal data during the operation of the robotic arm, including visual images, tactile signals, control actions, and task language descriptions; S2: Encode and fuse multimodal data to generate a unified latent space state representation; S3: Screen controllable factors, establish a structural causal matrix, and learn the causal dependencies between variables; S4: Perform a three-stage training process to optimize model parameters; S5: Verify the fact prediction performance and counterfactual prediction performance of the model; S6: Connect the trained world model to the robotic arm control stack to achieve closed-loop control.

9. The method according to claim 8, characterized in that, The specific steps for implementing closed-loop control in step S6 are as follows: D1: Synchronously acquire multimodal sensing data; D2: Generate causally aligned structured latent representations; D3: Generate candidate action plans and predict future state trajectories in parallel; D4: Conduct risk assessment based on causal representation and select the optimal action. The risk assessment indicators include grasping stability, collision risk, goal achievement degree and causal consistency. D5: Execute actions and provide real-time feedback and correction.

Images