Cross-robot control method and system based on physical world model

Abstract

The application discloses a cross-robot control method and system based on a physical world model, and comprises a model training stage: historical multi-modal data is encoded into historical latent states, and action data is encoded into latent actions; a latent space prediction model is trained to predict future latent states from the historical latent states and the latent actions, a total loss function at least contains a prediction loss and a time series straightening loss, and a trained physical world model is obtained; a task execution stage: current multi-modal data is encoded into current latent states, and task input is parsed into target latent states; based on the current latent states, the target latent states and the latent space prediction model, a candidate latent action sequence is unfolded in a future state, and an optimal latent action sequence is solved; the optimal latent action sequence is decoded into skill parameters, and is mapped and adapted to corresponding robots to generate control instructions. The application separates the environment state representation and the action intention representation from specific robot entities, and solves the problem that the physical world model is difficult to be reused across robots.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a cross-robot control method and system based on a physical world model. Background Technology

[0002] World models, as the core of embodied intelligence, aim to learn the state transition patterns and action consequences of the environment to form an internal model capable of internal prediction, simulation, and planning, thereby supporting autonomous decision-making and behavior in unknown or dynamic environments. However, current mainstream solutions typically deeply couple visual encoders, dynamic predictors, and action representations with the specific robot's ontological parameters (such as degrees of freedom configuration, joint kinematics, end effector type, and sensor layout) when constructing world models. While this tightly coupled design can achieve high task performance on a single platform, the cost is that the model is completely bound to specific hardware. Once the robot platform is changed, due to fundamental changes in observation distribution, action space, and dynamic characteristics, the original model cannot be directly reused. A complete redevelopment process, from data acquisition and model architecture adjustment to training and optimization, must be carried out, resulting in high R&D costs and long cycles. This directly leads to the difficulty in scaling world models from one robot to another, severely hindering the large-scale deployment and application of the technology.

[0003] The Chinese patent document "A Control Method, System, Robot, and Medium for an Embossed Intelligent Robot," publication number CN119748443A, published on April 4, 2025, includes: acquiring interaction commands after the embodied intelligent robot enters an activated state; determining whether the embodied intelligent robot has entered a task planning phase based on the interaction commands; acquiring a tool list if the embodied intelligent robot has entered the task planning phase; performing task planning on the embodied intelligent robot based on the interaction commands and the tool list to obtain a feasible action sequence; and having the embodied intelligent robot execute the interaction commands according to the action sequence. This technology addresses the technical problem that embodied intelligent robots fail to effectively utilize existing tools in the environment, thus making originally feasible tasks unplannable. However, this technology emphasizes deep optimization within a single robot platform, a single task cluster, or a single scene, achieving good results in local scenarios, but often trades performance for platform coupling, lacking sufficient support for cross-ontology migration, cross-scene expansion, and online self-learning. Summary of the Invention

[0004] The present invention aims to overcome the problems of poor reusability, high cost, and low scalability in cross-robot task execution caused by the use of a modeling method that is tightly coupled with the robot body in the prior art, which leads to the world model and intelligent agent being heavily dependent on specific hardware. The invention provides a cross-robot control method and system based on a physical world model.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A cross-robot control method based on a physical world model, including a model training phase: Historical multimodal data is encoded into historical latent states, and action data is encoded into latent actions; Train a latent space prediction model to predict future latent states based on historical latent states and latent actions. The total loss function during training includes at least prediction loss and temporal straightening loss, resulting in a trained physical world model. Task execution phase: Encode the current multimodal data into the current latent state, and parse the task input into the target latent state; Based on the current latent state, the target latent state, and the latent space prediction model, the candidate latent action sequence is expanded into future states to solve for the optimal latent action sequence. The optimal latent motion sequence is decoded into skill parameters and mapped to the corresponding robot to generate control commands.

[0006] The cross-robot control method in this invention is designed for robots, especially humanoid robots, dual-arm manipulator robots, wheeled / legged mobile robots, and hybrid robots. It constructs an integrated technical architecture encompassing "environmental representation, latent space dynamics, target understanding, latent planning, skill execution, self-learning, cross-robot adaptation, and safety monitoring." The world model is designed as an autonomous intelligent foundation that continues to be updated after deployment. During operation, the system learns from observed data and generates incremental data through real-world interactions. A meta-controller determines when to observe, explore, plan, execute, review, and update, thus forming a continuous learning loop in the physical world. Temporal straightening constraints are introduced into the latent space. By reducing the curvature of the latent trajectory, the Euclidean distance in the latent space better approximates the actual reachable path distance, thereby improving the stability of gradient planning, model predictive control, and hierarchical task planning. By adopting a decoupled cross-robot architecture, the environmental state representation and action intention representation are separated from the specific robot body, and the robot differences are converged to the robot adaptation layer. This allows the same upper-level world model and planner to be migrated to different robot platforms, requiring only the replacement or minor calibration of the body adaptation module, thus solving the problem of the difficulty in reusing the world model across robots.

[0007] As an alternative, a self-learning phase is also included: During task execution, multi-dimensional decision signals are continuously collected, comprehensive decision values ​​are constructed, and corresponding self-learning modes are switched based on preset threshold conditions. In the experience replay learning mode, the value of the data is evaluated based on several value indicators, and the data with the highest value are selected for incremental training or lightweight adaptation.

[0008] Preferably, the latent space prediction model performs time-series fusion modeling of historical latent state sequences and latent action sequences, extracts the state evolution context, learns the state transition function of latent states under the action of latent actions, and generates latent state prediction results for one or more future times using recursive expansion or parallel methods.

[0009] Preferably, the temporal straightening loss is constructed as follows: for any three consecutive time points within the training segment, the latent state z... t z t+1 z t+2 Construct the first potential velocity vector v t equals z t+1 -z t Second latent velocity vector v t+1 equals z t+2 -z t+1 Calculate the cosine value Ct of the angle between the two, and use 1-Ct as the local curvature penalty term. Summate or average the values ​​at each time point within the training segment to obtain the temporal straightening loss.

[0010] Preferably, encoding the motion data into latent motions includes: Receive raw motion input, perform time alignment and motion segmentation, and extract intermediate features that are independent of the robot body; By using pre-defined rules, parameterized mapping networks, or a combination of both, intermediate features are encoded into corresponding semantic dimensions to form latent actions.

[0011] Preferably, parsing the task input into the target latent state includes: For different types of task inputs, corresponding encoding methods are used for encoding, and the encoding results are parsed to generate a structured task representation; a task-related object relationship graph is generated based on the current latent state and task input; the structured task representation is converted into planning constraints; and target latent states, target latent trajectories, target condition distributions, or hierarchical target graphs are generated through latent target mapping.

[0012] Preferably, after solving for the optimal potential action sequence, a model predictive control-based rolling replanning is employed. At the current moment, the optimal latent action sequence is obtained. After executing the first K steps of the optimal latent action sequence, new multimodal data is collected and encoded to obtain a new current latent state. Starting from the new current latent state, the subsequent latent action sequence is solved again. The above process is repeated until the task is completed or terminated.

[0013] Preferably, decoding the optimal latent action sequence into skill parameters includes: Boundary detection is performed on continuous latent action sequences to identify action phase switching points; similarity matching is performed between latent action segments of each phase and skill prototypes in the skill library to obtain the corresponding skill selection results, and latent action segments are mapped to corresponding skill parameters.

[0014] A cross-robot control system based on a physical world model, comprising: The state coding module encodes multimodal data into latent states; The motion encoding module encodes motion data into latent actions; The latent space dynamics prediction module uses a latent space prediction model to predict future latent states based on historical latent states and latent actions. The latent target generation module parses the task input into the latent state of the target; The latent planning module optimizes and solves the sequence of latent actions based on the current latent state, the target latent state, and the latent space prediction model. The skill decoding module decodes the optimal latent action sequence into skill parameters that are independent of the robot itself. The meta controller schedules and controls each module based on the system's operating status.

[0015] As a preferred option, the cross-robot control system also includes: The body adaptation module maps and adapts the skill parameters output by the skill decoding module to the corresponding robot-generated control commands; The experience replay module stores runtime data, evaluates the value of the data based on preset value indicators, and performs incremental training or lightweight adaptation. The multimodal data acquisition module collects multimodal data of the robot in the physical environment.

[0016] This invention offers the following advantages: It reduces training and planning costs and improves the ability to preserve the physical essence by replacing pixel-level future predictions with a latent space physical world model; it improves the geometric properties of the latent space through temporal straightening constraints, making latent space distances more consistent with reachability and significantly improving the solution stability of gradient programming and MPC; it enhances the robot's self-learning ability by achieving unified scheduling of observation learning, action learning, and experience playback through a meta-control mechanism; it enables the system to be migrated to different robot platforms and reduces cross-platform deployment costs by decoupling the upper-layer physical world model from the lower-layer robot ontology; it reduces the dependence of the world model on pixel details and single ontology, making the representation closer to the physically predictable structure, thereby improving planning efficiency, stability, and cross-task reusability; and it enables the robot to continuously acquire high-value experience from the real physical environment and learn independently after deployment, thus mitigating performance degradation caused by domain offset, long-tailed samples, and environmental non-stationarity. Attached Figure Description

[0017] Figure 1This is a flowchart of the cross-robot control method based on a physical world model in this invention.

[0018] Figure 2 This is a flowchart of the rolling replanning process using model predictive control in the latent planning process of this invention.

[0019] Figure 3 This is a schematic diagram of the cross-robot control system based on the physical world model in this invention.

[0020] In the diagram: 1. Multimodal data acquisition module; 2. State coding module; 3. Action coding module; 4. Latent space dynamics prediction module; 5. Latent target generation module; 6. Latent planning module; 7. Skill decoding module; 8. Body adaptation module; 9. Execution robot; 10. Meta controller; 11. Experience playback module; 12. Edge-end collaboration module; 13. Safety constraint module. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be noted that all data in the present invention were collected under the premise of legality and compliance.

[0022] like Figure 1 As shown, a cross-robot control method based on a physical world model includes a model training phase: Historical multimodal data is encoded into historical latent states, and action data is encoded into latent actions; Train a latent space prediction model to predict future latent states based on historical latent states and latent actions. The total loss function during training includes at least prediction loss and temporal straightening loss, resulting in a trained physical world model.

[0023] Task execution phase: Encode the current multimodal data into the current latent state, and parse the task input into the target latent state; Based on the current latent state, the target latent state, and the latent space prediction model, the candidate latent action sequence is expanded into future states to solve for the optimal latent action sequence. The optimal latent motion sequence is decoded into skill parameters and mapped to the corresponding robot to generate control commands.

[0024] It also includes a self-learning stage: During task execution, multi-dimensional decision signals are continuously collected, comprehensive decision values ​​are constructed, and corresponding self-learning modes are switched based on preset threshold conditions. In the experience replay learning mode, the value of the data is evaluated based on several value indicators, and the data with the highest value are selected for incremental training or lightweight adaptation.

[0025] The cross-robot control method in this invention is designed for robots, especially humanoid robots, dual-arm manipulator robots, wheeled / legged mobile robots, and hybrid robots. It constructs an integrated technical architecture encompassing "environmental representation, latent space dynamics, target understanding, latent planning, skill execution, self-learning, cross-robot adaptation, and safety monitoring." The world model is designed as an autonomous intelligent foundation that continues to be updated after deployment. During operation, the system learns from observed data and generates incremental data through real-world interactions. A meta-controller determines when to observe, explore, plan, execute, review, and update, thus forming a continuous learning loop in the physical world. Temporal straightening constraints are introduced into the latent space. By reducing the curvature of the latent trajectory, the Euclidean distance in the latent space better approximates the actual reachable path distance, thereby improving the stability of gradient planning, model predictive control, and hierarchical task planning. By adopting a decoupled cross-robot architecture, the environmental state representation and action intention representation are separated from the specific robot body, and the robot differences are converged to the robot adaptation layer. This allows the same upper-level world model and planner to be migrated to different robot platforms, requiring only the replacement or minor calibration of the body adaptation module, thus solving the problem of the difficulty in reusing the world model across robots.

[0026] As a specific implementation, during the model training phase, it is first necessary to obtain the input for model training, including historical latent states and latent actions. Historical latent states are obtained through multimodal data encoding, and latent actions are obtained through action data encoding.

[0027] Specifically, encoding multimodal data into a unified latent state can be achieved through the following sub-networks: Visual encoding sub-network: extracts scene geometry, object layout, contact areas, and motion trends; Tactile / force coding subnetwork: Extracts contact position, normal force, friction state, and slippage tendency; Ontology encoding sub-network: Extracts joint state, pose, velocity, execution margin, and energy consumption state; Language / Task Condition Encoding Sub-Network: Extracts task semantics and constraints; Fusion coding subnetwork: Outputs a unified latent state. The latent state can be in vector form or in tensor form that preserves spatial structure. It is preferable to preserve some spatial structure rather than prematurely compressing it into a single global vector, in order to improve the predictability of object relationships and contact dynamics.

[0028] Furthermore, encoding motion data into latent actions includes: Receive raw motion input, perform time alignment and motion segmentation, and extract intermediate features that are independent of the robot body; By using pre-defined rules, parameterized mapping networks, or a combination of both, intermediate features are encoded into corresponding semantic dimensions to form latent actions.

[0029] Specifically, the received raw motion inputs include at least one of the following: teleoperation trajectory, taught motion sequence, robot joint state change, end effector trajectory, force / torque change, contact event, mobile chassis speed command, visual observation sequence, and task command.

[0030] First, the original motion input is time-aligned, coordinate-unified, and motion-segmented. Then, intermediate features unrelated to the robot body are extracted, including end-effector pose changes, object position changes, contact establishment / dissolution events, force direction and magnitude, chassis displacement trends, dual-arm collaboration relationships, and task stage labels.

[0031] Based on this, intermediate features are encoded into corresponding semantic dimensions through pre-defined rules, parameterized mapping networks, or a combination of both. For example: the end-effector trajectory is encoded as the semantics of end-effector spatial pose increment; gripper opening and closing, tactile abrupt changes, and force abrupt changes are encoded as the semantics of contact state switching; the visually tracked manipulated object and target object are encoded as the semantics of the action object and target object identification; force / torque and impedance control parameters are encoded as the semantics of force control / impedance adjustment intention; chassis linear velocity, angular velocity, and displacement trajectory are encoded as the semantics of the moving chassis displacement intention; the synchronization relationship between two arms or multiple actuators is encoded as the semantics of cooperative relationship; and the stage boundary information in the action sequence is encoded as the semantics of task stage identification.

[0032] The parameterized mapping network can employ one or more of the following: multilayer perceptron, temporal convolutional network, recurrent neural network, Transformer, or graph neural network. It is used to map continuous action sequences into fixed-length or variable-length action semantic vectors. Predefined rules are used to constrain and correct the semantic dimension based on thresholds, contact events, object relationships, and action templates. Thus, joint commands on a specific robot are converted into unified, ontology-independent motion intent codes, which are then used for prediction and latent planning in the upper-level world model.

[0033] As a specific example, the model training phase also includes the specific construction and training of the latent space prediction model.

[0034] The latent space prediction model extracts the state evolution context by temporally fusing historical latent state sequences and latent action sequences, learns the state transition function of latent states under the action of latent actions, and generates latent state prediction results for one or more future times using recursive expansion or parallel methods.

[0035] Specifically, the system receives historical latent state sequences and historical latent action sequences. First, a temporal coding unit extracts the state evolution context. Then, an action condition fusion unit injects latent actions into the state transition process. Finally, a state transition prediction unit outputs the latent state prediction results for one or more future timeframes. For example: first, the latent state z at each timeframe is... i With the corresponding latent action a i By splicing, weighted summation, cross-attention fusion, or gating fusion, the joint state-action representation h is obtained. i The joint representation is then input into a temporal prediction network, which can be a recurrent neural network, a temporal convolutional network, a Transformer, a state-space model, or a combination thereof, to learn the transition patterns of the latent states over time. Finally, the prediction head outputs the next latent state z. t+1 And z can be obtained recursively. t+2 , z t+3 , ..., z t+H .

[0036] Among them, multi-step future latent state prediction includes two methods: one is the recursive expansion method, that is, first predicting z t+1 Then z t+1 Use the next potential action as input to continue predicting z. t+2 One approach is to iterate sequentially until the predicted time domain length H is reached. Another approach is parallel direct prediction, where the same prediction network directly outputs a sequence of latent states for multiple future time points. For branch prediction, the network can simultaneously output multiple candidate future latent trajectories and their corresponding probabilities to represent various possible state evolution outcomes in the environment. For conditional prediction, task objectives, constraints, object identifiers, contact patterns, or robot ontology descriptions can be further added to the input to ensure the prediction results meet specific task conditions. For uncertainty prediction, an additional confidence header can output the variance, interval, probability distribution parameters, or risk score for each prediction time point for risk weighting in subsequent planning.

[0037] To improve prediction accuracy and long-term stability, the latent space prediction model can simultaneously optimize the following objectives during training: first, the prediction error loss between future latent states and actual latent states; second, temporal consistency loss, used to constrain the consistency between short-term and long-term predictions; third, latent trajectory geometric regularization loss, used to suppress excessive curvature of the latent state sequence; and fourth, contact event, object relationship, or physical constraint preservation loss, used to enhance the physical rationality of the prediction results. Therefore, through a technical path of historical latent state encoding + latent action condition injection + temporal state transition modeling + multi-step unfolding / parallel prediction + confidence output, plannable prediction of future latent states can be achieved.

[0038] Furthermore, during the training phase, not only is the future latent state prediction error minimized, but a temporal straightening regularization is also applied to the latent state sequence output by the encoder. The temporal straightening loss is constructed as follows: for any three consecutive latent states z within the training segment... t z t+1 z t+2 Construct the first potential velocity vector v t equals z t+1 -z t Second latent velocity vector v t+1 equals z t+2 -z t+1 Calculate the cosine value Ct of the angle between the two. Using 1-Ct as the local curvature penalty term, the summation or mean of each time step within the training segment is used to obtain the temporal straightening loss. To prevent numerical stability terms with a denominator of zero, when Ct is closer to 1, it indicates that the directions of adjacent underwater velocities are more consistent, the underwater trajectory is closer to a straight line locally, and the local curvature is smaller; when Ct is small or even negative, it indicates that the underwater trajectory has undergone a significant turn and the local curvature is larger.

[0039] In this invention, temporal straightening is not an independent post-processing step, but rather embedded as a joint training constraint of the latent space dynamics prediction module during the training process of the physical world model. The temporal straightening loss and the latent state prediction loss jointly constitute the overall training objective, which is used to jointly update the state encoding module, action encoding module, and latent space dynamics prediction module through backpropagation, thereby enabling the encoded latent trajectory to have higher local directional consistency in the time dimension. For the latent state tensor that preserves the spatial structure, the cosine value of the included angle can be calculated position-by-position, by mean aggregation, by flattening, or by a learnable aggregation head.

[0040] Furthermore, it is also possible to analyze the latent state z at any three consecutive time points within the training segment. t z t+1 z t+2 Construct the second-order difference form Δ 2 z t =z t+2 -2z t+1 +z t A second-order difference penalty term is added, averaging across all time points within the training segment as a supplementary curvature constraint. The cosine term of the time-series straightening loss constrains directional consistency; the second-order difference term constrains step size variation and smoothness; the combination of these two can simultaneously suppress two latent trajectory distortions detrimental to planning: "excessive turning" and "sudden acceleration / deceleration." Through these methods, the Euclidean distance in the latent space more accurately reflects the reachability of the real state transition, thereby improving the stability and success rate of gradient planning and model predictive control based on latent space expansion.

[0041] As a specific example, during the task execution phase, it is first necessary to obtain the inputs for prediction (current latent state and target latent state). The target latent state can be represented in four forms: single-point target latent state, target latent trajectory, target condition distribution, and hierarchical target map.

[0042] The method for encoding current multimodal data into current latent states is the same as the method for encoding historical multimodal data into historical latent states, so it will not be described in detail.

[0043] Parsing the task input into the target latent state includes: For different types of task inputs, corresponding encoding methods are used for encoding, and the encoding results are parsed to generate a structured task representation; a task-related object relationship graph is generated based on the current latent state and task input; the structured task representation is converted into planning constraints; and target latent states, target latent trajectories, target condition distributions, or hierarchical target graphs are generated through latent target mapping.

[0044] It should be noted that the parsing method in this embodiment is used to uniformly convert heterogeneous task inputs into a target representation that can be directly called by the subsequent latent planning module. This includes steps such as task input encoding, task semantic parsing, object relation extraction, target constraint generation, latent target mapping, and target verification.

[0045] The task input encoding step receives at least one task input, including: natural language instructions; example video; teleoperation teaching trajectory; target image; CAD model, process template or digital twin prior; rule constraints or security constraints.

[0046] Different encoding methods are used for different inputs: For natural language instructions, a text encoding network is used to extract task lexical features, identifying action words, object words, spatial relation words, sequence words, constraint words, and termination condition words to form a text task vector. For example videos and target images, a visual encoding network is used to extract object categories, object poses, scene layouts, target contact states, and change trends to form a visual task vector. For teleoperation teaching trajectories, the end-point trajectory, contact establishment and dissolution events, object state changes, stage switching points, and successful final states are extracted to form a teaching task vector. For CAD / process priors, the target assembly position, tolerance range, mating direction, insertion sequence, no-collision areas, and process steps are extracted to form a priori constraint vector. For rule constraints or safety constraints, prohibited areas, maximum speed, maximum force, minimum safe distance, sequence restrictions, and tolerance error range are extracted to form a constraint vector.

[0047] The task semantic parsing step uniformly parses the above input encoding results to generate a structured task representation. The structured task representation includes at least: the operation object; the target object or target region; the target relationships between objects; the sequence of action stages; success criteria; prohibited states; tolerance error range; and priority and constraint levels. The task semantic parsing step outputs a task graph structure, where nodes represent objects, regions, stages, or constraints, and edges represent spatial relationships, temporal relationships, contact relationships, or logical dependencies.

[0048] For example, for the task "Place the red cup in the center of the tray without knocking over the bottles next to it", the parsing result can be represented as follows: Operation object: red cup; Target object: tray; Target relationship: the cup is located in the center area of ​​the tray; Action phases: approach, grasp, lift, move, place, release; Success criteria: the center of the cup is located in the target area of ​​the tray, and the posture deviation is less than the threshold; Prohibited state: the bottle is tilted or the collision exceeds the threshold; Tolerance error: the position error is not greater than the first threshold, and the posture error is not greater than the second threshold.

[0049] The object relationship extraction step, based on the current latent state and task input, identifies task-related objects and their relationships, and constructs target conditions. Specifically, this includes: locating target and reference objects from visual observation; estimating the object's current pose, velocity, contact state, and reachability; calculating the positional, orientational, and topological relationships between the object and the target region; determining the presence of obstacles, restricted areas, or objects to be avoided; and generating a task-related object subgraph. For example, in an assembly task, this unit can extract: plug objects; socket objects; insertion direction; mating clearance; assembly starting pose; permitted contact areas; and prohibited collision areas.

[0050] The target constraint generation step transforms the structured task representation into planning constraints. Planning constraints include at least one or more of the following: 1. Final state constraints: The target object must reach a specified position, posture, or contact state; 2. Process constraints: Execution must follow a predetermined sequence of stages; 3. Relationship constraints: Objects must maintain relationships such as inclusion, contact, alignment, collision avoidance, and support; 4. Safety constraints: Objects cannot enter restricted areas or exceed speed / force thresholds; 5. Tolerance constraints: Allowable errors must be within a given range; 6. Time constraints: Specific stages must be completed within a specified time window; 7. Resource constraints: Tasks must be completed collaboratively by a single arm, dual arms, or a mobile chassis. These constraints can be represented as hard constraints or as soft constraint terms in the cost function.

[0051] The latent goal mapping step maps the structured task representation and planning constraints into a goal form that can be processed by the latent planning module. Specifically, it includes the following implementations: Single-point target latent state generation. When the task objective is mainly represented by a final state, the current state encoding module first encodes the target image, target scene, or successful final state teaching to obtain the target latent state. During planning, the predicted latent state is designed to be close to the target latent state. For example, the image of "the cup after being placed in the center of the tray" is input into the encoder to obtain the target latent state, which serves as the final state optimization objective.

[0052] Target latent trajectory generation. When a task has a defined phase sequence, the task is broken down into multiple sub-phases, and a stage-specific target latent state z is generated for each phase. g1 , z g2 , ..., z gn This forms the target's latent trajectory or target latent sequence. The latent planning module sequentially approximates the latent targets at each stage. For example, the insertion task can be broken down into four stages: "pre-alignment - contact - insertion - locking," each generating a corresponding latent target.

[0053] Target condition distribution generation. When the task has multiple solutions or allows multiple equivalent final states, a single-point target is not output; instead, the target condition distribution p(z) is output. g |T) or the set of target latent states {z g 1 , z g 2 , ..., z g m The latent planning module can search for feasible trajectories that satisfy the constraints within the distribution or set. For example, if several pieces of tableware are placed on a tray, requiring only that they fall within a specified area and do not overlap, then a set of target states can be output instead of a single final state.

[0054] Hierarchical target graph generation. For long-term complex tasks, a hierarchical target graph is generated. The upper layer consists of task stage nodes, and the lower layer consists of the target latent states or constraint sets corresponding to each stage; nodes are connected by sequential dependencies. The latent planning module first selects the current stage, and then performs local planning for the target latent states of that stage.

[0055] The target validation step performs consistency and reachability checks on the generated latent targets to avoid generating unexecutable targets. Specifically, this includes: checking if the target object exists in the current scene; checking if target relationships conflict with current physical constraints; checking if the phase sequence is complete; checking if the target latent state falls within the reachable latent space region; checking for violations of safety constraints and ontology constraints; if unreachable, outputting a corrected target or triggering human-machine confirmation. Target validation can be achieved through: short-view rollout validation using a world model; reachingability estimation using an ontology adaptation module; and checking for restricted areas and risks using a safety constraint module.

[0056] Specifically, taking the task "grab workpiece A and insert it into the hole of workpiece B" as an example: the task input coding step reads the language instructions and CAD priors; the task semantic parsing step identifies "workpiece A" as the operation object, "hole of workpiece B" as the target object, and "insertion" as the target relationship; the object relationship extraction step locates the current pose of A and B and identifies the hole axis direction; the target constraint generation step outputs the constraints of "axial alignment, contact force not exceeding the threshold, and insertion depth reaching the threshold"; the latent target mapping step generates three stages of targets: "pre-aligning the latent target, contacting the latent target, and inserting the latent target"; the latent planning module optimizes and executes the three stages in sequence.

[0057] As a specific implementation, after inputting the current latent state and the target latent state during the task execution phase, it is necessary to combine the latent space prediction model, based on the state transition prediction results in the latent space, to solve the latent action sequence in the future time domain, and output the action intention sequence (optimal latent action sequence) for the skill decoding module or the ontology adaptation module to execute.

[0058] The input for latent programming includes at least the current latent state z. t ; Target potential state z g Target latent trajectory τ g Or target condition set; prediction time domain length H; latent space prediction model fθ; initial latent action sequence A z ={a t z a t+1 z , ..., a t+H-1 z Task constraints include at least one of collision constraints, contact constraints, time constraints, energy consumption constraints, safety constraints, and ontology reachability constraints; optional uncertainty estimation, risk scoring, or feasibility masking. The initial potential action sequence can be generated from zero-action initialization, historical action extension, candidate actions from the skill library, randomly sampled actions, or actions retrieved from the experience library.

[0059] After input is completed, latent trajectory unfolding is performed. Instead of searching directly in the original image space or original joint space, it calls the latent space prediction model in the latent space to unfold the future state of the given latent action sequence.

[0060] Specifically, given the current latent state z t and candidate latent action sequence A z Generate the future latent state sequence based on the following recursive relation: z t+1 =fθ(z t a t z );z t+2 =fθ(z t+1 at+1 z );...;z t+H =fθ(z t+H-1 a t+H-1 z Thus, the predicted latent trajectory is obtained: τ={z t+1 , z t+2 , ..., z t+H}

[0061] After the unfolding is complete, an optimization objective function is defined for the candidate potential action sequence. The optimization objective function includes at least one or more of the following: target proximity term, stage target term, action smoothing term, energy consumption or execution cost term, risk and uncertainty term, and constraint violation term. The overall objective function can be defined as a weighted sum of multiple optimization objective functions.

[0062] For the target proximity term, when the task reaches its final state, it is defined as the final potential state z. t+H With the target potential state z g The distance cost Jgoal between them; when the task is trajectory tracking, it is defined as predicting the latent trajectory z. t+i With the target's potential trajectory z gi The cumulative distance between them is Jtraj.

[0063] For stage objectives, when a task is broken down into multiple stages, a stage objective latent state or stage constraint is set for each stage: Jstage is ∑α j ||z t+hj -z gj ||2 2 , where hj represents the time corresponding to the j-th stage, and the summation ranges from j=1 to j=N.

[0064] To avoid drastic changes in the latent action sequence, an action smoothing term is added: Jsmooth is ∑||a t+i z -a t+i-1 z ||2 2 The summation proceeds from i=1 to i=H-1.

[0065] To reduce the execution burden, an action amplitude penalty is added as an energy consumption or execution cost term: Jenergy is ∑||a t+i z ||2 2 The summation proceeds from i=0 to i=H-1.

[0066] If the latent space dynamics prediction module outputs uncertainty σ t+i Then, add the risk and uncertainty term: Jrisk is ∑β i *σ t+iThe summation proceeds from i=1 to i=H.

[0067] Define constraint violations for collision, restricted areas, force thresholds, contact errors, or unreachable states: Jcons is ∑γ*Φ*(z t+i , C), where C represents the constraint set and Φ represents the constraint violation function.

[0068] By minimizing the overall objective function, the optimal sequence of latent actions can be obtained by using gradient optimization, sampling optimization, or hierarchical optimization.

[0069] Optional, such as Figure 2 As shown, after solving for the optimal potential action sequence, model predictive control rolling replanning can be used: The optimal sequence of latent actions is obtained at the current moment. After executing the first K steps of the optimal sequence, new multimodal data is collected and encoded to obtain a new current latent state. The subsequent sequence of latent actions is solved again starting from the new current latent state. This process is repeated until the task is completed or terminated. In the event of environmental changes, object slippage, contact anomalies, or human intervention, the system can replan based on the latest latent state without relying entirely on the initial open-loop plan.

[0070] During the latent planning process, the system continuously compares the predicted latent state with the current latent state obtained from the actual encoding. If the deviation exceeds a threshold, at least one of the following mechanisms is triggered: local latent re-optimization; stage goal rollback; invocation of recovery skills; switching to human takeover mode; or writing the failed segment into the experience replay module.

[0071] Taking "grabbing the cup and placing it in the center of the tray" as an example: the current latent state z is obtained by encoding the current observation multimodal data. t The target image is used by the target generation module to obtain the target latent state z. g Initialize the future H-step latent action sequence; call the latent space dynamics module to unfold the future latent trajectory; calculate the distance between the final latent state and the target latent state, and add collision and action smoothing penalties; optimize the latent action sequence to obtain the optimal latent action sequence; execute the skill corresponding to the previous latent action; if the cup slips and causes the new observed latent state to deviate from the prediction, then replan the subsequent actions.

[0072] As a specific implementation, after obtaining the optimal potential action sequence during the task execution phase, it is necessary to decode the optimal potential action sequence representing the action intention sequence into the corresponding skill parameters.

[0073] Decoding the optimal latent action sequence into skill parameters involves: receiving the action intent sequence output by the latent planning module; firstly, performing boundary detection on the continuous latent action sequence through the intent segmentation unit to identify action phase transition points; then, using the skill matching unit to perform similarity matching between the latent action segments of each phase and skill prototypes in the skill library to obtain the corresponding skill selection results. Similarity matching can be jointly calculated based on latent action embedding distance, target object relationship, contact pattern, and task phase label. After matching, the parameter decoding unit maps the latent action segment into corresponding skill parameters, including end-target pose, direction of action, velocity, force control / impedance parameters, gripper opening / closing amount, duration, and termination threshold; mapping can be achieved through lookup table rules, parameter regression networks, or template filling methods.

[0074] Furthermore, the skill decoding module also includes a switching condition generation unit and a recovery condition generation unit. Switching conditions are generated based on whether the target corresponding to the current skill has been reached, whether a contact state has been established, whether the duration has been met, and whether the error has converged. Recovery conditions are generated based on the deviation between the predicted latent state and the current latent state, contact anomalies, force perception anomalies, target loss, or actuator over-limit. When a deviation exceeding a threshold is detected during execution, the recovery condition is triggered, and the system calls skills from the recovery skill set, such as retreat, re-grab, re-align, deceleration probe, or request replanning. Finally, the skill decoding module outputs a structured execution instruction for the skill parameters (skill type + skill parameters + execution duration + switching condition + recovery condition) and sends it to the ontology adaptation module.

[0075] The body adaptation module maps skill parameters to the corresponding robot to generate control commands. After a new robot is connected, the body parameter model of the robot is automatically built through adaptive calibration and written into the corresponding body adaptation module, forming the robot's mapping parameters, constraint boundaries, and control interface model for subsequent conversion of motion intentions to execution commands. Adaptive calibration automatically identifies the new robot's degrees of freedom, workspace, joint limits, end-effector attributes, control delay, and impedance boundaries by reading the robot's structural parameters and combining active excitation tests and response data fitting within a safe range.

[0076] As a specific implementation, during the self-learning phase, the meta-controller constructs a comprehensive decision value based on multi-dimensional decision signals and switches the corresponding self-learning mode based on preset threshold conditions.

[0077] Self-learning modes include: In the observational learning mode, the robot mainly performs perception and recording without actively engaging in large-scale operations; it collects visual, tactile, force, and proprioceptive state data to update the world model and object relationship representation.

[0078] The imitation and reproduction mode allows the robot to reproduce the action process based on the taught trajectory, teleoperation samples, or historical successful trajectories, in order to quickly complete existing tasks and accumulate standard execution data.

[0079] In the active exploration mode, the robot actively tries new actions, new ways of contact, or new perspectives under safety constraints in order to obtain high information gain data and improve its ability to model unknown objects, unknown scenes, and unknown dynamics.

[0080] In the latent planning and execution mode, the robot optimizes the sequence of action intentions in the latent space based on the current latent state, the target latent state, and the prediction model, and executes the corresponding skills to complete the target task.

[0081] In the failure recovery mode, when the robot detects excessive prediction deviation, grasping failure, abnormal contact, trajectory deviation, or target loss, it invokes strategies such as backing away, retrying, realigning, regrasping, or local replanning to recover.

[0082] In the experience replay learning mode, the robot selects data from historical successful samples, failure samples, and high-value samples for offline or online replay training, which is used to correct the world model, skill parameters, or adapter parameters.

[0083] In the simulation-imagination mode, the robot first unfolds multiple action plans and predicts the results in an internal world model or simulation environment, without directly acting on the real environment. This mode is used for low-risk verification and strategy selection.

[0084] In the human collaboration / help-seeking mode, when the task exceeds the current capability boundary or the risk is too high, the robot requests human instruction, confirmation, correction or takeover, and transforms human feedback into subsequent learning samples.

[0085] In safety freeze mode, when a collision risk, excessive force, excessive speed, instability, intrusion into a restricted area, or system abnormality is detected, the robot immediately stops or restricts its action output to maintain a safe state and await further processing.

[0086] Multidimensional decision signals include: prediction error Et (the deviation between the current latent state and the predicted latent state); model uncertainty Ut (output by the latent space dynamics prediction module); and task urgency Pt (determined by the remaining time and remaining stages, w1*t). used / t limit +w2*n remain / n total Historical success rate St; Environmental change level Ct (||z) t -z t-1||2, or obtained statistically from changes in object position, occlusion, and contact); sensor anomaly score, calculated from frame drop rate, noise amplitude, drift degree, and out-of-bounds ratio, denoted as Qt; actuator health score, calculated from current, temperature, saturation count, tracking error, and vibration amplitude, denoted as Ht; safety risk level Rt, calculated comprehensively from collision probability, minimum distance, over-force risk, and instability risk. The comprehensive decision score Dt is α1*Et+α2*Ut+α3*Ct+α4*Qt+α5*Rt-α6*St-α7*Ht+α8*Pt, where αi is the weighting coefficient.

[0087] Then, select the mode based on the score range and key threshold (τ). For example: if Rt is greater than τR or Qt is greater than τQ, enter the safety freeze mode; if Et is greater than τE and Rt is less than τR, enter the failure recovery mode; if Ut is greater than τU and Rt is less than τR, enter the active exploration mode; if St is greater than τS, Et is less than τE, and Ut is less than τU, enter the latent planning execution mode; if Pt is greater than τP and there are already successful examples, prioritize entering the imitation and reproduction mode; if the task exceeds the capability boundary or fails multiple times, enter the human collaboration / help mode; if the system is idle and there are high-value samples (samples with low comprehensive decision scores), enter the experience playback learning mode or simulation imagination mode.

[0088] The specific training process in this invention includes five stages. Stage 1, General Observation Pre-training: Collecting multi-scene, multi-task, multi-robot, and multi-view video and sensor data; training the world state encoder (state encoding module) through joint embedding prediction to enable it to learn transferable latent representations; incorporating anti-collapse mechanisms during training, such as stopping gradients, variance regularization, and covariance regularization. Stage 2, Action Condition World Model Training: Encoding the taught trajectory, teleoperated trajectory, and autonomous exploration trajectory into latent state-latent action sequences; training the action condition predictor (latent space dynamics prediction module) to predict future latent states from historical latent states and latent actions. Training losses include: prediction error loss, temporal straightening loss, contrast consistency loss, contact event consistency loss, and constraint preservation loss. Among them, temporal straightening loss reduces curvature by constraining the angle between adjacent latent velocities. Stage 3, Joint Training of Latent Planning and Skill Decoding: Training on the trained world model for tasks such as target arrival, grasping, transporting, assembling, and navigation; aligning the latent planning results with the actual execution effect in a closed loop; training the skill selector, skill parameter generator, and failure recovery unit. Phase Four, Cross-Robot Adaptation Training: Freeze some upper-layer world model parameters and train only the ontology adaptation module, a small number of LoRA / Adapters or motion mapping heads; enabling the same world model to be transferred to different robots. Phase Five, Post-Deployment Incremental Learning: During runtime, the meta-controller selects new samples based on data value; performs lightweight online adaptation at the edge; and performs periodic world model distillation, adapter updates, and skill library updates in the cloud; the updated model is then securely verified before being deployed to the robots.

[0089] In addition to a cross-robot control method based on a physical world model, this invention also provides, for example, Figure 3 The cross-robot control system shown includes: Multimodal data acquisition module 1 collects multimodal data of the robot in the physical environment (vision, touch, force, voice, body state, environmental state, etc.), performs spatiotemporal alignment and data processing, and then sends it to the state coding module; State encoding module 2 encodes multimodal data into latent states and sends them to the latent space dynamics prediction module and the latent target generation module; Action encoding module 3 encodes action data (historical actions, skill tags, task semantics) into latent actions and sends them to the latent space dynamics prediction module and the latent planning module; The latent space dynamics prediction module 4 uses a latent space prediction model to predict future latent states based on historical latent states and latent actions.

[0090] The latent target generation module 5 parses the task input (language task, demonstration trajectory, visual target, constraints) into the latent state of the target; The latent planning module 6 optimizes the latent action sequence based on the current latent state, the target latent state, and the latent space prediction model, and obtains the optimal latent action sequence. Skill decoding module 7 decodes the optimal latent motion sequence into skill parameters (such as executable skill segments, joint reference trajectories, end effector trajectories, or contact sequences) that are independent of the robot itself. The body adaptation module 8 maps and adapts the skill parameters output by the skill decoding module to the corresponding robot (the joints, drivers, mobile chassis, dexterous hand or actuator interface of the specific robot) to generate control commands. The execution robot 9 is a specific task execution unit. Each execution robot has its own body adaptation module, which adapts the skill parameters into specific operation and control instructions.

[0091] The meta controller 10 performs scheduling and control of each module according to the system's operating status; The experience replay module 11 stores runtime data and writes online data into multi-level memory for subsequent self-supervised training, counterfactual review, and strategy updates; it also evaluates the value of the data based on preset value indicators and performs incremental training or lightweight adaptation. Safety constraint module 12 performs constraint detection and auditing before planning, during execution, and after execution; The edge-to-device collaboration module 13 performs lightweight adaptation on the edge and incremental training and updates in the cloud.

[0092] The multimodal data acquisition module addresses the problem that existing world models rely solely on single-modal vision or low-dimensional states, making it difficult to form a physically operable understanding. It includes at least one or more sensors: RGB cameras, depth cameras, fisheye cameras, event cameras, millimeter-wave radar, lidar, haptic arrays, six-dimensional force sensors, IMUs, body encoders, joint current sensors, temperature sensors, microphone arrays, and external environment sensors.

[0093] The collected data is written to the buffer bus according to a unified timestamp; for each modality, the following processes are performed: synchronization, distortion correction, coordinate extrinsic parameter calibration, noise filtering, time slicing, event boundary detection, and confidence estimation. To improve usability in embodied scenarios, a time fusion method of "fixed-frequency master clock + asynchronous event frame interpolation" can be adopted. For haptic and visual senses, hysteresis compensation and contact event marking mechanisms are used to maintain the time alignment of short-term physical events such as grasping, plugging, pushing, pulling, and collision.

[0094] The state encoding module addresses the problems of high redundancy in original high-dimensional observations, the difficulty of pixel prediction in expressing physical essence, and the inability to form a shared state space between different scenes. It encodes multimodal observations into a unified latent state. This includes: a visual encoding sub-network (extracting scene geometry, object layout, contact areas, and motion trends); a tactile / force encoding sub-network (extracting contact positions, normal forces, friction states, and slippage trends); a ontology encoding sub-network (extracting joint states, postures, velocity, execution margins, and energy consumption states); a language / task condition encoding sub-network (extracting task semantics and constraints); and a fusion encoding sub-network (outputting the unified latent state).

[0095] Specifically, one or more of the following can be used: Transformer, Spatiotemporal Convolutional Network, Graph Neural Network, Sparse Voxel Network, Visual Language Encoder, and Multimodal Fusion Transformer. Scene entities are decomposed into object-level tokens, contact-level tokens, body-level tokens, and task-level tokens, and then attention fusion is performed. Additional outputs include uncertainty estimates, contact confidence, and reachability masks.

[0096] The motion encoding module addresses the issues of strong coupling between motion representation and the robot body, and difficulties in cross-robot transfer. Motion is not directly represented as a command for a specific robot joint, but rather as a body-independent motion intent encoding. This encoding includes at least one or more of the following semantic dimensions: end-effector spatial pose increment; contact state switching; identification of the action object and target object; force / resistance adjustment intent; chassis displacement intent; gait / support switching intent; dual-arm / multi-arm collaborative relationships; and task stage identifiers.

[0097] Action semantics are encoded to form latent actions, which are then used by the latent space dynamics prediction module and the latent planning module. In this way, the upper-level model learns "how the intention changes the world," rather than "how a specific joint of a particular robot rotates." The kinematic, dynamic, and actuator differences between different robots are handled in the ontology adaptation module.

[0098] The latent space dynamics prediction module addresses the problems of large long-term prediction errors, difficult planning, and high computational costs associated with existing world models. Based on the current latent state, historical latent states, and latent actions, this module predicts future latent states, forming: single-step prediction; multi-step unfolding prediction; branching prediction; conditional prediction; and uncertainty prediction.

[0099] The latent space dynamics prediction module can consist of the following components: a temporal coding unit, an action condition fusion unit, a state transition prediction unit, a confidence head, and a multi-resolution prediction head. The state transition prediction unit can be a Transformer, a state-space model, a recursive network, or a graph temporal network. Curvature regularization is introduced into the latent state sequence during training. Specifically, this involves constraining the angle between continuous latent velocity vectors to increase the cosine similarity of adjacent latent velocities; equivalently, penalizing the local curvature of the latent trajectory. The training objective includes the sum of the prediction error term and the curvature regularization term.

[0100] The latent target generation module addresses the challenge of transforming language tasks, visual demonstrations, hands-on teaching, and abstract task objectives into programmable states. It converts user-inputted goals or tasks into latent states, latent trajectories, or distributions of target conditions. Inputs can include: natural language instructions; example videos; teleoperation teaching trajectories; target images; CAD / process priors; and rule or safety constraints. The system parses the task into: objects, relationships, action stages, success criteria, prohibited states, and tolerance error ranges. These are then mapped to latent targets: single-point target latent states, target latent trajectories, multi-constraint target sets (target condition distributions), or hierarchical target maps.

[0101] The latent planning module addresses the problems of long planning times, poor robustness to environmental changes, and lack of a unified planning layer for different tasks in existing robot systems. It plans within the learned latent space, rather than searching directly in pixel space or the robot's original joint space. Its inputs include: current latent state, target latent state / trajectory, constraints, set of executable skills, latent space prediction model, and uncertainties. Its outputs are: latent action sequence, skill sequence, execution priority, and termination condition.

[0102] The executable skill set comes from the skill decoding module (which contains a skill library) and includes callable skills such as grasping, carrying, plugging, pushing, pulling, walking, and obstacle avoidance. The latent space prediction model comes from the latent space dynamics prediction module and is used to predict future latent trajectories based on the current latent state and candidate latent actions. Uncertainty comes from the confidence output or risk assessment of the latent space prediction model, representing the reliability of the future latent state prediction, and can be used for risk weighting during planning.

[0103] The latent planning module also includes several cost functions during planning: Target approach cost: calculated from the distance between the predicted latent state and the target latent state / target trajectory. Constraint violation cost: obtained by comparing task constraints, process constraints, or safety constraints with the predicted latent trajectory. Collision risk cost: calculated from the spatial relationship between the environmental model, obstacle states, and the predicted trajectory. Energy consumption cost: estimated from the motion amplitude, joint torque, current, or power corresponding to the candidate motion sequence. Contact instability cost: obtained from tactile, force, contact state prediction, or slip risk estimation. Actuator saturation cost: obtained by mapping candidate motions to the robot body and comparing them with actuator upper limits such as joint speed, torque, current, and stroke. Uncertainty penalty term: obtained from the variance, confidence level, or risk score output by the latent space dynamics prediction module. Safety barrier term: calculated by the safety constraint module based on safety boundaries such as restricted areas, minimum distance, maximum force, and maximum speed. Human-robot interaction comfort cost: calculated from indicators such as the distance between the human and the robot, relative speed, orientation, and degree of motion abruptness.

[0104] The skill decoding module addresses the challenge of stable implementation of end-to-end low-level control. After latent planning, the system does not directly output motor current but instead first enters the skill decoding layer. The skill library within the skill decoding module contains several reusable atomic and combinable skills, such as: approach, grasp, carry, rotate, plug, twist, push, pull, open, obstacle avoidance, hand switching, walking, crouching, raising, and two-arm coordination. The sequence of action intent output by the latent planning module is converted by the skill decoding module into skill selection results, skill parameters, duration, switching conditions, and failure recovery conditions. During execution, the predicted latent state can be compared with the observed current latent state in real time. If the deviation exceeds a threshold, local replanning or skill switching for recovery is triggered.

[0105] The body adaptation module addresses the challenge of migrating physical world models to different robot platforms. It maps unified latent motion / skill parameters to specific robot bodies. This includes: a kinematics adaptation unit; a dynamics constraint unit; an actuator interface unit; a sensor topology mapping unit; a body calibration and self-identification unit; and a tool / end-effector switching unit.

[0106] To achieve cross-robot migration, this invention incorporates differences into the adaptation layer. The same upper-level world model is applicable to multiple robots, while different robots only need to provide ontology description files, calibration parameters, actuator mapping tables, contact models, and local controllers. When switching robots, the system first performs adaptive calibration: identifying degrees of freedom, workspace, joint limits, end-effector attributes, time delay model, and impedance boundaries. Then, a unified skill is mapped to the new platform.

[0107] The meta-controller addresses the lack of autonomy in humanoid robots and the inability of physical world models to learn independently. It includes at least the following modes: observational learning mode; imitation and reproduction mode; active exploration mode; latent planning execution mode; failure recovery mode; experience playback learning mode; simulation and imagination mode; human collaboration / assistance mode; and safety freeze mode. The meta-controller makes decisions based on the following signals: prediction error, model uncertainty, task urgency, historical success rate, degree of environmental change, sensor anomalies, actuator health status, user priority, and safety risk level.

[0108] The experience replay module addresses the problem of existing systems relying on large amounts of manually collected data and lacking efficient self-learning capabilities. Data generated during online operation is stored in a hierarchical manner: short-term cache memory; task-level episode memory; skill-level summary memory; long-term structured memory; a failure case library; and a rare event library. Simultaneously, a value score is calculated for each data segment, based at least on: prediction error gain, information gain, uncertainty reduction potential, task scarcity, failure recovery significance, cross-platform transfer value, and human feedback value. High-value data is prioritized for incremental training and simulation review. Furthermore, experiences can be categorized into success samples, failure samples, marginal samples, and long-tail samples; key replays are performed on the windows before and after failures for training recovery strategies and safety models; high-level experiences are shared across different robot platforms, while low-level logs related to the ontology are used only locally.

[0109] The edge-to-edge collaboration module addresses the challenges of high sampling costs and risks associated with real-world robots, as well as the difficulty in rapidly training large-scale world models. It supports joint training from the following data sources: real-world operation logs; teleoperation teaching data; publicly available videos / environmental videos; simulation-generated data; autonomous exploration data; and data from failure replays and imaginative expansion.

[0110] During training, latent states are pre-trained using large-scale observational data, latent actions are trained using action conditional data, and then corrected using real robot data. To address the differences between simulation and reality, methods such as domain randomization, style alignment, dynamic residual learning, contact model correction, and ontology embedding alignment are employed. Active exploration and long-field-of-view planning are performed at high frequency in the simulation environment; calibration is performed using low-risk data in the real environment. Edge deployment retains only the inference graph and a small-scale online adapter; periodic incremental training is performed in the cloud, followed by the transmission of weights or adaptation patches.

[0111] The safety constraint module addresses the safety and controllability issues in the online learning and autonomous planning of autonomous robots. It consists of three layers: pre-planning review, which checks whether the target exceeds its authority, is dangerous, or violates spatial and action boundaries; in-execution monitoring, which checks for collisions, pinching, instability, overcurrent, overheating, human-robot distance, and intrusion into restricted areas in real time; and post-execution auditing, which records the planning basis, model version, risk score, execution log, and human intervention points. Upon receiving a high-risk alarm, the meta-controller can switch to a safety freeze, human takeover, or restricted recovery mode.

[0112] Example 1: Adapted to a home service humanoid robot, with the task of "tidying up the table, organizing tableware, and placing cups on a tray".

[0113] The execution process includes: the robot visually recognizing objects on the table and encoding the current scene; The language task is transformed into the target latent state; The latent planning module generates the action intent of first approaching, then grabbing, then moving, and then placing; If the cup slides beyond the prediction during execution, the system immediately triggers local replanning based on tactile and visual deviations. Upon completion of the task, the newly emerging cup shape, material, and friction mode will be written into the experience pool.

[0114] Example 2: Adapted to an industrial dual-arm assembly robot, the task is to "insert flexible wire harnesses and tighten connectors".

[0115] The execution process includes: the system uses a combination of vision and force to encode the contact state; The latent space dynamics prediction module predicts pose and contact changes during the insertion process; The latent planning module generates action sub-sequences for insertion and tightening respectively; If the insertion resistance is abnormal, the recovery strategy is to switch to micro-vibration probing, angle correction, and re-insertion; When switching to another robot with a different arm span and gripper, only the body adaptation module needs to be updated to reuse the upper world model.

[0116] Example 3: Adapted to a mobile operation composite robot, whose task is to "walk to the shelf, take out the specified box and put it into the transfer box".

[0117] The execution process includes: the system first performs latent planning of the movement path, and then performs latent planning of the dual-arm pick-up and drop-off. The upper-level objectives remain consistent, but the mobile chassis and robotic arm controllers are handled by different body adapters. If the environment in front of the shelf changes during the task, the meta-controller switches to observation mode to rebuild the local world model before resuming execution.

[0118] Example 4: Adapted to a laboratory automation robot, with the task of "identifying reagent bottles, grasping them, positioning them on the reaction table, and placing them in a prescribed order".

[0119] The execution process includes: the system combining visual and textual labels with rule constraints to generate the target's latent trajectory; The safety constraint module constrains hazardous liquid areas and human-machine shared areas; For any high-risk anomalies, automatically switch to freeze mode pending manual confirmation.

[0120] The above embodiments are further elaborations and descriptions of the present invention to facilitate understanding, and are not intended to limit the present invention in any way. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-robot control method based on a physical world model, characterized in that, Including the model training phase: Historical multimodal data is encoded into historical latent states, and action data is encoded into latent actions; Train a latent space prediction model to predict future latent states based on historical latent states and latent actions. The total loss function during training includes at least prediction loss and temporal straightening loss, resulting in a trained physical world model. Task execution phase: Encode the current multimodal data into the current latent state, and parse the task input into the target latent state; Based on the current latent state, the target latent state, and the latent space prediction model, the candidate latent action sequence is expanded into future states to solve for the optimal latent action sequence. The optimal latent motion sequence is decoded into skill parameters and mapped to the corresponding robot to generate control commands.

2. The cross-robot control method based on a physical world model according to claim 1, characterized in that, It also includes a self-learning stage: During task execution, multi-dimensional decision signals are continuously collected, comprehensive decision values ​​are constructed, and corresponding self-learning modes are switched based on preset threshold conditions. In the experience replay learning mode, the value of the data is evaluated based on several value indicators, and the data with the highest value are selected for incremental training or lightweight adaptation.

3. A cross-robot control method based on a physical world model according to claim 1 or 2, characterized in that, The latent space prediction model performs time-series fusion modeling of historical latent state sequences and latent action sequences, extracts the state evolution context, learns the state transition function of latent states under the action of latent actions, and generates latent state prediction results for one or more future times using recursive expansion or parallel methods.

4. The cross-robot control method based on a physical world model according to claim 3, characterized in that, The temporal straightening loss is constructed as follows: for any three consecutive time points within the training segment, the latent state z... t z t+1 z t+2 Construct the first potential velocity vector v t equals z t+1 -z t Second latent velocity vector v t+1 equals z t+2 -z t+1 Calculate the cosine value Ct of the angle between the two, and use 1-Ct as the local curvature penalty term. Summate or average the values ​​at each time point within the training segment to obtain the temporal straightening loss.

5. A cross-robot control method based on a physical world model according to claim 1, 2, or 4, characterized in that, Encoding motion data into latent actions includes: Receive raw motion input, perform time alignment and motion segmentation, and extract intermediate features that are independent of the robot body; By using pre-defined rules, parameterized mapping networks, or a combination of both, intermediate features are encoded into corresponding semantic dimensions to form latent actions.

6. The cross-robot control method based on a physical world model according to claim 1, characterized in that, The step of parsing the task input into the target latent state includes: For different types of task inputs, corresponding encoding methods are used for encoding, and the encoding results are parsed to generate a structured task representation; a task-related object relationship graph is generated based on the current latent state and task input; the structured task representation is converted into planning constraints; and target latent states, target latent trajectories, target condition distributions, or hierarchical target graphs are generated through latent target mapping.

7. A cross-robot control method based on a physical world model according to claim 1 or 6, characterized in that, After solving for the optimal potential action sequence, a model predictive control-based rolling replanning is employed. At the current moment, the optimal latent action sequence is obtained. After executing the first K steps of the optimal latent action sequence, new multimodal data is collected and encoded to obtain a new current latent state. Starting from the new current latent state, the subsequent latent action sequence is solved again. The above process is repeated until the task is completed or terminated.

8. A cross-robot control method based on a physical world model according to claim 1 or 6, characterized in that, Decoding the optimal latent action sequence into skill parameters includes: Boundary detection is performed on continuous latent action sequences to identify action phase switching points; similarity matching is performed between latent action segments of each phase and skill prototypes in the skill library to obtain the corresponding skill selection results, and latent action segments are mapped to corresponding skill parameters.

9. A cross-robot control system based on a physical world model, applicable to the cross-robot control method as described in any one of claims 1-8, characterized in that, include: The state coding module encodes multimodal data into latent states; The motion encoding module encodes motion data into latent actions; The latent space dynamics prediction module uses a latent space prediction model to predict future latent states based on historical latent states and latent actions. The latent target generation module parses the task input into the latent state of the target; The latent planning module optimizes and solves the sequence of latent actions based on the current latent state, the target latent state, and the latent space prediction model. The skill decoding module decodes the optimal latent action sequence into skill parameters that are independent of the robot itself. The meta controller schedules and controls each module based on the system's operating status.

10. A cross-robot control system based on a physical world model according to claim 9, characterized in that, include: The body adaptation module maps and adapts the skill parameters output by the skill decoding module to the corresponding robot-generated control commands; The experience replay module stores runtime data, evaluates the value of the data based on preset value indicators, and performs incremental training or lightweight adaptation. The multimodal data acquisition module collects multimodal data of the robot in the physical environment.

Images