Robot dexterous hand control method based on multi-modal world model and robot

Abstract

The application relates to a robot dexterous hand control method and robot based on a multimodal world model, the method generates a task semantic graph according to a visual image and a language instruction by using a VLM, and maps the task semantic graph into a task semantic hidden variable through a task semantic adapter; four modal data of vision, force sense, touch and proprioceptive perception are feature-encoded, a multimodal fusion network is used for fusion, and a unified state hidden variable is obtained by fusing the fusion result and the task semantic hidden variable; a world model prediction module with a structured RSSM as a core recursively generates a future interaction track, and state, reward and risk prediction are calculated through a multi-task prediction head; a reinforcement learning strategy network generates a candidate action, a rule-based safety constraint module corrects the candidate action, and an action control parameter meeting a safety boundary is output. The method solves the conflict between complex intention understanding and high real-time safety control in a nursing scene, and realizes an interpretable, high-reliable and soft dexterous operation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a robot dexterity hand control method and robot based on a multimodal world model. Background Technology

[0002] In recent years, research on home care service robot technology has mainly focused on three aspects: multimodal perception, intelligent control, and human-robot collaboration. At the perception level, research has gradually expanded from single-vision to the fusion of vision, force, and touch to enhance the understanding of complex environments and contact situations. At the control level, reinforcement learning has become the mainstream method. At the system level, hierarchical control and safety constraint mechanisms are used to solve problems related to multi-task collaboration and safe human-robot interaction. Simultaneously, considering the unstructured and highly uncertain characteristics of the home environment, research explores key technologies such as contact dynamics modeling, adaptive control, and simulation-to-real-world transfer to improve the stability and safety of robots in complex operations. Future research will focus on developing multimodal perception fusion, contact perception world models, and adaptive control methods to enhance the safety and generalization capabilities of robots in complex human-robot interaction scenarios.

[0003] Home care robots are needed to assist humans in completing a range of daily living and care-related tasks, such as feeding water and administering medication, tidying up, cleaning, and assisting with movement. These tasks typically require the robot's dexterous hand to possess the ability to manipulate objects and people flexibly, ensuring smooth and safe movements during interaction. While home care robots are developing rapidly due to the pressure of an aging society, significant shortcomings remain in practical applications. On the one hand, systems rely heavily on visual perception, lacking effective integration of force and touch, making it difficult to achieve safe and reliable contact operations. On the other hand, existing control methods are mostly based on preset models or single-task learning, exhibiting weak generalization and adaptive capabilities, making it difficult to cope with the varied tasks and individual differences in the home environment. Furthermore, insufficient contact dynamics modeling leads to instability or excessive force during tasks such as grasping and assisting, affecting safety. Summary of the Invention

[0004] To address the problems existing in the above-mentioned traditional methods, this invention proposes a robot dexterity hand control method and robot based on a multimodal world model, which can solve the contact uncertainty problem that is difficult to handle by traditional vision methods, and enable the system to have the ability to adapt to unknown objects and complex environments in home care scenarios.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a robot dexterous hand control method based on a multimodal world model is provided, wherein the robot is equipped with at least a vision sensor, a force sensor, and a tactile sensor, and the method includes: Step 1: Based on the structured prompts and the acquired visual images and language instructions, a pre-trained VLM is used to generate a task semantic graph, and the task semantic graph is mapped to task semantic latent variables through a task semantic adapter.

[0006] Step 2: After feature encoding of the real-time acquired visual image data, force data, tactile data, and proprioceptive data, the data are input into a multimodal fusion network for cross-modal fusion. The cross-modal fusion result is then fused with the task semantic latent variables to obtain a unified state latent variable.

[0007] Step 3: Based on the unified state latent variables and candidate actions, the world model prediction module with the structured cyclic state space model as its core is used to recursively generate future interaction trajectories, and the state prediction, reward prediction and risk prediction are calculated through the multi-task prediction head.

[0008] Step 4: Based on the state prediction, reward prediction, and risk prediction, candidate action control parameters are generated through a reinforcement learning policy network; based on the state prediction and risk prediction, a safety constraint module is used to construct safety constraint conditions, which are then applied to the candidate action control parameters to output action control parameters that meet the preset safety boundary conditions, thereby driving the robot's dexterous hand to perform actions.

[0009] On the other hand, a robot is also provided, comprising: a robot body and a robot controller, wherein the robot body adopts a wheeled humanoid structure and is equipped with at least a vision sensor, a force sensor and a tactile sensor; the robot controller executes any of the above-mentioned robot dexterous hand control methods based on a multimodal world model to control the robot dexterous hand.

[0010] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned robot dexterous hand control method and robot based on a multimodal world model utilizes a VLM to generate a task semantic graph based on visual images and language instructions, which is then mapped to task semantic latent variables through a task semantic adapter. Feature encoding is performed on four modal data (visual, force, tactile, and proprioceptive), and a multimodal fusion network is used for fusion. The fusion result is then fused with the task semantic latent variables to obtain a unified state latent variable. A world model prediction module with a structured RSSM as its core recursively generates future interaction trajectories and uses a multi-task prediction head to infer state, reward, and risk predictions. A reinforcement learning policy network generates candidate actions, which are then modified by a rule-based safety constraint module, outputting action control parameters that satisfy safety boundaries.

[0011] This method employs an asynchronous parallel architecture combining slow upper-layer semantic reasoning with fast lower-layer physical prediction and control. The upper layer focuses on long-term task understanding and intent reasoning, while the lower layer focuses on high-frequency state perception and real-time action generation. This effectively resolves the time-scale conflict between complex semantic understanding and high-real-time physical interaction in nursing scenarios, achieving a stable closed loop between long-term task planning and precise action control.

[0012] A world-model-driven reinforcement learning model is employed as the VLA model architecture for the lower-level execution strategy. Visual, force, and tactile modal information are integrated to achieve the robot's intrinsic representation of the environment and contact processes. By learning the environmental dynamics model, future states are predicted in the latent space, enabling decision-making and planning, and reducing real-world trial-and-error costs. A unified state space for vision, force, and touch is constructed, and a contact perception value function is designed, incorporating grasping stability, sliding risk, and interaction safety into the optimization objectives. This ensures that the strategy not only pursues task success but also considers safety. By simulating contact processes in the world model, the prediction and optimization of contact-intensive operations are achieved, solving the contact uncertainty problem that traditional vision methods struggle to handle. This enables the system to adapt to unknown objects and complex environments in home care scenarios. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating a robot dexterous hand control method based on a multimodal world model in one embodiment; Figure 2 This is a schematic diagram of the specific structure of the task semantic adapter in one embodiment; Figure 3 This is a schematic diagram of the specific structure of the multimodal fusion module of the world model in one embodiment; Figure 4 This is a schematic diagram of the specific structure of the prediction module of the world model in one embodiment; Figure 5 This is a schematic diagram of the specific structure of a reinforcement learning strategy in one embodiment. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0017] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] In one embodiment, such as Figure 1 As shown, a robot dexterous hand control method based on a multimodal world model is provided. The robot is equipped with at least a visual sensor, a force sensor, and a tactile sensor. The robot prototype adopts a wheeled humanoid structure and may include the following processing steps 1 to 4: Step 1: Based on the structured prompts and the acquired visual images and language instructions, a pre-trained VLM is used to generate a task semantic graph, and the task semantic graph is mapped to task semantic latent variables through a task semantic adapter.

[0020] Specifically, the upper layer uses a pre-trained VLM to handle low-frequency task understanding and semantic planning.

[0021] VLM does not directly generate robot actions. Instead, based on human natural language commands and external RGB images (i.e., visual images), it outputs a task semantic graph using structured prompts. This decouples semantic reasoning from low-level control, preserving VLM's general multimodal reasoning capabilities while enabling scalable embodied task planning. For example, the task semantic graph parsed for "helping an elderly person stand up from a chair" would be... It might look like the following example: Node set: ; Edge set: ; Nodes correspond to various objects with specific attributes, and edges correspond to the relationships between nodes.

[0022] The task semantic adapter converts the high-level task semantic graph generated by the upper-level VLM into task semantic latent variables that can be used by the lower-level world model.

[0023] VLM encodes language instructions and visual scenes into a task semantic graph, and then maps them into task semantic latent variables through a task semantic adapter, driving the world model to predict contact forces, slip, balance and target state changes at several future moments.

[0024] Step 2: After feature encoding of the real-time acquired visual image data, force data, tactile data, and proprioceptive data, the data are input into a multimodal fusion network for cross-modal fusion. The cross-modal fusion result is then fused with the task semantic latent variables to obtain a unified state latent variable.

[0025] Specifically, the multimodal world model and reinforcement learning strategy employ a collaborative "prediction-decision" mechanism. The multimodal fusion module of the world model takes visual, force, tactile, ontological perception, and task semantics as inputs and calculates a unified state representation as output.

[0026] A visual encoder is used to encode features in visual image data, a force encoder to encode features in force data, a tactile encoder to encode features in tactile data, and an ontology encoder to encode features in ontology-perceived data, resulting in visual feature codes, force feature codes, tactile feature codes, and ontology-perceived feature codes. These visual, force, tactile, and ontology-perceived feature codes are then processed using a linear mapping layer and fused across modalities using a masked multimodal Transformer. The fused result is then fused with the task's semantic latent variables using a unified state encoder to obtain unified state latent variables.

[0027] The multimodal encoder consists of a visual encoder, a force encoder, a tactile encoder, and a body encoder. The multimodal fusion module of the world model consists of the multimodal encoder, a linear mapping layer, a masked multimodal Transformer, and a unified state encoder.

[0028] The multimodal fusion module of the world model uses the robot's RGB-D vision, force, touch, proprioception and task semantic information as model inputs to learn a unified state latent variable representation of the environment state, human body state and contact state. In order to solve the problems of occlusion and sensor failure in real nursing scenarios, a modal missing processing mechanism is introduced. The hidden state is completed by using the cross-modal generation relationship between vision, force and touch, so as to maintain stable control when vision is occluded or touch is missing.

[0029] Step 3: Based on the unified state latent variables and candidate actions, the world model prediction module with the structured cyclic state space model as its core is used to recursively generate future interaction trajectories, and the state prediction, reward prediction and risk prediction are simultaneously calculated through the multi-task prediction head.

[0030] Specifically, the reinforcement learning policy network generates candidate actions based on unified state latent variables, while the world model's prediction module predicts future human state, contact force, slip risk, balance stability, and robot state evolution based on these candidate actions. The long-term benefits and safety risks of candidate actions are evaluated based on the future states predicted by the world model. A safety constraint module filters and corrects actions that may lead to excessive contact force, human imbalance, or robot instability, ultimately yielding the optimal action.

[0031] The Structured Cyclic State-Space Model (Structured RSSM) is an improvement on the original RSSM. It refers to the implicit state representation and state transition model having explicit structural constraints, rather than an uninterpretable global vector, which can make the world model more stable and the representation more decoupled.

[0032] Step 4: Based on the state prediction, reward prediction, and risk prediction, candidate action control parameters are generated through a reinforcement learning policy network; based on the state prediction and risk prediction, a safety constraint module is used to construct safety constraint conditions, which are then applied to the candidate action control parameters to output action control parameters that meet the preset safety boundary conditions, thereby driving the robot's dexterous hand to perform actions.

[0033] Specifically, the reinforcement learning policy network generates candidate actions based on the unified state latent variables; and in conjunction with the prediction module of the world model, iteratively generates a deduced trajectory of a specified length. ; in, express t The unified state latent variable at time step. This represents the predicted reward at that moment. The total length of the extrapolated trajectory of a specified length.

[0034] Policy networks can use inferred trajectories for updates, without requiring the robot to actually perform these actions, which can significantly improve training efficiency.

[0035] The aforementioned robot dexterous hand control method based on a multimodal world model utilizes a Visual Model (VLM) to generate a task semantic graph from visual images and verbal instructions, which is then mapped to task semantic latent variables via a task semantic adapter. Feature encoding is performed on four modalities of data: visual, force, tactile, and proprioceptive perception. These modalities are then fused using a multimodal fusion network, and the fusion result is combined with the task semantic latent variables to obtain a unified state latent variable. A world model prediction module with a structured RSSM as its core recursively generates future interaction trajectories and uses a multi-task prediction head to infer state, reward, and risk predictions. A reinforcement learning policy network generates candidate actions, which are then modified by a rule-based safety constraint module, outputting action control parameters that satisfy safety boundaries. This method resolves the conflict between complex intent understanding and high real-time safety control in nursing scenarios, achieving interpretable, highly reliable, compliant, and dexterous operation.

[0036] In one embodiment, the pre-trained VLM uses a pre-trained Qwen3-VL-8B model.

[0037] Specifically, the pre-trained VLM uses the pre-trained Qwen3-VL-8B model as the upper-layer semantic planning VLM. This lightweight and efficient multimodal model, launched by Alibaba, achieves near-72B multimodal inference capabilities with only 8B parameters, significantly lowering the barrier to deployment. It employs a SigLIP-2 visual encoder and DeepStack feature fusion, resulting in outstanding detail awareness and support for 32K context and multilingual understanding. It demonstrates strong Chinese image and text understanding and instruction compliance capabilities, adapts to structured hints and JSON constraint decoding, making it suitable as a semantic planner, balancing performance, efficiency, and generalization.

[0038] In one embodiment, the task semantic adapter includes: a heterogeneous graph Transformer and a task semantic encoder; the task semantic encoder employs a multilayer perceptron network; mapping the task semantic graph to task semantic latent variables through the task semantic adapter includes: inputting the task semantic graph into the heterogeneous graph Transformer to obtain a global representation of the task semantic graph; and using a multilayer perceptron network to map the global representation of the task semantic graph to task semantic latent variables.

[0039] Specifically, the specific structure of the task semantic adapter is as follows: Figure 2 As shown.

[0040] (1) Heterogeneous graph Transformer Task semantic graphs are composed of various types of nodes and semantic relationships, making them typical heterogeneous graphs. General graph Transformers typically use shared parameters for information propagation, which struggles to fully characterize the semantic differences between different node and relationship types. Heterogeneous graph Transformers, by constructing attention mechanisms related to node and edge types, can learn specific representations for different types of entities and adaptively model multi-relational interactions during message passing. This allows for more accurate extraction of contact objects, interaction relationships, and task objectives, providing richer and more discriminative graph representations for generating latent semantic variables in the task.

[0041] For a certain task semantic graph Node set for: ; in, For the first i 1 node N This represents the total number of nodes.

[0042] edge set for: ; in, To connect the first i The node and the first j Edges of each node.

[0043] Establish an initial feature representation for each node. for: ; And establish edge type embedding for each relationship. for: ; in, and These represent ordinary embedding functions for nodes and relationships, respectively.

[0044] For different node types, construct type-related query, key, and value projection matrices respectively, and calculate: ; in, For the first i The query projection matrix of each node. , The first j The key-value projection matrix of each node. The first i The node and the first j The feature representation of each node. This indicates the node type; different types use different parameter matrices. and Perform mapping.

[0045] For nodes i Its neighboring nodes j Combined with edge type Introducing the relational transformation matrix, we calculate the heterogeneous attention coefficients: ; in, For nodes i Its neighboring nodes j Attention coefficient between them For normalized exponential functions, This is an attention transformation matrix related to edge type; different parameters correspond to different relationships, thus enabling the differentiation of various interaction relationships. d This is the scaling factor.

[0046] By aggregating neighborhood information using attention weights, we obtain the node update representation: ; in, For the first i The updated feature representation of each node. This is the message passing matrix corresponding to the edge type. Represents a node i The set of adjacent nodes.

[0047] The node representation update is completed by combining residual connections and feedforward networks: ; in, For the first i The node of the first l +1 round of update results, For layer normalization operations, This is for MLP network computation.

[0048] go through L After layer iteration, each node fully integrates information from multiple node types and relationships. Finally, graph-level pooling is used to obtain a global representation of the entire task semantic graph. ; in, This is a global representation of the semantic graph for the entire task. This is for graph-level pooling operations.

[0049] (2) Task semantic encoder The task semantic encoder employs a multilayer perceptron (MLP) network to... Mapped to task semantic latent variables: ; in, These are implicit variables for task semantics.

[0050] Task semantic latent variables comprehensively encode high-level semantic information such as task objectives, interaction objects, and contact relationships. These are input as conditional variables into the world model and reinforcement learning policy network, thereby enabling the effective transfer of high-level task semantics to low-level action control.

[0051] The "Heterogeneous Graph Transformer + MLP" task semantic adapter can explicitly model the complex topological relationships between heterogeneous nodes (people, objects, tools) and heterogeneous edges (spatial, temporal, causal) in the task semantic graph, and accurately map them into dense latent variables that the world model can "digest". This enables the physical predictions of the lower-level world model to have structured physical common sense, distinguish between "contact with a human body" and "contact with an object", implement differentiated compliant strategies, and significantly improve the physical rationality of the predictions.

[0052] In one embodiment, step 2 includes: processing the visual image data using a visual encoder to obtain visual feature encoding; processing the force data, tactile data, and proprioceptive data using an MLP network to obtain force feature encoding, tactile feature encoding, and proprioceptive feature encoding, respectively; processing the visual feature encoding, force feature encoding, tactile feature encoding, and proprioceptive feature encoding through a linear mapping layer to obtain fixed-dimensional lexical representations corresponding to each modality feature encoding; concatenating the lexical representations of all modalities to obtain a multimodal data lexical sequence; inputting the multimodal data lexical sequence into a masked multimodal Transformer to obtain fused observation features; concatenating the fused observation features and the task semantic latent variables and processing them using a unified state encoder to generate unified state latent variables at time t; the unified state encoder uses an MLP network.

[0053] Specifically, the "random mask training + cross-modal generative completion" mechanism overcomes the limitations of completely missing modalities (such as visual occlusion or tactile loss). Through training with a masked multimodal Transformer, the model actively learns to "imagine" the occluded visual state or missing tactile state using force perception and proprioception, generating a complete latent state representation. This "active generative completion" has an irreplaceable and stable control advantage in nursing scenarios (often accompanied by liquid splashes and limb occlusion).

[0054] In one embodiment, the visual encoder adopts an RGB-D dual-stream encoder structure, consisting of an RGB encoding branch, a depth encoding branch, and a cross-modal fusion module. The RGB encoding branch uses MobileNetV2 as the backbone network, and after removing the global average pooling layer and the classification layer, it constructs a five-level feature extraction structure. The depth encoding branch adopts the same hierarchical structure as the RGB encoding branch, but retains only two inverse residual blocks at each stage, forming a lightweight depth encoder. The visual image data is processed by the visual encoder to obtain visual feature encoding, including: inputting the visual image data into the RGB encoding branch to obtain RGB image features; inputting the visual image data into the depth encoding branch to obtain image depth features; inputting the RGB image features and image depth features into the cross-modal fusion module for cross-modal fusion, performing global average pooling on the fusion result, and aggregating the spatial dimension information into a fixed-length feature vector to obtain data feature encoding.

[0055] In one embodiment, the multimodal data lexical sequence is input into a masked multimodal Transformer to obtain fused observation features, including: defining a modal mask: ; Multiply the modality mask and the lexical representation of each modality to obtain the actual input lexical units of the Transformer for each modality; concatenate the actual input lexical units of all modalities into the Transformer to obtain a new input lexical sequence; perform cross-modal attention modeling using the Transformer based on the new input lexical sequence to capture the dependencies between modalities and obtain the output features of the Transformer; perform attention pooling on the output features of the Transformer to obtain the fused observation features: ; ; Where exp is an exponential function. ω For learnable parameter vectors, h i for H The vectors of each word in the text.

[0056] Specifically, the multimodal fusion module of the world model consists of a visual encoder, a force encoder, a tactile encoder, and a ontology encoder, forming a multimodal encoder. This multimodal encoder comprises a multimodal encoder, a linear mapping layer, a masked multimodal Transformer, and a unified state encoder. The specific structure of the multimodal fusion module of the world model is as follows: Figure 3 As shown.

[0057] (1) Multimodal data encoder The visual encoder employs a highly efficient RGB-D dual-stream encoder structure similar to the MobileSal network, consisting of an RGB encoding branch, a depth encoding branch, and a cross-modal fusion module. The RGB branch uses MobileNetV2 as its backbone, removing the global average pooling layer and classification layer to construct a five-level feature extraction structure. The depth branch uses the same hierarchical structure as the RGB branch, but retains only two inverse residual blocks at each stage, forming a lightweight depth encoder to reduce computational cost. Considering that depth maps primarily provide spatial geometry and distance priors, while RGB images contain rich semantic information, the common multi-layer fusion strategy is not adopted. Instead, a single cross-modal fusion is performed on the lowest-resolution deep features. Finally, global average pooling is applied to the fused features, aggregating spatial dimensional information into a fixed-length feature vector, achieving a unified encoding representation of visual semantic information and depth geometric information, ultimately yielding the visual feature code. .

[0058] The force encoder, tactile encoder, and proprioceptive encoder all employ MLP networks to map force, tactile, and proprioceptive perception data into corresponding modal feature codes. and .

[0059] (2) Linear mapping layer Each modal feature is encoded and converted into a fixed-dimensional lexical representation through linear mapping: ; in, For modality i Lexical representation, For modality i Feature encoding, This is the mapping matrix corresponding to each mode.

[0060] By concatenating the sequences, a complete multimodal data lexical sequence can be obtained. T : ; in, These correspond to visual, force, tactile, and proprioceptive terms, respectively.

[0061] (3) Masked Multimodal Transformer The Masked Multimodal Transformer learns cross-modal associations between vision, force, touch, and proprioception. By randomly masking some modalities and utilizing the remaining modalities for feature inference and completion, the model can generate a stable, unified state representation even under sensor occlusion, failure, or missing conditions. Compared to traditional feature concatenation methods, this structure can fully exploit complementary information between modalities, improving the robustness and generalization ability of world model state estimation, and providing reliable state input for subsequent dynamic prediction and reinforcement learning decision-making.

[0062] To enhance the robustness of the world model to modality loss, modality masking is used during training to randomly mask different modal inputs, enabling the model to learn the statistical correlations between different modalities instead of relying solely on a single modality. Studies have shown that the masking strategy can significantly improve robustness in the case of modality loss.

[0063] Define the modal mask: ; The actual vocabulary input to the Transformer is: ; Obtain a new input word sequence : ; Cross-modal attention modeling using Transformer captures inter-modal dependencies, outputting: ; in, The results of cross-modal attention modeling This indicates a normal Transformer operation.

[0064] Attention pooling is applied to the Transformer output to obtain fused observation features: ; ; in, For the first i The weights corresponding to each word element The calculated fused observation features are exp, where exp is an exponential function. ω For learnable parameter vectors, h i for H The feature integrates visual, force, tactile, and proprioceptive perception, and can maintain stable expression even when any one modality is missing.

[0065] (4) Unified state encoder The unified state encoder uses an MLP network and introduces task semantic latent variables to generate... t Unified state hidden variables at all times: ; Task semantics is introduced here, making the perceptual representation independent of task semantics and more adaptable. At the same time, task semantics affects the uniform state distribution, enabling it to reflect task priors.

[0066] In one embodiment, the world model prediction module includes an input fusion layer and a structured RSSM; the input fusion layer employs an MLP network; in the world model prediction module: the unified state latent variables and candidate actions are input into the input fusion layer; the unified state latent variables and candidate actions are concatenated and mapped to an RSSM input vector using an MLP network; the RSSM input vector and the world model latent state at time t are input into the structured RSSM; the world model latent state is updated using a gated recurrent unit network to obtain the world model latent state at the next time step; the world model latent state at the next time step is processed by an MLP network and then linearly mapped to obtain the predicted unified state latent variables.

[0067] Specifically, the specific structure of the prediction module in the world model is as follows: Figure 4 As shown.

[0068] (1) Input fusion layer The input fusion layer uses an MLP network to map the two types of inputs to RSSM input vectors: ; in, for t Consistent state hidden variables at all times This is a candidate action for that moment.

[0069] (2) Structured RSSM The structured RSSM is an improvement on the original RSSM. It means that the hidden state representation and state transition model have explicit structural constraints, rather than an uninterpretable global vector, which can make the world model more stable and the representation more decoupled.

[0070] The core of the prediction module uses a gated recurrent unit (GRU) network to update the hidden state of the world model. ; in, for t The hidden state of the world model at any given moment.

[0071] Based on the prior distribution of the unified state latent variables at the next time step calculated using an MLP network, we obtain: ; ; in, for t The mean of the distribution at time +1, which is the predicted unified state latent variable. , The weight matrix for mean prediction. This is the bias vector.

[0072] In one embodiment, the policy network for reinforcement learning adopts an MLP network; the reinforcement learning process includes: inputting unified state latent variables into the policy network to generate candidate action control parameters; based on the candidate action control parameters and the unified state latent variables, using a world model prediction module to iteratively generate a deduced trajectory of a specified length; using the world model to predict future contact force, target equilibrium, slip probability and other safety-related states using learned environmental dynamics; constructing control obstacle function constraints using a safety constraint module; and performing minimum amplitude correction on the candidate actions through quadratic programming to ensure that the final action control parameters always meet the conditions of contact force, target stability and collision distance.

[0073] Specifically, the specific structure of reinforcement learning strategies is as follows: Figure 5 As shown.

[0074] The reinforcement learning policy network uses an MLP network to generate candidate actions based on the unified state hidden variables. ; By working in conjunction with the world model prediction module, a projected trajectory of a specified length is generated iteratively: ; in, express t The unified state latent variable at time step. This indicates the predicted reward at that moment.

[0075] Policy networks can use inferred trajectories for updates, without requiring the robot to actually perform these actions, which can significantly improve training efficiency.

[0076] The world model uses learned environmental dynamics to predict future contact forces, target equilibrium, slip probability, and other safety-related states, providing predictive risk information to the safety constraint module (preferably a rule-based function module). The safety constraint module constructs safety constraints using the contact forces, target stability, and collision distance predicted by the world model, and performs minimum-amplitude corrections on candidate actions through quadratic programming, ensuring that the final control input always meets the safety constraints of contact forces, target stability, and collision distance. Both modules respectively undertake the responsibilities of risk prediction and safety assurance, jointly forming a closed-loop safety control mechanism of "prediction-evaluation-correction."

[0077] This method introduces a "rule-based function module" as a safety constraint layer, overturning the uncertain paradigm of existing methods that rely on "AI governing AI" (implicit regularization or probabilistic scoring). This module, acting as a white-box, deterministic, and interpretable hard constraint defense, utilizes physical quantities (future contact force / slippage) predicted by the world model and exercises final arbitration power over the output of the reinforcement learning strategy through preset deterministic thresholds or mathematical functions (such as dynamic impedance adjustment). This design retains the task adaptability of reinforcement learning while ensuring absolute safety boundaries through deterministic rules, highly aligning with the stringent industrial safety certification standards for nursing robots.

[0078] In one embodiment, the multi-task prediction head includes: a contact state prediction head, a target state prediction head, an ontology state prediction head, a reward prediction head, and a risk prediction head; the contact state prediction head, the target state prediction head, the ontology state prediction head, the reward prediction head, and the risk prediction head all adopt MLP-based prediction heads.

[0079] Specifically, multiple contact state prediction heads, target state prediction heads, ontology state prediction heads, reward prediction heads, and risk prediction heads based on MLP networks are established to predict the system's state at the next moment, including: Contact state prediction: ; Target state prediction: ; Ontology state prediction: ; Reward Prediction: ; Risk prediction: ; These predictions are primarily used for training optimization and security constraints of reinforcement learning strategies, enabling the world model and reinforcement learning strategies to work in synergy.

[0080] Multiple data types (images, joint angles, force, touch, and verbal commands) are collected through human behavior videos, teleoperation systems, and simulation platforms, with unified timestamps and coordinate systems. Data cleaning is performed to remove abnormal trajectories, and the data is segmented and labeled, breaking long sequences into skill fragments and automatically generating task semantic labels. Multimodal perception data is aligned and normalized, encoding visual, motion, and tactile information into a unified data format to construct a standardized dataset, providing high-quality input for model training.

[0081] The training process of the model used in the robot dexterous hand control method based on a multimodal world model includes: The training process employs a "staged pre-training + world model reinforcement learning joint optimization + safety constraint fine-tuning" approach. First, the world model is pre-trained using multimodal robot interaction data. Simultaneously, a task semantic adapter is trained using task semantic graph data to generate task-related latent variables. Then, the world model is jointly trained with a reinforcement learning policy, updating the world model using real interaction trajectories and performing imaginary trajectory rolling within the learned latent space. The policy network is optimized by predicting future states and rewards. Finally, a safety constraint module is introduced, constructing safety constraints using contact forces, target stability, and collision distances predicted by the world model.

[0082] This method forms a complete technical closed loop: "VLM semantic topology parsing → adapter structured driving → prediction module based on structured RSSM to achieve multi-physical quantity prediction → rule-based safety hard constraints". The functions of each link are coupled and causally progressive, which together realize the high-order intelligent behavior of the robot's dexterous hand in complex and unstructured nursing environments, which can "understand the intention, see the topology, measure the physics accurately, and control the safety".

[0083] It should be understood that, although the above Figure 1 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above... Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0084] In one embodiment, a robot is also provided, the robot comprising: a robot body and a robot controller, the robot body adopting a wheeled humanoid structure and equipped with at least a vision sensor, a force sensor and a tactile sensor; the robot controller executing the above-described robot dexterous hand control method based on a multimodal world model to control the robot dexterous hand.

[0085] Specifically, the robot controller uses a multimodal world model-based robot dexterous hand control method, which involves a variety of models and a considerable number of parameters. The environmental and ontological perception data modalities are complex, and the application scenarios have high requirements for real-time performance and reliability, resulting in a large overall computing power demand. Therefore, it is deployed on the robot end side based on a high-performance computing platform (such as NVIDIA Jetson AGX Thor) to ensure smooth system operation.

[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. A robot dexterous hand control method based on a multimodal world model, characterized in that, The robot is equipped with at least a vision sensor, a force sensor, and a tactile sensor, and the method includes: Step 1: Based on the structured prompts and the acquired visual images and language instructions, a pre-trained VLM is used to generate a task semantic graph, and the task semantic graph is mapped to task semantic latent variables through a task semantic adapter; Step 2: After feature encoding of the real-time acquired visual image data, force data, tactile data, and proprioceptive data, the data are input into a multimodal fusion network for cross-modal fusion. The cross-modal fusion result is then fused with the task semantic latent variables to obtain a unified state latent variable. Step 3: Based on the unified state latent variables and candidate actions, the world model prediction module with the structured cyclic state space model as its core is used to recursively generate future interaction trajectories, and the state prediction, reward prediction and risk prediction are simultaneously calculated through the multi-task prediction head. Step 4: Based on the state prediction, reward prediction, and risk prediction, candidate action control parameters are generated through a policy network of reinforcement learning; based on the state prediction and risk prediction, a safety constraint module is used to construct safety constraint conditions, which are then applied to the candidate action control parameters to output action control parameters that meet the preset safety boundary conditions, so as to drive the robot's dexterous hand to perform actions. Specifically, step 2 includes: processing the visual image data using a visual encoder to obtain visual feature codes; and processing the force data, tactile data, and proprioceptive data using an MLP network to obtain force feature codes, tactile feature codes, and proprioceptive feature codes, respectively. The visual feature encoding, force feature encoding, tactile feature encoding, and ontology perception feature encoding are processed through a linear mapping layer to obtain fixed-dimensional lexical representations corresponding to each modality feature encoding. The lexical representations of all modalities are concatenated to obtain a multimodal data lexical sequence. The multimodal data lexical sequence is input into the masked multimodal Transformer to obtain fused observation features; The fused observation features and the task semantic latent variables are concatenated and then processed by a unified state encoder to generate unified state latent variables at time t; the unified state encoder uses an MLP network.

2. The robot dexterous hand control method based on a multimodal world model according to claim 1, characterized in that, The pre-trained VLM uses pre-trained Qwen3. VL-8B model.

3. The robot dexterous hand control method based on a multimodal world model according to claim 1, characterized in that, The task semantic adapter includes a heterogeneous graph Transformer and a task semantic encoder; the task semantic encoder employs a multilayer perceptron network. Mapping the task semantic graph to task semantic latent variables through a task semantic adapter includes: The task semantic graph is input into the heterogeneous graph Transformer to obtain a global representation of the task semantic graph; A multilayer perceptron network is used to map the global representation of the task semantic graph into latent task semantic variables.

4. The robot dexterous hand control method based on a multimodal world model according to claim 1, characterized in that, The visual encoder adopts an RGB-D dual-stream encoder structure, which consists of an RGB encoding branch, a depth encoding branch, and a cross-modal fusion module. The RGB encoding branch uses MobileNetV2 as the backbone network, and after removing the global average pooling layer and the classification layer, it constructs a five-level feature extraction structure. The depth coding branch adopts the same hierarchical structure as the RGB coding branch, but only two inverted residual blocks are retained in each stage, forming a lightweight depth encoder; Visual image data is processed using a visual encoder to obtain visual feature encoding, including: The visual image data is input into the RGB encoding branch to obtain the RGB image features; Visual image data is input into the depth coding branch to obtain image depth features; RGB image features and image depth features are input into the cross-modal fusion module for cross-modal fusion. The fusion result is then subjected to global average pooling to aggregate spatial dimensional information into a fixed-length feature vector, thus obtaining the data feature encoding.

5. The robot dexterous hand control method based on a multimodal world model according to claim 1, characterized in that, The multimodal data lexical sequence is input into a masked multimodal Transformer to obtain fused observation features, including: Define the modal mask: Multiply the modality mask by the lexical representation of each modality to obtain the actual lexical input to the Transformer for each modality; Concatenate all the actual input words of all modalities into the Transformer to obtain a new input word sequence; Based on the new input word sequence, cross-modal attention modeling is performed using Transformer to capture the dependencies between modalities and obtain the output features of Transformer; Attention pooling is applied to the output features of the Transformer to obtain fused observation features: Where exp is an exponential function. ω For learnable parameter vectors, h i for H The vectors of each word in the text.

6. The robot dexterous hand control method based on a multimodal world model according to claim 1, characterized in that, The world model prediction module includes an input fusion layer and a structured RSSM; the input fusion layer uses an MLP network. In the world model prediction module: The unified state hidden variables and candidate actions are input into the input fusion layer, and the unified state hidden variables and candidate actions are concatenated and mapped into RSSM input vectors using an MLP network. The RSSM input vector and the world model hidden state at time t are input into the structured RSSM, and the world model hidden state is updated using a gated recurrent unit network to obtain the world model hidden state at the next time step. The world model hidden state at the next time step is processed by an MLP network and then linearly mapped to obtain the predicted unified state hidden variables.

7. The robot dexterous hand control method based on a multimodal world model according to claim 1, characterized in that, The policy network for reinforcement learning employs an MLP network; The objective function of the policy network includes a task completion reward term and an interaction security constraint term. The reinforcement learning process includes: The unified state latent variables are input into the policy network of reinforcement learning to generate candidate action control parameters; Based on the candidate action control parameters and the unified state latent variables, a deduced trajectory of a specified length is iteratively generated using the world model prediction module. Based on the world model, the learned environmental dynamics are used to predict future contact force, target equilibrium, slip probability and other safety-related states. Safety constraint conditions are constructed using a safety constraint module, and candidate actions are modified with minimum amplitude through quadratic programming so that the final action control parameters always meet the conditions of contact force, target stability and collision distance.

8. The robot dexterous hand control method based on a multimodal world model according to claim 1, characterized in that, The multi-task prediction head includes: a contact state prediction head, a target state prediction head, an ontology state prediction head, a reward prediction head, and a risk prediction head; the contact state prediction head, the target state prediction head, the ontology state prediction head, the reward prediction head, and the risk prediction head all adopt MLP-based prediction heads.

9. A robot, characterized in that, The robot includes: a robot body and a robot controller. The robot body adopts a wheeled humanoid structure and is equipped with at least a vision sensor, a force sensor, and a tactile sensor. The robot controller executes the robot dexterous hand control method based on a multimodal world model as described in any one of claims 1 to 8 to control the robot dexterous hand.

Images