A humanoid robot scene adaptive task planning method based on a multi-modal-motion coupled world model and related equipment

Abstract

The application provides a humanoid robot scene adaptive task planning method based on a multi-modal-motion coupling world model and related equipment, and belongs to the field of artificial intelligence and robot technology. The application comprises: receiving a task instruction, collecting multi-modal data, and uniformly encoding to obtain multi-modal fusion embedded data; performing feature correlation on the multi-modal fusion embedded data through an improved multi-modal-motion coupling world model to obtain a text action instruction and a multi-modal feature vector; performing motion constraint oriented feature mapping according to the multi-modal feature vector to obtain a conditional feature vector; generating a fusion graph to obtain the fusion graph; taking the text action instruction as a new task instruction and taking the fusion graph as new multi-modal data, returning to the step of receiving a task instruction and collecting multi-modal data for uniform encoding until the output fusion graph meets a preset condition. The application can improve the action coordination, scene adaptability and task success rate of the humanoid robot.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and robotics, and in particular to a humanoid robot scene adaptive task planning method and related equipment based on a multimodal-motion coupled world model. Background Technology

[0002] In related technologies, task planning methods for humanoid robots often rely on a single visual modality, which leads to low task execution accuracy. Most planning systems depend on pre-built environment maps or simple state machines, whose internal models can only simulate the static properties of the environment and cannot handle dynamically changing environments.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a humanoid robot scene adaptive task planning method and related equipment based on a multimodal-motion coupled world model, which can realize efficient autonomous task planning of humanoid robots in industrial and household scenarios, and improve motion coordination, scene adaptability and task success rate.

[0005] To achieve the above objectives, one aspect of this application proposes an adaptive task planning method for humanoid robots based on a multimodal-motion coupled world model, the method comprising the following steps: After receiving task instructions and collecting multimodal data, the data is uniformly encoded to obtain multimodal fusion and embedding data; By performing feature association on the multimodal fusion and embedding data using an improved multimodal-motion coupled world model, text action instructions and multimodal feature vectors are obtained. Based on the multimodal feature vectors, motion constraint-guided feature mapping is performed to obtain conditional feature vectors; Based on the conditional feature vector and the improved scene adaptive diffusion model, a fusion map is generated to obtain a fusion map that couples the environmental state and limb movement. The text action command is used as a new task command, and the fused graph is used as new multimodal data. The process of receiving the task command and collecting multimodal data is returned and then uniformly encoded to obtain multimodal fused embedded data. This process continues until the output fused graph meets the preset conditions.

[0006] In some embodiments, the step of receiving task instructions and collecting multimodal data followed by unified encoding to obtain multimodal fusion embedded data includes: Receive task instructions, map the task instructions to the embedding space through text encoding, and obtain task instruction embedding data; Multimodal data of the current environment is acquired through acquisition devices mounted on a humanoid robot; the acquisition devices include an RGB-D camera, a six-dimensional force sensor of an end effector, and a flexible tactile array of mechanical fingers; the multimodal data includes environmental visual images, force sequence data, and tactile array data; The environmental visual image is mapped to the embedding space through visual encoding to obtain visual feature embedding data; The force sequence data is mapped to the embedding space through force coding to obtain force feature embedding data; The tactile array data is encoded and projected into the embedding space to obtain tactile feature embedding data; The task instruction embedding data, the visual feature embedding data, the force feature embedding data, and the tactile feature embedding data are used as multimodal fusion embedding data.

[0007] In some embodiments, the step of performing feature association on the multimodal fused embedding data through an improved multimodal-motion coupled world model to obtain text action instructions and multimodal feature vectors includes: An improved multimodal-motion coupled world model is constructed based on a large language model; the multimodal-motion coupled world model includes a TransformerDecoder structure with a preset number of layers and a limb kinematics constraint module; the limb kinematics constraint module is used to set the robot joint angle corresponding to the text action command within a preset limit range when generating text action commands; The multimodal fusion and embedding data is input into the multimodal-motion coupled world model, and the multimodal features are associated with the task instructions through a self-attention mechanism to generate text action instructions and multimodal feature vectors; the multimodal feature vectors are structured vectors that fuse visual features, force features and tactile features.

[0008] In some embodiments, performing motion-constraint-guided feature mapping based on the multimodal feature vector to obtain a conditional feature vector includes: The dimensions of the multimodal feature vectors are converted to a preset dimension size, and the joint angles corresponding to the multimodal feature vectors are verified to meet the preset conditions to obtain verification information. Based on the verification information, if the joint angle exceeds the range, the feature value is adjusted; if the verification passes, a conditional feature vector is output. The conditional feature vector includes environmental features, multimodal perception features, and limb movement constraint information.

[0009] In some embodiments, the generation of the fusion map based on the conditional feature vector and the improved scene adaptive diffusion model to obtain a fusion map coupling the environmental state and limb movement includes: An improved scene-adaptive diffusion model is constructed based on the diffusion model; the scene-adaptive diffusion model includes scene weight factors and a U-Net decoder; the scene weight factors include visual modality weights, force modality weights and tactile modality weights; The conditional feature vector is input into the scene adaptive diffusion model, and the scene weight factor is used to add noise forward to obtain noisy data. Based on the conditional feature vector and the noisy data, the U-Net decoder is used to perform reverse denoising to generate a fusion map of coupled environmental state and limb movement.

[0010] In some embodiments, the formula used to input the conditional feature vector into the scene adaptive diffusion model and perform forward noise addition using the scene weight factor to obtain noisy data includes: ; in, The data with added noise for the current iteration number; This is the noisy data from the previous iteration; , For noise variance; It is Gaussian noise; As a scene weighting factor; This is the scene feature vector.

[0011] In some embodiments, the formula used to generate a fusion map of coupled environmental state and limb movement by performing inverse denoising using the U-Net decoder based on the conditional feature vector and the noisy data includes: ; in, This is a fusion diagram that couples environmental states with limb movements; The conditional feature vector; It follows a Gaussian distribution; The data with added noise for the current iteration number; This is the noisy data from the previous iteration; This is the mean value after noise reduction; The variance parameter after denoising; This is the ordinal number of the noise reduction step.

[0012] To achieve the above objectives, another aspect of this application proposes a humanoid robot scene adaptive task planning device based on a multimodal-motion coupled world model, used to implement the method described above. The device includes: The first module is used to receive task instructions and collect multimodal data, then encode it uniformly to obtain multimodal fusion and embedding data; The second module is used to perform feature association on the multimodal fusion and embedding data through an improved multimodal-motion coupled world model to obtain text action instructions and multimodal feature vectors; The third module is used to perform motion constraint-guided feature mapping based on the multimodal feature vectors to obtain conditional feature vectors; The fourth module is used to generate a fusion map based on the conditional feature vector and the improved scene adaptive diffusion model, so as to obtain a fusion map of coupled environmental state and limb movement. The fifth module is used to take the text action instruction as a new task instruction, take the fusion graph as new multimodal data, return to the step of receiving the task instruction and collecting multimodal data and then uniformly encoding it to obtain multimodal fusion embedded data, until the output fusion graph meets the preset conditions.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0016] The embodiments of this application include at least the following beneficial effects: This application provides a humanoid robot scene adaptive task planning method, device, electronic device, storage medium, and program product based on a multimodal-motion coupled world model. The solution includes: receiving task instructions and collecting multimodal data, then uniformly encoding it to obtain multimodal fusion embedded data; performing feature association on the multimodal fusion embedded data through an improved multimodal-motion coupled world model to obtain text action instructions and multimodal feature vectors; performing motion constraint-guided feature mapping based on the multimodal feature vectors to obtain conditional feature vectors; generating a fusion graph to obtain a fusion graph; using the text action instructions as new task instructions and the fusion graph as new multimodal data, returning to the step of receiving task instructions and collecting multimodal data, and uniformly encoding, until the output fusion graph meets preset conditions. This application can improve the humanoid robot's motion coordination, scene adaptability, and task success rate. Attached Figure Description

[0017] Figure 1This is a flowchart of the humanoid robot scene adaptive task planning method based on a multimodal-motion coupled world model provided in the embodiments of this application; Figure 2 This is an overall data flow diagram of the humanoid robot scene adaptive task planning method based on a multimodal-motion coupled world model provided in the embodiments of this application; Figure 3 This is a structural diagram of the multimodal coding module provided in an embodiment of this application; Figure 4 This is a structural diagram of the improved DeepSeek-Motion model provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the working principle of the scene adaptive diffusion model provided in the embodiments of this application; Figure 6 This is a schematic diagram of the assembly and integration sequence of industrial motor end caps provided in the embodiments of this application; Figure 7 This is a schematic diagram of the home-use elderly water cup delivery fusion sequence provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0021] 1) The multimodal-motion coupled world model (DeepSeek-Motion), based on an improved large language model, can simultaneously understand and simulate the complex relationships between environmental states, multi-sensor information, and the robot's own motion; 2) Large language models, which are large-scale deep learning models trained on massive amounts of text data, including DeepSeek-7B, GPT series (such as GPT-4, GPT-3.5-turbo) and LLaMA series and their derivative models, etc. 3) WordPiece, a word segmentation algorithm, is a technique used in natural language processing to break down words into smaller sub-units; 4) BPE (Byte Pair Encoding), a word segmentation algorithm; 5) ByteLevelBPE (Byte-level BPE), a variant of BPE that operates at the byte level; 6) ViT (Vision Transformer), a model that applies the Transformer architecture originally used for natural language processing to the field of computer vision; 7) Q-Former (Querying Transformer): A component used to bridge the visual encoder and the large language model; 8) MLP (Multi-Layer Perceptron): This machine is responsible for mapping the feature vectors output by different modal encoders to the same dimension so that subsequent feature fusion can be performed. 9) Encoder: A part of a neural network or model responsible for compressing input data (such as images or text) and converting it into a more abstract, denser low-dimensional feature representation; 10) Transformer, a deep learning model architecture based on self-attention mechanism; 11) TransformerDecoder, the Transformer decoder, is used to generate tasks; 12) DH parameters (Denavit–Hartenberg parameters), a standard method in robotics for describing the geometric relationships between links in a serial robot; 13) FeedForward, a sub-module that follows the self-attention layer in each encoder / decoder layer of the Transformer; 14) mismatch, specifically refers to a mismatch in feature dimensions or semantic space; 15) 1D-CNN, one-dimensional convolutional neural network; 16) RGB-D camera, where RGB represents the three primary colors of red, green, and blue, and D represents depth; 17) U-Net decoder, U-shaped network decoder.

[0022] This application considers that humanoid robots, as a new generation of intelligent equipment, possess core value in simulating human limb movement and environmental interaction, and have broad application prospects in industrial automation and home services. Current task planning methods for humanoid robots mainly suffer from the following technical bottlenecks: 1. Limited and insufficiently integrated perceptual modality: Most related methods rely on a single visual modality, but precision assembly in industrial scenarios requires force sensing of bolt tightening torque (e.g., the torque error in motor end cover assembly needs to be ≤ ±0.1N). m) In home settings, caring for the elderly requires tactile perception of the support force (to avoid injury), but the lack of multimodal coordination of vision, force, and touch leads to low task execution accuracy.

[0023] 2. Disconnect between large language models and kinematic constraints: When related technologies use large language models (such as Llama2 and DeepSeek) to generate motion commands, they fail to combine them with the kinematic characteristics of humanoid robot limbs (such as joint angle limits and link length constraints), resulting in unexecutable commands and a high task failure rate.

[0024] 3. Poor scene adaptability and lack of dedicated world models: Industrial scenarios require priority to ensure force perception accuracy (such as torque control), while home scenarios require priority to ensure visual and tactile safety (such as avoiding collisions with the elderly). However, existing methods do not design adaptive mechanisms for different scenarios, and the uniform planning strategy leads to an efficiency drop of more than 50% when applied across scenarios. At the same time, existing world models only simulate environmental states and lack coupled modeling of "multimodal perception-limb movement-environmental states", which makes it impossible to predict the impact of actions on the environment and results in decision lag in dynamic scenarios (such as the inability to predict the collision risk of robotic arm movement on workpieces in industrial handling).

[0025] In view of this, this application provides a humanoid robot scene adaptive task planning method and related equipment based on a multimodal-motion coupled world model to solve the above-mentioned technical problems. This solution combines multimodal perception fusion, kinematic constraints, scene adaptation and coupled world model, involving the intersection of artificial intelligence and robotics technology, specifically belonging to the field of humanoid robot task planning technology based on multimodal-motion coupled world model, and is particularly suitable for industrial manufacturing (precision assembly, parts handling) and smart home (household chores, elderly care) scenarios. The core of the humanoid robot's autonomous decision-making, multimodal perception fusion and limb movement collaborative control is to improve the adaptability of complex scenarios through "multimodal-motion-environment" coupled modeling.

[0026] The purpose of this application is to provide a humanoid robot scene adaptive task planning method and related equipment based on a multimodal-motion coupled world model. By constructing a three-in-one coupled world model of "multimodal perception-limb movement-environmental state", the method enables humanoid robots to achieve efficient autonomous task planning in industrial and household scenarios, thereby improving motion coordination, scene adaptability and task success rate.

[0027] Figure 1 This is an optional flowchart of a humanoid robot scene adaptive task planning method based on a multimodal-motion coupled world model provided in this application embodiment. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0028] Step S101: After receiving the task instruction and collecting multimodal data, the data is uniformly encoded to obtain multimodal fusion and embedding data; Step S102: Feature association is performed on the multimodal fusion embedded data through the improved multimodal-motion coupled world model to obtain text action instructions and multimodal feature vectors; Step S103: Perform motion constraint-guided feature mapping based on the multimodal feature vectors to obtain conditional feature vectors; Step S104: Generate a fusion map based on the conditional feature vector and the improved scene adaptive diffusion model to obtain a fusion map of coupled environmental state and limb movement. Step S105: Take the text action instruction as the new task instruction, take the fusion graph as the new multimodal data, return to receive the task instruction and collect the multimodal data and encode it uniformly to obtain the multimodal fusion embedded data, until the output fusion graph meets the preset conditions.

[0029] Steps S101 to S105, as illustrated in the embodiments of this application, involve using a model to transform task instructions into specific actions and a fusion graph. The fusion graph is then sent back to the starting point as new data to initiate a new round of cognition and planning. The steps include: (1) Receive task instructions and acquire multimodal data of the current environment through the acquisition devices carried by the humanoid robot; the acquisition devices include an RGB-D camera (acquiring visual images of the environment), a six-dimensional force sensor of the end effector (acquiring force sequence data), and a flexible tactile array of mechanical fingers (acquiring tactile array data); map the task instructions to the embedding space through text encoding to obtain task instruction embedding data. Visual images are mapped to an embedding space through visual encoding to obtain visual feature embedding data. Force sequence data is mapped to the embedding space using force encoding (3-layer 1D-CNN) to obtain force feature embedding data. The tactile array data is mapped to the embedding space through tactile encoding (4-layer Transformer encoder) to obtain tactile feature embedding data. ; (2) After uniformly mapping to the same dimension d (d=512 or 1024) using a single-layer MLP, the resulting multimodal fusion embedding is obtained. The DeepSeek-Motion model (a multimodal-motion coupled world model, based on the DeepSeek basic model, integrating a limb kinematic constraint module based on DH parameters, and adapted to DeepSeek's Transformer pre-normalized layer structure) is an improved large language model that integrates multimodal features with task instructions through a self-attention mechanism, combined with limb kinematic constraints. Generate text action commands and multimodal feature vectors. ; (3) The input feature mapping module transforms the dimensions using an MLP, then performs encoding and decoding using a motion-constrained Transformer (6 Encoder layers + 6 Decoder layers, with joint angle constraint layers at the Decoder end) to generate conditional feature vectors. ; (4) Adopting a scene-adaptive diffusion model, based on... and scene weighting factors , (Corresponding to the visual, force, and tactile modal weights respectively), a "environmental state-limb movement" fusion map is generated; during the forward noise addition process of the improved scene adaptive diffusion model, the noise variance is dynamically adjusted according to the scene. (Industrial precision assembly scenario) Home care scenarios ); (5) Use the text action instructions generated in step (2) as the new task instructions in step (1), and use the fusion graph generated in step (4) as the new multimodal data input source in step (1). Iterate through steps (1)-(4) until the structural similarity index (SSIM) of the fusion graph generated in two consecutive steps is ≥98% or the number of iterations is ≥8. Stop the iteration and output the complete action sequence and environmental state record.

[0030] These five steps together form a closed loop of "perception-cognition-planning-verification-iteration", which makes the method both universal and scenario-specific.

[0031] In some embodiments, step S101 may include, but is not limited to, steps S111 to S116: Step S111: Receive task instructions, map the task instructions to the embedding space through text encoding, and obtain task instruction embedding data; Step S112: Acquire multimodal data of the current environment through the acquisition device carried by the humanoid robot; the acquisition device includes an RGB-D camera, a six-dimensional force sensor of the end effector, and a flexible tactile array of mechanical fingers; the multimodal data includes environmental visual images, force sequence data, and tactile array data; Step S113: Map the environmental visual image to the embedding space through visual encoding to obtain visual feature embedding data; Step S114: Map the force sequence data to the embedding space through force coding to obtain force feature embedding data; Step S115: The tactile array data is encoded into the embedding space to obtain tactile feature embedding data; Step S116: Use task instruction embedded data, visual feature embedded data, force feature embedded data and tactile feature embedded data as multimodal fusion embedded data.

[0032] In some embodiments, steps S111-S116 specifically involve: inputting the task instruction X into the WordPiece module and segmenting it into a character set using the BPE algorithm. (m is the number of characters after word segmentation), converted into a word set based on the DeepSeek pre-built vocabulary (containing 32k words). ( x For character sets X The elements in the text are replaced with DeepSeek's ByteLevelBPE (which supports mixed Chinese and English command segmentation), and then mapped to the embedding space R through linear embedding to obtain the task command embedding data. ( d (Represents dimension).

[0033] Visual encoding employs a "ViT+Q-Former+MLP" structure: dividing the visual image into n parts. Non-overlapping image patches (P=16 or 32) are processed by ViT (12-layer encoder, 8 attention heads) to extract global semantic features, Q-Former (containing 64 learnable query vectors) to filter task-relevant features, and finally mapped to the embedding space R by MLP to obtain visual feature embedding data. (c is the number of image channels, c=3 for RGB images).

[0034] The parameters of the 3-layer 1D-CNN for force coding are as follows: kernel sizes of 3, 5, and 3, respectively; number of kernels of 32, 64, and 128, respectively; activation function of ReLU; and output force feature embedding data. .

[0035] The parameters of the 4-layer Transformer encoder for haptic coding are: 8 attention heads, 256 hidden layer dimensions, and output... .

[0036] This embodiment standardizes and modularizes the entire data preprocessing process by specifying the specific architecture and parameters (such as the number of layers and the number of attention heads) of each encoder, making the technical solution reproducible.

[0037] In some embodiments, step S102 may include, but is not limited to, steps S201 to S202: Step S201: An improved multimodal-motion coupled world model is constructed based on the large language model. The multimodal-motion coupled world model includes a TransformerDecoder structure with a preset number of layers and a limb kinematics constraint module. The limb kinematics constraint module is used to set the robot joint angles corresponding to the text action instructions within a preset limit range when generating text action instructions. Step S202: Input the multimodal fusion embedded data into the multimodal-motion coupled world model, and associate the multimodal features with the task instructions through the self-attention mechanism to generate text action instructions and multimodal feature vectors; the multimodal feature vectors are fused with visual features, force features and tactile features to form a structured vector.

[0038] In steps S201-S202 of some embodiments, the limb kinematic constraint module of DeepSeek-Motion (multimodal-motion coupled world model) models the shoulder, elbow, and wrist joints of the humanoid robot based on DH parameters, and the constraint formula is as follows: ( For joint angle, , The limits of joint movement, such as the pitch angle of the shoulder joint. , The loss function of the model is: ; Where m is the total number of characters in the generated text action command. For the generated text action instruction characters, Pre-trained parameters for DeepSeek (based on DeepSeek-7B or DeepSeek-R1 weights, specifically the attention head parameters and FeedForward layer weights of the 12-layer TransformerDecoder). These are the kinematic constraint parameters for joint angle limit training. The input probability needs to be calculated after being uniformly mapped to the same dimension d through a 1-layer MLP.

[0039] The multimodal feature vector output in this embodiment not only integrates environmental features but also includes the motion intent planned to achieve the task.

[0040] In some embodiments, step S103 may include, but is not limited to, steps S301 to S302: Step S301: Convert the dimension of the multimodal feature vector to a preset dimension size, verify whether the joint angles corresponding to the multimodal feature vector meet the preset conditions, and obtain verification information; Step S302: Based on the verification information, if the joint angle exceeds the range, adjust the feature value; if the verification is passed, output the conditional feature vector. The conditional feature vector includes environmental features, multimodal perception features, and limb movement constraint information.

[0041] In some embodiments, steps S301-S302 yield the feature mapping formula for the conditional feature vector: ; in, For conditional eigenvectors, For multimodal feature vectors, A linear adaptation layer is required to ensure matching with DeepSeek's feature dimensions (1024 dimensions) and avoid mismatch in the input dimensions of the diffusion model. For MLP parameters (input dimension d, output dimension 2048). Transformer encoding-decoding parameters, The 128-dimensional motion constraint query vector is generated by pre-training on the joint angle limit dataset.

[0042] This embodiment converts the dimension to a preset dimension size, so that task planning from different sources and with different levels of complexity can be adapted to a unified input format expected by the scene adaptive diffusion model. The output conditional feature vector includes environmental features, multimodal perception features and limb motion constraint information, providing guidance for the scene adaptive diffusion model.

[0043] In some embodiments, step S104 may include, but is not limited to, steps S401 to S403: Step S401: An improved scene-adaptive diffusion model is constructed based on the diffusion model; the scene-adaptive diffusion model includes scene weight factors and a U-Net decoder; the scene weight factors include visual modal weights, force modal weights and tactile modal weights; Step S402: Input the conditional feature vector into the scene adaptive diffusion model, and use the scene weight factor to add noise forward to obtain the noisy data; Step S403: Based on the conditional feature vector and the noisy data, reverse denoising is performed using the U-Net decoder to generate a fusion map of the coupled environmental state and limb movement.

[0044] In steps S401-S403 of some embodiments, the introduction of scene weighting factors transforms the differentiated requirements of industrial scenarios (high force perception weight) and home scenarios (high visual / tactile perception weight) from a vague concept into precise parameters that can be quantified and embedded in the model. Using the U-Net decoder, both global context and detailed information are captured, generating a fusion map rich in detail. The U-Net decoder can be replaced with other decoders, such as VAE decoders and Transformer-based decoders. Different scene weighting factors lead to different initial noise distributions, allowing industrial tasks to be generated with a greater emphasis on force perception accuracy, while home tasks are more focused on safety and obstacle avoidance. Guided by conditional feature vectors, the model progressively denoises random noise, generating a structured fusion map. The fusion map not only includes environmental objects but also the robot's motion state information (such as joint angles and end-effector pose), achieving a unified representation of the environment and motion.

[0045] In some embodiments, the formula used in step S402 includes: ; in, The data with added noise for the current iteration number; This is the noisy data from the previous iteration; , For noise variance; It is Gaussian noise; As a scene weighting factor; This is the scene feature vector.

[0046] In step S402 of some embodiments, industrial scenario Home use scenario Industrial scenarios Home use scenario , It is a 64-dimensional scene feature vector (industrial scene includes bolt torque threshold and assembly gap threshold, household scene includes water cup grip force threshold and tilt threshold).

[0047] In some embodiments, the formula used in step S403 includes: ; in, This is a fusion diagram that couples environmental states with limb movements; These are conditional feature vectors; It follows a Gaussian distribution; The data with added noise for the current iteration number; This is the noisy data from the previous iteration; This is the mean value after noise reduction; The variance parameter after denoising; This is the ordinal number of the noise reduction step.

[0048] The method of this application embodiment can be applied to industrial scenarios (precision assembly of motor end caps, handling of heavy parts) or household scenarios (delivering water cups to the elderly, floor cleaning); when applied to industrial assembly scenarios, the torque is calculated by combining the Z-axis force value collected by the six-dimensional force sensor with the radius of the robotic arm end (torque = force value × radius), and the tightening action is adjusted in real time based on feedback; when applied to household delivery scenarios, the tilt of the water cup is calculated by converting the pressure difference of the flexible tactile array (…). , (For pressure difference), ensure the tilt angle is ≤5°.

[0049] As an optional implementation, the core technical solution of this application includes five key steps, each revolving around a "coupled world model," as referenced. Figure 2 The details are as follows: Step 1: Multimodal Data Acquisition and Unified Encoding Data Acquisition: Multimodal data is acquired through specialized equipment mounted on the humanoid robot—an RGB-D camera (1920×1080 resolution) captures environmental visual images (such as motor housings in industrial settings, and coffee tables and water cups in household settings); a six-dimensional force sensor (500Hz sampling frequency) acquires force sequence data from the end effector (such as tightening force during assembly, and gripping force when picking up a cup); a 64×64 flexible tactile array acquires tactile array data from the mechanical fingers (such as bolt contact pressure, and pressure distribution when a water cup is tilted); and the robot also receives natural language task commands input by the user (such as "Industrial: Assemble M5 bolts, torque 4N"). m" "Household use: handing a water cup to the elderly, water temperature 40℃").

[0050] refer to Figure 3 Uniform coding: Text encoding: WordPiece + BPE algorithm segmentation → DeepSeek pre-built vocabulary (32k vocabulary) mapping → linear embedding, output ( or ); Visual encoding: Image segmentation (16×16 or 32×32) → ViT (12-layer encoder) extracts global features → Q-Former (64 learnable query vectors) filters task-relevant features → MLP mapping, output ; Force encoding: A 3-layer 1D-CNN (3 / 5 / 3 kernels, 32 / 64 / 128 kernels) processes force sequences → global average pooling → linear mapping, outputting force sequences. ; Haptic encoding: A 4-layer Transformer encoder (8 attention heads, 256 hidden layers) processes the haptic array → linear mapping, outputting... ; All embedded vectors are unified in dimension through a single MLP layer, ensuring that multimodal information can be fused and input into subsequent modules.

[0051] Step 2: Multimodal feature association with motion constraints refer to Figure 4 Embedding multimodal fusion after unifying dimensions The input is the improved large language model DeepSeek-Motion, which is based on DeepSeek-7B and retains its 12-layer TransformerDecoder structure and pre-normalization design. The newly added limb kinematics constraint module is embedded in the FeedForward layer: Core Improvement: A new limb kinematics constraint module has been added—based on DH parameters, kinematic models of humanoid robot joints (shoulder, elbow, and wrist joints) are built, and the angle limits of each joint are preset (e.g., shoulder pitch angle [-90°, 90°], elbow flexion angle [0°, 135°]). When generating motion commands, the model must meet these constraints. To prevent instructions from becoming unexecutable; Feature Association and Output: Multimodal features and task instructions are associated through a self-attention mechanism. After multi-layer understanding, two types of outputs are generated: ① Text action instructions (e.g., "Industrial: Move the robotic arm above the bolt, lower the end effector by 5cm, and apply a torque of 4N"). m””Home use: A robotic arm picks up a water cup from a table, with the elbow bent at 60° and the tactile force at 15N”); ②Multimodal feature vector (A structured vector that integrates visual, force, and tactile features, with a dimension of 2048).

[0052] Step 3: Motion Constraint-Guided Feature Mapping The feature mapping module consists of "MLP + Transformer with motion constraints", and its core is to... Convert to a conditional feature vector that conforms to the input of the diffusion model: MLP will (Dimension 2048) is transformed into a dimension that conforms to the Transformer input (Dimension 4096). The Transformer consists of 6 Encoder layers (responsible for feature encoding) and 6 Decoder layers (responsible for feature decoding). An additional joint angle constraint layer is added to the Decoder to verify in real-time whether the joint angles corresponding to the feature vectors conform to the constraint. If the range is exceeded, the characteristic value will be dynamically adjusted. The final output is a conditional feature vector (4096 dimensions), which contains environmental features, multimodal perception features, and limb movement constraint information, providing precise guidance for the diffusion model.

[0053] Step 4: Generation of scene-adaptive "environment-motion" fusion map refer to Figure 5 An improved scene-adaptive diffusion model is used to generate a fusion map that couples environmental state and limb movement. The core of this model is to achieve adaptation through scene weight factors. Forward noise addition: Gaussian noise is injected into the original multimodal data, and adjustments are made according to the scene type. Industrial scenarios (Force perception has the highest weight to ensure torque accuracy), for home use. (High visual / tactile weighting to ensure safety); the noise addition process follows a formula. ,in This is a scene-specific feature vector (including scene constraint thresholds). , For noise variance; It is Gaussian noise; Inverse denoising: Under guidance, noise was removed by iterating 10 times using the U-Net decoder (4 layers of Encoder + 4 layers of Decoder) to generate an "environmental state-limb movement" fusion map. The map simultaneously displays key environmental information (such as the position of the workpiece and torque value in an industrial scene, and the position of the water cup and the position of the elderly's hand in a household scene) and robot movement information (such as joint angles and end effector positions), intuitively reflecting the relationship between task progress and action coordination.

[0054] Formula for inverse denoising: ; in, This is a fusion diagram that couples environmental states with limb movements; These are conditional feature vectors; It follows a Gaussian distribution; The data with added noise for the current iteration number; This is the noisy data from the previous iteration; This is the mean value after noise reduction; The variance parameter after denoising; This is the ordinal number of the noise reduction step.

[0055] Step 5: Iterate and optimize until the task is completed. Use the text action instructions generated in step 2 as the new task instructions, and the fusion graph generated in step 4 as the new multimodal data input source, and repeat steps 1-4: Iteration termination conditions: ① The structural similarity index (SSIM) of the fusion graph generated in two consecutive iterations is ≥98% (the task state is stable and there are no obvious changes); ② The number of iterations is ≥8 (to avoid infinite loops and adapt to complex tasks). Final output: complete action sequence (such as "take bolt → align bolt hole → pre-tighten → tighten → torque detection" in industrial assembly), environmental state change record (such as workpiece position and torque value in each step), and fusion diagram sequence.

[0056] The factor definitions in the formulas of this application embodiment include: 1. Joint angles of a humanoid robot (unit: °), including key motion joints such as the shoulder joint, elbow joint, and wrist joint; 2. Limits of joint movement (e.g., shoulder joint pitch angle) , elbow flexion angle , ); 3. (Motion constraint query vector) needs to be fine-tuned based on DeepSeek's pre-trained semantic vectors to ensure alignment with DeepSeek's feature space. It is generated through pre-training on a dataset of 5000 sets of joint angle limits and is used for motion constraint verification during Transformer decoding. 4. Scene weighting factor / / The weights correspond to the visual, force, and tactile modalities, respectively: [0.3, 0.4, 0.3] for industrial scenarios and [0.4, 0.2, 0.4] for home scenarios. 5. 64-dimensional scene feature vector, industrial scene including bolt torque threshold (e.g., 5N) m), assembly gap threshold (e.g., 0.1mm), and for household use, include water cup grip strength threshold (e.g., 15N) and tilt threshold (e.g., 5°). 6. Noise variance of diffusion model in industrial scenarios (Precision priority, strict noise control), home use scenario (Safety first, high noise tolerance); 7.m: The total number of characters in the text action command, such as m=12 for "take water cup → hand it to the elderly person"; 8.d: Embedded spatial dimension, with a value of 512 (lightweight scenarios) or 1024 (complex scenarios). 9. The pressure difference of the flexible tactile array (unit: kPa) refers to the pressure difference between the sensors on both sides of the water cup. 10. : Cup tilt (unit: °), through Conversion .

[0057] The solutions of this application embodiment will be described in detail and explained below with reference to specific application examples: I. Reference Figure 6 This application provides an industrial scenario—precision assembly of motor end caps: 1.1 Task Instructions "Assemble the end cover for the Cobotix-120 motor, using M5 bolts, with a tightening torque of 5N." m, the gap between the rear end cover and the motor housing is ≤0.1mm, and the assembly time is ≤3 minutes.

[0058] 1.2 Equipment Parameters Humanoid robot: 14-DOF robotic arm (each arm has 2-DOF for the shoulder joint, elbow joint, and wrist joint), with a six-dimensional force sensor (range 0-500N) and an RGB-D camera (1920×1080) at the end. Tactile array: The mechanical finger is equipped with a 64×64 flexible tactile array (pressure range 0-100kPa, sampling frequency 500Hz).

[0059] 1.3 Steps to be executed 1.3.1 Multimodal Data Acquisition and Encoding: Visual images: An RGB-D camera captures images of the motor housing (coordinates X=100mm, Y=200mm, Z=300mm), end cap (coordinates X=300mm, Y=200mm, Z=300mm), and bolts (coordinates X=200mm, Y=100mm, Z=300mm), divides them into 32×32 image blocks (n=64), encodes them using ViT+Q-Former+MLP, and outputs the results. ; Force data: A six-dimensional force sensor collects the force baseline of the robotic arm when it is unloaded. During assembly, the Z-axis force value is collected in real time (initial). After being encoded by a 3-layer 1D-CNN (3 / 5 / 3 convolutional kernels, 32 / 64 / 128 kernels), the output is... ; Tactile data: The tactile array collects the pressure distribution during bolt contact (initial pressure P=10kPa), which is then encoded by a 4-layer Transformer encoder (8 attention heads, 256 hidden layers) and output. ; Text instruction encoding: output ; Unified Dimension: Through a 1-layer MLP (input dimension) (Output dimension 512) unifies multimodal embedding into .

[0060] 1.3.2 Feature Association with Motion Constraints: Kinematic constraints: shoulder joint pitch angle elbow flexion angle wrist joint rotation angle ; DeepSeek-Motion generates text-based motion commands: "① Move the robotic arm to the bolt storage position (shoulder joint pitch angle 30°, elbow joint flexion 90°), fingers spread to 8mm diameter, tactile pressure 15kPa to grasp the bolt; ② Move to above the screw hole on the motor housing (wrist joint rotation 30°), visually align with the screw hole (deviation ≤0.5mm); ③ Lower the end effector by 10mm, pre-tighten the bolt 2 turns (Z-axis force..." ); ④ Start torque control and tighten to 5N. m (torque = Z-axis force × end radius, end radius 50mm, therefore target) ); ⑤ Tactile detection end cap gap (pressure difference) Corresponding gap )”; generate .

[0061] 1.3.3 Feature Mapping: MLP will (2048 dimensions) are converted to 4096 dimensions to adapt to Transformer input; Transformer with motion constraints (6 layers of Encoder + 6 layers of Decoder) encoding-decoding, Decoder end joint angle constraint layer verification: In step ①, the elbow joint bending 90°≤135° meets the constraint; in step ④, the wrist joint rotation 30°≤180° meets the constraint. Output .

[0062] 1.3.4 Generation of fused graph: Scene parameters: , Including torque threshold of 5N m, gap threshold 0.1mm; Forward noise addition: ; Reverse denoising: The U-Net decoder iterates 10 times to generate a fused image—displaying the motor housing (coordinate X=100mm), bolts (coordinate X=200mm), robotic arm joint angles (shoulder joint 30°, elbow joint 90°, wrist joint 30°), and Z-axis force. (Pre-tightening stage).

[0063] 1.3.5 Iterative Optimization: First iteration: The fusion diagram shows that pre-tightening is complete. ); Second iteration: The fusion graph shows tightening in progress. (torque) ),gap ; Third iteration: The fusion diagram shows that tightening is complete. (torque) ),gap ; Fourth iteration: The fused graph and the SSIM of the third iteration are 99.2%, which meets the termination condition, and the iteration stops.

[0064] 1.4 Task Results Assembly success rate: 98.5%; average assembly time: 2.5 minutes (≤3 minutes); torque error: ±0.05N. m (≤±0.1N) m), gap error 0.08mm (≤0.1mm), joint overtravel jamming rate 0%.

[0065] II. Reference Figure 7 This application provides a home scenario—delivering water cups to the elderly: 2.1 Task Instructions Take a 300ml glass (water temperature 40℃) from the coffee table in the living room (coordinates X=500mm, Y=300mm, Z=700mm) and hand it to the elderly person (the elderly person is sitting, hand coordinates X=800mm, Y=400mm, Z=900mm), avoid tilting the glass to prevent spillage (tilt angle ≤5°), grip strength ≤20N.

[0066] 2.2 Equipment Parameters Bipedal humanoid robot: 14-DOF robotic arms (7-DOF per arm), with an RGB-D camera (1280×720) and a six-dimensional force sensor (range 0-100N) at the end. Tactile array: The mechanical finger is equipped with a 32×32 flexible tactile array (pressure range 0-50kPa).

[0067] 2.3 Steps to be executed 2.3.1 Multimodal Data Acquisition and Encoding: Visual image: Acquired image of a coffee table (wooden, area) Images of a water glass (transparent glass, 80mm in diameter) and an elderly person's hand (skin color, clenched fist) were divided into 16×16 image blocks (n=128), encoded using ViT+Q-Former+MLP, and output. ; Force data: baseline of unloaded force of robotic arm ( The target grip strength when picking up the cup is 15N. This is encoded using a 1D-CNN and the output is... ; Tactile data: The pressure distribution upon contact with the water cup (initial pressure P=8kPa) is collected, encoded using a Transformer, and output. ; Text instruction encoding: output ; Unified Dimensions: Unified using a single-layer MLP (input dimension 1024 + 128 × 16 × 16 × 3 + 1024 + 1024 = 1024 + 98304 + 1024 + 1024 = 101376, output dimension 1024) .

[0068] 2.3.2 Feature Association with Motion Constraints: Kinematic constraints: Horizontal rotation angle of the shoulder joint (Avoid bumping into the edge of the coffee table), elbow bend angle (Avoid overextending your arms); DeepSeek-Motion generates text-based motion commands: "① The robotic arm moves to the left side of the coffee table and water glass (shoulder joint rotates horizontally 30°, elbow joint bends 60°), fingers spread to a diameter of 90mm; ② Slowly approaches the water glass, tactile detection of contact (pressure P=5kPa), grip force adjusted to 15N ( ③ Raise the robotic arm to Z=900mm (elbow joint bent at 90°) to avoid colliding with the coffee table (distance from the edge of the coffee table ≥50mm); ④ Move it to the front of the elderly person's hand (shoulder joint rotated horizontally by 10°), visually confirm the hand position (deviation ≤10mm); ⑤ Slowly hand out the water cup, tactilely detect the tilt ( "After receiving the cup, the elderly man's grip strength dropped to 5N, and his fingers loosened." generate .

[0069] 2.3.3 Feature Mapping and Fusion Map Generation: Feature mapping: MLP dimensionality transformation → Transformer encoding-decoding (verifying elbow flexion 60°≤120°, meeting the constraint), output ; Scene parameters: , , Includes a grip strength threshold of 20N and a tilt threshold of 5°; Image generation: Displays coffee table (X=500mm), water glass (X=500mm), elderly person's hand (X=800mm), robotic arm joint angles (shoulder joint 30°, elbow joint 60°), grip strength 15N, tilt angle 3°. ).

[0070] 2.3.4 Iterative Optimization: First iteration: The fusion graph shows that the cup was picked up, with a grip strength of 15N and an inclination of 3°; Second iteration: The fusion graph shows the cup raised to Z=900mm, with the elbow bent at 90°; Third iteration: The fusion graph shows the water cup being handed to the elderly person at a 2° tilt, and the elderly person's force in receiving the cup. ; Fourth iteration: The fused graph and the SSIM of the third iteration are 98.5%, so the iteration stops.

[0071] 2.4 Task Results Delivery success rate 99%, water cup spill rate 0%, elderly contact comfort rating 4.8 / 5 (100 tests), probability of colliding with furniture 0%, grip strength control error ±1N (≤±2N).

[0072] The innovative aspects of this application are: 1. Multimodal-motion-environment coupled world model: For the first time, a coupled model of "visual-force-tactile perception + limb movement constraints + environmental state" is constructed, which solves the shortcomings of existing world models that only simulate the environment and improves the foresight of decision-making; 2. Scene-adaptive multimodal weighting mechanism: through Dynamically adjust the visual / force / tactile weights to adapt to the differentiated needs of industrial (force-first) and home (visual / tactile-first) scenarios, improving cross-scenario adaptability by more than 50%. 3. Deep integration of kinematic constraints and LLM: A kinematic constraint module based on DH parameters is embedded in DeepSeek-7B to ensure that the generated motion commands are executable, and the joint overtravel jamming rate is reduced from 30% to below 5%. 4. Reproducible multimodal coding scheme: The specific parameters of 1D-CNN and Transformer (convolutional kernel, number of layers, number of attention heads) are disclosed, as well as the conversion formulas for haptic-tilt and force-torque, to ensure that those skilled in the art can reproduce the scheme.

[0073] This application embodiment also provides a humanoid robot scene adaptive task planning device based on a multimodal-motion coupled world model, which can implement the above method. The device includes: The first module is used to receive task instructions and collect multimodal data, then encode it uniformly to obtain multimodal fusion and embedding data; The second module is used to perform feature association on multimodal fusion and embedding data through an improved multimodal-motion coupled world model to obtain text action instructions and multimodal feature vectors; The third module is used to perform motion constraint-guided feature mapping based on the multimodal feature vectors to obtain conditional feature vectors; The fourth module is used to generate a fusion map based on the conditional feature vector and the improved scene adaptive diffusion model, so as to obtain a fusion map that couples the environmental state and limb movement. The fifth module is used to take text action instructions as new task instructions, take the fusion graph as new multimodal data, return the task instructions received and multimodal data collected and uniformly encoded to obtain multimodal fusion embedded data, until the output fusion graph meets the preset conditions.

[0074] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0075] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0076] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0077] Another embodiment of the hardware structure of the electronic device, the electronic device including: The processor can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solutions provided in the embodiments of this application. The memory can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and called by the processor to execute the methods described in the embodiments of this application. Input / output interfaces are used to implement information input and output; The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). A bus is used to transfer information between various components of a device, such as processors, memory, input / output interfaces, and communication interfaces. The processor, memory, input / output interfaces, and communication interfaces communicate with each other within the device via a bus.

[0078] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0079] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0080] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0081] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0082] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0083] The humanoid robot scene adaptive task planning method, device, electronic device, storage medium, and program product based on a multimodal-motion coupled world model provided in this application aim to solve the problems of humanoid robots relying on single perception, lacking limb movement coordination, poor scene adaptability, and the disconnect between the world model and movement in related technologies. This method uses an RGB-D camera, a six-dimensional force sensor, and a flexible tactile array mounted on the humanoid robot to collect multimodal data such as visual images, end-effector force sequences, and finger tactile arrays; it combines an improved large language model (DeepSeek-Motion, based on a modified DeepSeek basic model) with integrated limb kinematic constraint modules and a scene adaptive diffusion model to construct a dedicated world model (multimodal-motion coupled world model) that couples "multimodal perception - limb movement - environmental state". First, the multimodal fusion is embedded into the improved large language model DeepSeek-Motion (the model is based on the DeepSeek basic model, integrates a limb kinematic constraint module based on DH parameters, and adapts to the Transformer pre-normalized layer structure of DeepSeek) to generate collaborative action instructions and multimodal feature vectors. Then, the feature mapping module converts them into conditional feature vectors, driving the improved diffusion model to generate an "environmental state-limb movement" fusion map. Finally, the input data is iteratively updated until all action sequences for completing the task are generated, which can significantly improve the action coordination, task execution efficiency and success rate in complex scenarios.

[0084] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0085] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0088] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0089] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0091] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A humanoid robot scene adaptive task planning method based on a multimodal-motion coupled world model, characterized in that, The method includes the following steps: After receiving task instructions and collecting multimodal data, the data is uniformly encoded to obtain multimodal fusion and embedding data; By performing feature association on the multimodal fusion and embedding data using an improved multimodal-motion coupled world model, text action commands and multimodal feature vectors are obtained, including: An improved multimodal-motion coupled world model is constructed based on a large language model; the multimodal-motion coupled world model includes a TransformerDecoder structure with a preset number of layers and a limb kinematics constraint module; the limb kinematics constraint module is used to set the robot joint angle corresponding to the text action command within a preset limit range when generating text action commands; The multimodal fusion and embedding data is input into the multimodal-motion coupled world model, and the multimodal features are associated with the task instructions through a self-attention mechanism to generate text action instructions and multimodal feature vectors; the multimodal feature vectors are structured vectors that fuse visual features, force features and tactile features; Based on the multimodal feature vectors, motion constraint-guided feature mapping is performed to obtain conditional feature vectors; Based on the conditional feature vector and the improved scene adaptive diffusion model, a fusion map is generated to obtain a fusion map that couples the environmental state and limb movement, including: An improved scene-adaptive diffusion model is constructed based on the diffusion model; the scene-adaptive diffusion model includes scene weight factors and a U-Net decoder; the scene weight factors include visual modality weights, force modality weights and tactile modality weights; The conditional feature vector is input into the scene adaptive diffusion model, and the scene weight factor is used to add noise forward to obtain noisy data. Based on the conditional feature vector and the noisy data, the U-Net decoder is used to perform reverse denoising to generate a fusion map of coupled environmental state and limb movement. The text action instruction is used as a new task instruction, and the fusion graph is used as new multimodal data. The process of receiving the task instruction, collecting multimodal data, and then uniformly encoding it is repeated until the output fusion graph meets the preset conditions. The preset conditions include that the structural similarity index of two consecutively generated fusion graphs is greater than or equal to a first threshold or the number of iterations is greater than or equal to a second threshold. When applied to industrial assembly scenarios, the torque is calculated by combining the Z-axis force value collected by the six-dimensional force sensor with the radius of the robotic arm's end effector, and the tightening action is adjusted accordingly; the torque = force value × radius. When applied to home delivery scenarios, the tilt angle of a water cup is calculated by the pressure difference of a flexible tactile array.

2. The method according to claim 1, characterized in that, The process of receiving task instructions and collecting multimodal data, followed by unified encoding, yields multimodal fusion and embedding data, including: Receive task instructions, map the task instructions to the embedding space through text encoding, and obtain task instruction embedding data; Multimodal data of the current environment is acquired through acquisition devices mounted on a humanoid robot; the acquisition devices include an RGB-D camera, a six-dimensional force sensor of an end effector, and a flexible tactile array of mechanical fingers; the multimodal data includes environmental visual images, force sequence data, and tactile array data; The environmental visual image is mapped to the embedding space through visual encoding to obtain visual feature embedding data; The force sequence data is mapped to the embedding space through force coding to obtain force feature embedding data; The tactile array data is encoded and projected into the embedding space to obtain tactile feature embedding data; The task instruction embedding data, the visual feature embedding data, the force feature embedding data, and the tactile feature embedding data are used as multimodal fusion embedding data.

3. The method according to claim 1, characterized in that, The step of performing motion constraint-guided feature mapping based on the multimodal feature vectors to obtain conditional feature vectors includes: The dimensions of the multimodal feature vectors are converted to a preset dimension size, and the joint angles corresponding to the multimodal feature vectors are verified to meet the preset conditions to obtain verification information. Based on the verification information, if the joint angle exceeds the range, the feature value is adjusted; if the verification passes, a conditional feature vector is output. The conditional feature vector includes environmental features, multimodal perception features, and limb movement constraint information.

4. The method according to claim 1, characterized in that, The step of inputting the conditional feature vector into the scene adaptive diffusion model and using the scene weight factor for forward noise addition to obtain the noisy data includes the following formulas: ； in, The data with added noise for the current iteration number; This is the noisy data from the previous iteration; , For noise variance; It is Gaussian noise; As a scene weighting factor; This is the scene feature vector.

5. The method according to claim 1, characterized in that, The step involves using the U-Net decoder to perform inverse denoising based on the conditional feature vector and the noisy data to generate a fusion map of the coupled environmental state and limb movement. The formulas used include: ； in, This is a fusion diagram that couples environmental states with limb movements; The conditional feature vector; It follows a Gaussian distribution; The data with added noise for the current iteration number; This is the noisy data from the previous iteration; This is the mean value after noise reduction; The variance parameter after denoising; This is the ordinal number of the noise reduction step.

6. A humanoid robot scene adaptive task planning device based on a multimodal-motion coupled world model, used to implement the method as described in any one of claims 1 to 5, characterized in that, The device includes: The first module is used to receive task instructions and collect multimodal data, then encode it uniformly to obtain multimodal fusion and embedding data; The second module is used to perform feature association on the multimodal fusion and embedding data through an improved multimodal-motion coupled world model to obtain text action instructions and multimodal feature vectors; The third module is used to perform motion constraint-guided feature mapping based on the multimodal feature vectors to obtain conditional feature vectors; The fourth module is used to generate a fusion map based on the conditional feature vector and the improved scene adaptive diffusion model, so as to obtain a fusion map of coupled environmental state and limb movement. The fifth module is used to take the text action instruction as a new task instruction, take the fusion graph as new multimodal data, return to the step of receiving the task instruction and collecting multimodal data and then uniformly encoding it to obtain multimodal fusion embedded data, until the output fusion graph meets the preset conditions.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images