Robot control method based on multi-modal large model

Abstract

The application relates to a robot control method based on a multimodal large model and relates to the technical field of robots. The method comprises the following steps: acquiring multimodal input data and extracting context features; analyzing semantic features and extracting a core instruction intention; extracting constraint conditions from user instruction text to obtain key constraint conditions; recursively decomposing a complex instruction into an atomic task sequence according to the core instruction intention and the key constraint conditions to generate a time-space continuous action planning candidate set; outputting cross-modal correlation features in a joint embedding space; calculating the context similarity of the cross-modal correlation features and historical interaction data features; screening an optimal action planning candidate with the highest similarity from the action planning candidate set based on the context similarity; and controlling the robot to move based on the optimal action planning candidate. The application can improve the decision-making capability of the robot in a complex scene.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a robot control method based on a multimodal large model. Background Technology

[0002] With the development of artificial intelligence and robotics, robot interaction systems are gradually evolving from single-modal instruction execution to multimodal intelligent interaction. In traditional robot interaction systems, visual, linguistic, and action modal features are usually processed independently without constructing a unified joint embedding space for cross-modal semantic alignment. This makes it difficult to achieve deep coupling between instruction intent, environmental state, and action execution, thus limiting the robot's intelligent decision-making capabilities in complex scenarios. Summary of the Invention

[0003] Therefore, it is necessary to provide a robot control method based on a multimodal large model that can improve the robot's decision-making ability in complex scenarios, addressing the aforementioned technical problems.

[0004] This application provides a robot control method based on a multimodal large model. The method includes: acquiring multimodal input data; wherein the multimodal input data includes: user command text, environmental visual images, and robot body state data; preprocessing the multimodal input data to extract contextual features; wherein the contextual features include: semantic features, visual spatial features, and robot kinematic features; parsing the semantic features based on a large language model to extract the core intent of the command; extracting constraints from the user command text to obtain key constraints; and recursively decomposing complex commands into atomic task sequences based on the core intent of the command and the key constraints, combined with a preset task knowledge base. The process involves: generating a spatiotemporally continuous action planning candidate set based on the atomic task sequence, the visual spatial features, and the robot kinematic features; outputting cross-modal association features in a joint embedding space based on the user command text, the environmental visual image, and the action planning candidate set; wherein the joint embedding space is a pre-constructed embedding space for vision, language, and action; calculating the contextual similarity between the cross-modal association features and historical interaction data features; selecting the optimal action planning candidate with the highest similarity from the action planning candidate set based on the contextual similarity; and controlling the robot's movement based on the optimal action planning candidate.

[0005] In one embodiment, the step of acquiring multimodal input data includes: acquiring the user command text in real time through a microphone array or text input interface; acquiring the environmental visual image through an environmentally deployed camera or a vision sensor mounted on the robot; acquiring the robot body state data through an inertial measurement unit, joint encoder, and force sensor integrated into the robot body; and performing timestamp alignment and noise filtering on the multimodal input data.

[0006] In one embodiment, the step of preprocessing the multimodal input data and extracting contextual features includes: performing word segmentation, part-of-speech tagging, and named entity recognition on the user command text, and generating the semantic features by combining a pre-trained language model; performing object detection, semantic segmentation, and depth estimation on the environmental visual image, and extracting the visual spatial features containing object category, spatial location, and geometric dimensions; performing inverse kinematics calculation on the joint angles in the robot body state data, and obtaining the robot kinematic features by combining the extracted dynamic features; wherein, the dynamic features include: joint angular velocity, joint acceleration, and motor torque; and aligning the semantic features, the visual spatial features, and the robot kinematic features in a spatiotemporal dimension to form the multimodal fused contextual features.

[0007] In one embodiment, the step of parsing the semantic features based on a large language model to extract the core intent of the instruction includes: performing context modeling on the semantic features using a multi-layer Transformer encoder of the large language model, and classifying the intent using an intent classifier to obtain the core intent of the instruction; wherein, the core intent of the instruction includes: task execution, path planning, and status query.

[0008] In one embodiment, the step of extracting constraints from the user instruction text to obtain key constraints includes: extracting constraint keywords from the user instruction text based on rule matching and sequence labeling; logically completing the fuzzy constraints in the constraint keywords based on a preset constraint rule base and common sense reasoning to obtain the key constraints; wherein the key constraints include: time limit, spatial restricted area, and operation precision.

[0009] In one embodiment, the step of recursively decomposing a complex instruction into a sequence of atomic tasks based on the core intent of the instruction and the key constraints, combined with a preset task knowledge base, includes: matching a corresponding complex task template from the preset task knowledge base according to the core intent of the instruction; decomposing the complex task template layer by layer into indivisible atomic tasks based on a recursive decomposition algorithm; performing dependency checks on the execution order, resource consumption, and constraint compatibility of the atomic tasks to generate an atomic task execution tree containing task priority and timing constraints; and determining the atomic task sequence based on the leaf nodes of the atomic task execution tree.

[0010] In one embodiment, the step of generating a spatiotemporally continuous motion planning candidate set based on the atomic task sequence, the visual spatial features, and the robot kinematic features includes: for each atomic task in the atomic task sequence, combining the visual spatial features and the robot kinematic features, generating multiple candidate trajectories that satisfy dynamic constraints through a motion planning algorithm; spatiotemporally splicing the candidate trajectories of adjacent atomic tasks to form the motion planning candidate set covering the entire task flow; wherein each motion planning candidate in the motion planning candidate set includes: joint space trajectory, Cartesian space path, and execution time parameters, and the candidate trajectories of the same atomic task have different obstacle avoidance strategies or energy consumption optimization objectives.

[0011] In one embodiment, the step of outputting cross-modal association features in the joint embedding space based on the user instruction text, the environmental visual image, and the action planning candidate set includes: semantically encoding the user instruction text to obtain a language embedding vector; extracting features from the environmental visual image to obtain a visual embedding vector; encoding motion features from the action planning candidate set to obtain an action embedding vector; and semantically aligning the language embedding vector, the visual embedding vector, and the action embedding vector in the joint embedding space using a cross-modal contrastive learning algorithm to obtain the cross-modal association features.

[0012] In one embodiment, the step of calculating the context similarity between the cross-modal association feature and the historical interaction data feature includes: extracting historical successful interaction data of the same type of task from the historical interaction database and using it as the historical interaction data feature; calculating a weighted fusion value of the semantic matching degree, environmental feature matching degree, and motion trajectory smoothness between the cross-modal association feature and the historical interaction data feature based on preset weights, and using it as the context similarity; wherein the preset weights are dynamically adjusted according to the key constraints.

[0013] In one embodiment, after the step of controlling the robot's movement based on the optimal motion planning candidate, the method further includes: collecting execution feedback data in real time; and dynamically updating the joint embedding space based on the execution feedback data to improve the generalization execution capability of complex instructions.

[0014] The robot control method based on the multimodal large model described above, after determining the candidate set of action planning, outputs cross-modal association features in the joint embedding space based on user command text, environmental visual images and robot body state data to achieve semantic alignment of visual, linguistic and action features. Finally, by combining the similarity with historical interaction data, the optimal action planning candidate selected simultaneously satisfies semantic intent, environmental constraints and historical success experience, thereby improving the reliability of action execution and the robot's decision-making ability in complex scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a robot control method in one embodiment;

[0016] Figure 2 This is a schematic diagram of the process for obtaining multimodal input data in one embodiment;

[0017] Figure 3 This is a flowchart illustrating the process of determining context features in one embodiment;

[0018] Figure 4 This is a flowchart illustrating the process of determining key constraints in one embodiment.

[0019] Figure 5 This is a flowchart illustrating the process of determining an atomic task sequence in one embodiment;

[0020] Figure 6 This is a flowchart illustrating the process of determining a candidate set of action planning in one embodiment;

[0021] Figure 7 This is a flowchart illustrating the process of determining cross-modal association features in one embodiment;

[0022] Figure 8 This is a flowchart illustrating the process of determining context similarity in one embodiment;

[0023] Figure 9 This is a flowchart illustrating a robot control method in another embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] With the development of artificial intelligence and robotics, robot interaction systems are gradually evolving from single-modal instruction execution to multimodal intelligent interaction. However, traditional robot interaction systems generally suffer from the following technical bottlenecks:

[0026] Low efficiency of multimodal data fusion: Existing systems collect and process multimodal data such as user command text, environmental visual images, and robot body state in a relatively fragmented manner, lacking efficient spatiotemporal alignment and feature fusion mechanisms, resulting in incomplete understanding of contextual semantics;

[0027] Insufficient ability to decompose complex instructions: For complex tasks that require multi-step collaboration, traditional systems rely on preset rules or simple task splitting, which makes it difficult to combine dynamic environmental constraints and deep user intent to perform hierarchical recursive decomposition, resulting in poor task execution flexibility.

[0028] Action planning is disconnected from semantic intent: The action planning process is usually based solely on environmental perception data, lacking correlation analysis of user command semantics and historical interaction experience. This may lead to planning results that violate potential user constraints (such as operation precision, time limits, etc.) and insufficient interaction robustness.

[0029] Cross-modal associative reasoning is lacking: Visual, linguistic, and action modal features are usually processed independently, and a unified joint embedding space is not constructed for cross-modal semantic alignment. This makes it difficult to achieve deep coupling between instruction intent, environmental state, and action execution, thus limiting intelligent decision-making capabilities in complex scenarios.

[0030] To address the aforementioned issues, existing technologies attempt to improve performance through single-modal optimization (such as enhancing visual recognition algorithms or improving language models), but have not developed a systematic solution for multimodal deep fusion, hierarchical reasoning, and cross-modal correlation. To solve these technical problems, embodiments of this application provide a robot control method based on a multimodal large model.

[0031] In one embodiment, such as Figure 1 As shown, a robot control method based on a multimodal large model is provided. Taking the application of this method in a robot as an example, the method includes the following steps:

[0032] Step S110: Obtain multimodal input data.

[0033] Specifically, the robot first acquires multimodal input data, including user command text, environmental visual images, and robot body state data. User command text can be input via microphone array, keyboard, or touchscreen; it consists of real-time natural language commands (e.g., "move the red cup from the table to the second shelf of the cabinet"), supporting multilingual and conversational expressions. Environmental visual images can be images captured by devices such as RGB-D cameras and binocular cameras, providing visual information such as object color, shape, spatial position, and obstacle distribution. Robot body state data can be kinematic and dynamic state data such as robot joint angles, end effector position, velocity, and battery level, collected by encoders, inertial measurement units (IMUs), etc.

[0034] Step S120: Preprocess the multimodal input data and extract contextual features.

[0035] Specifically, after acquiring multimodal input data, the robot first preprocesses the data to obtain contextual features, including semantic features, visual spatial features, and robot kinematic features. This can be understood as follows: semantic features are obtained by preprocessing the user command text; visual spatial features are obtained by preprocessing the environmental visual image; and robot kinematic features are obtained by preprocessing the robot's body state data. Different data processing methods can be used when preprocessing different types of data.

[0036] Step S130: Based on the large language model, the semantic features are parsed to extract the core intent of the instruction.

[0037] Specifically, the robot will also analyze semantic features based on a large language model to extract the core intent of the command. For example, the robot analyzes semantic features through a built-in, finely tuned large language model (such as GPT-4o) to identify the core of the user's command text and use it as the core intent of the command (such as "moving", "assembling", "inspecting" etc.).

[0038] Step S140: Extract constraints from the user instruction text to obtain key constraints.

[0039] Specifically, the robot will also extract constraints from the user command text in the multimodal input data to obtain the key constraints in the user command text. For example, key constraints include: accuracy ±0.5mm, time ≤30s, obstacle avoidance priority, etc.

[0040] Step S150: Based on the core intent and key constraints of the instruction, and combined with the preset task knowledge base, the complex instruction is recursively decomposed into a sequence of atomic tasks.

[0041] Specifically, the pre-defined task knowledge base includes: atomic task definitions, execution logic, and constraint rules. For example, the "grab" atomic task needs to satisfy "object grasping point recognition" and "gripper opening and closing degree matching." When recursively decomposing complex instructions into atomic task sequences, a hierarchical task network (HTN) algorithm can be used. For example, the "assemble parts" task can be decomposed into an atomic task sequence such as "locate part A" → "grab part A" → "move to assembly point" → "align part B" → "release," with key constraints (such as current obstacle position, real-time part posture, etc.) and core instruction intent (such as "prioritize assembly accuracy") dynamically associated at each decomposition level. In some other embodiments, during task decomposition, historical interaction data (such as the user previously emphasizing "handle fragile items with care") can also be associated through an attention mechanism to correct the decomposition rules and achieve personalized intent understanding.

[0042] Step S160: Based on the atomic task sequence, visual spatial features and robot kinematic features, generate a spatiotemporally continuous action planning candidate set;

[0043] Specifically, for each atomic task sequence (such as "grasping"), multiple candidate trajectories need to be generated by combining visual spatial features (target object position, posture, etc.) and robot kinematic features (current pose, joint limits, etc.) using motion planning algorithms (such as Fast Exploratory Random Tree (RRT*, A* algorithm). The candidate trajectory includes the spatiotemporal coordinate sequence of the end effector (time resolution 10ms) and joint motion curves, simultaneously satisfying dynamic constraints (such as acceleration limits and joint velocity limits). Understandably, user constraints (such as the trajectory endpoint position error threshold corresponding to "operational accuracy ±0.5mm") and environmental dynamic constraints (such as real-time updated obstacle positions) can also be embedded during trajectory planning. Different constraint priorities are quantified through weight parameters (such as obstacle avoidance weight 0.6, accuracy weight 0.3, and time weight 0.1), thereby generating a diverse candidate set (≥5 candidate trajectories per atomic task). Once all atomic task sequences are planned, a spatiotemporally continuous motion planning candidate set is obtained.

[0044] Step S170: In the joint embedding space, output cross-modal association features based on user instruction text, environmental visual images, and action planning candidate sets.

[0045] Specifically, the joint embedding space is a pre-constructed embedding space for vision, language, and action. For example, a multimodal Transformer encoder can map user command text, environmental visual images, and action planning candidate sets to a unified high-dimensional space (e.g., 1024-dimensional). Specifically, the environmental visual image uses ResNet50 to extract visual space features, projected onto a linear layer to form a 1024-dimensional visual embedding vector; the user command text can directly use the output of the last layer of a large language model as a language embedding vector (1024-dimensional), which preserves command semantics and contextual information; the action planning candidate set generates an action embedding vector (1024-dimensional) by sampling trajectory keypoints (1 point every 50ms) and encoding temporal features using LSTM, containing joint motion patterns and end-effector trajectory features; the three interact through a multi-head self-attention mechanism to generate a joint embedding vector containing cross-modal semantics, i.e., cross-modal association features. In some embodiments, when constructing the joint embedding space: the three types of embedding vectors are fused through a cross-modal attention mechanism (e.g., Transformer multi-head attention), and the interaction matrix is ​​calculated using the following formula:

[0046]

[0047] Here, Q is the language embedding vector, K is the visual embedding vector, and V is the action embedding vector, thus achieving semantic alignment between modalities. The joint embedding space allows language intent (such as "handle gently") to be directly associated with action features (such as end-effector velocity ≤100mm / s) and visual features (such as fragile item recognition), thereby solving the problem of "semantic-action disconnect" in traditional methods. The probability of meeting the user's potential constraints during task execution increases from 60% to 85%.

[0048] Step S180: Calculate the contextual similarity between cross-modal association features and historical interaction data features.

[0049] Specifically, historical interaction data features are stored features of successfully executed tasks, including user feedback labels. When calculating context similarity, the cosine similarity between cross-modal association features and historical interaction data features is calculated and used as the context similarity.

[0050] Step S190: Select the optimal action planning candidate with the highest similarity from the action planning candidate set based on context similarity.

[0051] Specifically, based on the calculated context similarity, the action trajectory with the highest similarity is selected from the action planning candidate set and used as the optimal action planning candidate. In some embodiments, the optimal action planning candidate can also be determined by a similarity evaluation model. This model optimizes the similarity evaluation model by setting a reinforcement learning reward function (such as task completion success rate, user satisfaction feedback), dynamically adjusts the weights of each modality (such as language constraint weights automatically increasing to 0.5 in complex instructions), and comprehensively considers similarity score, constraint satisfaction (such as accuracy achievement rate), and energy efficiency (such as total joint movement distance). The optimal action planning candidate is determined by a multi-objective optimization algorithm (such as NSGA-II) to ensure that the action planning conforms to the user's semantic intent and adapts to the dynamic environment.

[0052] Step S210: Control the robot's movement based on the optimal motion planning candidate.

[0053] Specifically, after obtaining the optimal motion planning candidate, the robot is controlled to execute the action according to the optimal motion planning candidate to move. It is understandable that while controlling the robot's movement, it is also necessary to simultaneously monitor the robot's own state and environmental changes in real time (such as object sliding or new obstacles entering), and adjust the force of the end effector through force sensor feedback (such as dynamically reducing the gripper pressure when grasping fragile items). After the task is completed, the interaction data (instructions, environment, actions, and results) is stored in a historical database for subsequent similarity calculations and model iteration optimization.

[0054] The robot control method based on the multimodal large model described above, after determining the candidate set of action planning, outputs cross-modal association features in the joint embedding space based on user command text, environmental visual images and robot body state data to achieve semantic alignment of visual, linguistic and action features. Finally, by combining the similarity with historical interaction data, the optimal action planning candidate selected simultaneously satisfies semantic intent, environmental constraints and historical success experience, thereby improving the reliability of action execution and the robot's decision-making ability in complex scenarios.

[0055] In one embodiment, such as Figure 2 As shown, step S110, the step of acquiring multimodal input data, includes:

[0056] Step S111: Collect user command text in real time via microphone array or text input interface.

[0057] Specifically, the robot can acquire voice commands in real time through a set microphone array (such as an 8-channel MEMS microphone) and convert them into user-defined text commands by combining them with Automatic Speech Recognition (ASR) technology (such as the VGGish model). Alternatively, users can manually input corresponding user-defined text commands through a deployed text input interface (touchscreen / keyboard). In some embodiments, the text input interface employs an anti-shake algorithm (such as setting a 50ms input delay) to avoid accidental triggering.

[0058] Step S112: Obtain environmental visual images through cameras deployed in the environment or visual sensors mounted on the robot.

[0059] Specifically, the robot can acquire environmental visual images through cameras fixed in the environment (such as binocular cameras) or visual sensors mounted on the robot (such as RGB-D cameras). When acquiring environmental visual images, images are captured at a frame rate of 30 FPS, and hardware synchronization triggering (such as GPIO signals) is used to ensure time alignment of multiple cameras. The cameras also support automatic white balance and exposure compensation under dynamic environments (such as changes in lighting).

[0060] Step S113: Collect robot body state data through the inertial measurement unit, joint encoder, and force sensor integrated into the robot body.

[0061] Specifically, the robot collects its own state data through an integrated inertial measurement unit (IMU), joint encoders, and force sensors. For example, the joint encoders (resolution ≤ 0.1°) acquire joint angles in real time; the IMU collects acceleration and angular velocity at a frequency of 100Hz; and force sensors (such as six-dimensional force sensors) are installed on the end effector to collect contact force data at a frequency of 50Hz. In this case, the robot's state data includes joint angles, acceleration, angular velocity, and contact force data. The IMU, joint encoders, and force sensors achieve a timestamp accuracy of ≤ 10μs through a synchronization clock module (supporting the PTP protocol).

[0062] Step S114: Perform timestamp alignment and noise filtering on the multimodal input data.

[0063] Specifically, in this embodiment, user command text, environmental visual images, and robot body state data undergo timestamp alignment and noise filtering. For example, a unified timestamp (based on UTC clock) is added to each type of data. A sliding window algorithm (window size 200ms) is used to match data groups with timestamp errors ≤5ms, and abnormal timestamp data is removed. Kalman filtering (for IMU data) and median filtering (for visual noise) are used for noise reduction. For example, a state space model is established for joint angle data, and high-frequency noise is filtered out through a prediction-update step to improve data reliability. This embodiment, through hardware-level synchronization and software sliding window, can solve the semantic understanding deviation problem caused by multimodal data asynchrony in traditional systems. The timestamp alignment accuracy is improved to the industrial standard (≤10μs), laying the foundation for subsequent spatiotemporal fusion. Furthermore, dedicated filtering algorithms can be designed for different sensor characteristics, reducing body state data noise by 40% (e.g., reducing joint angle measurement error from ±0.5° to ±0.3°), and improving visual image target detection accuracy by 25% in low-light scenarios to ensure front-end data quality.

[0064] In one embodiment, such as Figure 3 As shown, step S120, which involves preprocessing the multimodal input data and extracting contextual features, includes:

[0065] Step S121: Perform word segmentation, part-of-speech tagging, and named entity recognition on the user command text, and generate semantic features by combining the pre-trained language model.

[0066] Specifically, semantic features are obtained after preprocessing the user command text. First, the user command text is segmented (e.g., using spaCy for segmentation), then part-of-speech tagging is performed (e.g., identifying verbs like "carry" and nouns like "red cup"), and key entities are located using Named Entity Recognition (NER) (e.g., the location entity in "second shelf of the cabinet"). Finally, the processed text is input into a pre-trained language model (e.g., a fine-tuned BERT model) to obtain semantic features (768 dimensions).

[0067] Step S122: Perform object detection, semantic segmentation and depth estimation on the environmental visual image, and extract visual spatial features including object category, spatial location and geometric size.

[0068] Specifically, the visual spatial features are obtained after preprocessing the environmental visual images. First, object detection is performed on the environmental visual images (e.g., using YOLOv8 to detect object categories and bounding boxes). Then, semantic segmentation is performed (e.g., combining Mask R-CNN to perform semantic segmentation to obtain object masks). After semantic segmentation, the output will contain the object category (e.g., "cup" or "obstacle") and two-dimensional coordinates (pixel level). Then, depth estimation is performed on the segmentation results (e.g., using the Monodepth2 model to estimate the monocular depth map) to convert the pixel coordinates into three-dimensional coordinates (X, Y, Z) in the robot's base coordinate system. The object size (length × width × height) and relative distance are calculated to generate visual spatial features (dimension: N × 6, where N is the number of objects, and the 6 dimensions are X, Y, Z, length, width, and height).

[0069] Step S123: Perform inverse kinematics calculation on the joint angles in the robot body state data, and combine the extracted dynamic features to obtain the robot's kinematic features.

[0070] Specifically, the Denavit-Hartenberg (DH) parametric model can be used to input the joint angles (θ1-θ6) from the robot's body state data into an inverse kinematics algorithm (such as analytical method + numerical iteration) for inverse kinematics calculation, thereby obtaining the end effector pose (homogeneous transformation matrix T6×6). Simultaneously, dynamic features such as joint angular velocity (° / s), joint acceleration (° / s²), and motor torque (N·m) are extracted and merged with the end effector pose to obtain the robot's kinematic features (dimension: 1×20, including joint states and end effector states).

[0071] Step S124: Align the semantic features, visual spatial features, and robot kinematic features in a spatiotemporal dimension to form a multimodal fusion contextual feature.

[0072] Specifically, when aligning semantic features, visual spatial features, and robot kinematic features in time, the robot control cycle (e.g., 10ms) is used as a benchmark, and features from different modalities are synchronized to the same timestamp through linear interpolation. When aligning semantic features, visual spatial features, and robot kinematic features in space, the visual spatial coordinates are uniformly converted to the robot's base coordinate system to ensure spatial consistency with the kinematic features. Finally, the semantic features, visual spatial features, and kinematic features are concatenated into a fusion vector (dimension: 768 + N × 6 + 20) to obtain the final contextual features.

[0073] This embodiment extracts multimodal features using NLP, CV, and robotics-specific algorithms, avoiding feature confusion issues caused by mixed processing in traditional systems. Its semantic entity recognition accuracy reaches 95% (a 30% improvement compared to traditional rule engines), with a visual spatial positioning error ≤2cm, providing precise input for subsequent intent parsing. The spatiotemporally aligned fused feature vectors support cross-modal processing in subsequent modules. For example, the large language model can directly associate the position of visual objects with the target object in the command, solving the problem of the disconnect between the command object and the actual object in traditional systems.

[0074] In one embodiment, step S130, which involves parsing semantic features based on a large language model to extract the core intent of an instruction, includes: performing contextual modeling of semantic features using a multi-layer Transformer encoder of the large language model, and classifying the intent using an intent classifier to obtain the core intent of the instruction.

[0075] Specifically, in this embodiment, semantic features are input into a large language model (such as GPT-4o, freezing the underlying parameters and fine-tuning the upper interaction layer), and context modeling is performed through the self-attention mechanism of a multi-layer Transformer encoder (such as the spatial relation dependency of "left side" in "place A to the left of B"). An intent classifier (such as a fully connected layer) is added to the model output layer, and Softmax is used for classification to obtain the core intents of instructions, including task execution, path planning, and status query. Through optimization using the cross-entropy loss function, the intent classification accuracy on the dataset reaches 98%.

[0076] In one embodiment, such as Figure 4 As shown, step S140, which involves extracting constraints from the user instruction text to obtain key constraints, includes:

[0077] Step S141: Extract constraint keywords from the user instruction text based on rule matching and sequence labeling.

[0078] Specifically, when extracting constraints, the constraint keywords are first located from the user command text based on rule matching (such as regular expression matching "within XX seconds" or "precision ±XX") and sequence labeling (such as BIOES labeling to mark entities such as "time limit" or "space restricted area").

[0079] Step S142: Based on the preset constraint rule base and common sense reasoning, logically complete the fuzzy constraints in the constraint keywords to obtain the key constraint conditions.

[0080] Specifically, by associating with a pre-defined constraint rule base (e.g., "handle gently" corresponds to a stress sensor threshold ≤5N, "high speed" corresponds to an end-effector speed ≥200mm / s), the system logically completes the fuzzy constraints (e.g., "be careful") in the constraint keywords, thereby generating a structured constraint list (e.g., {time limit: 30s, accuracy: ±0.5mm, restricted area: [X<0.5m]}), which is then used as key constraints. These key constraints include time limits, spatial restricted areas, and operational accuracy. In some other embodiments, synonym conversion rules from the task knowledge base (e.g., "move" = "transport" = "move") can be used to eliminate instruction ambiguity, and common sense reasoning (e.g., "high temperature environment" automatically completes the "heat protection" constraint) ensures that the parsing results contain complete task elements.

[0081] This embodiment overcomes the limitations of traditional methods that rely on keyword matching. By using contextual reasoning through a large language model to identify the user's implicit intent (e.g., "Don't touch the vase next to you" is automatically interpreted as "The path planning needs to avoid the area where the vase is located"), the completeness of complex instruction element extraction is improved from 70% to 95%. Furthermore, by combining rule-based and deep learning-based constraint extraction methods, it achieves the transformation from natural language to formal constraints (e.g., quantifying the colloquial instruction "Do it quickly" as "Execution time ≤ base time × 1.2 times"), providing clear constraint parameters for subsequent task decomposition and action planning.

[0082] In one embodiment, such as Figure 5 As shown, step S150, which involves recursively decomposing complex instructions into atomic task sequences based on the core intent and key constraints of the instruction, and in conjunction with a pre-defined task knowledge base, includes:

[0083] Step S151: Based on the core intent of the instruction, match the corresponding complex task template from the preset task knowledge base.

[0084] Specifically, an atomic task sequence comprises multiple atomic tasks, which can be stored in XML format. These tasks include functional descriptions (e.g., "grab": control gripper closure; input parameters: object pose; output parameters: successful grasp flag), execution logic (e.g., "grab" requires first detecting the object's grasping point before sending gripper control signals), and constraint compatibility tables (e.g., "vacuum suction cups are prohibited from use in high-temperature environments"). Complex task templates define hierarchical relationships (e.g., "assembly" = "positioning + grasping + aligning + releasing"). Each template contains corresponding decomposition rules (e.g., decomposition by spatial order / functional order) and parameter passing logic (e.g., the output pose of "positioning" serves as the input for "grab"). When determining the atomic task sequence, the corresponding complex task template is first matched from a pre-defined task knowledge base based on the core intent of the instruction (e.g., "assembly") to generate the root node.

[0085] Step S152: Based on the recursive decomposition algorithm, the complex task template is decomposed layer by layer into indivisible atomic tasks.

[0086] Specifically, when decomposing a complex task template layer by layer using a recursive decomposition algorithm, for each subtask, it is first determined whether it is an atomic task. If not, it is decomposed further (e.g., "localization" is decomposed into "camera calibration + object detection") until all leaf nodes are decomposed into indivisible atomic tasks (e.g., "moving joint 1 to θ=30°").

[0087] Step S153: Perform dependency checks on the execution order, resource consumption, and constraint compatibility of atomic tasks, and generate an atomic task execution tree that includes task priority and timing constraints.

[0088] Specifically, when performing dependency checks, a Directed Acyclic Graph (DAG) can be used for modeling. First, the execution order of atomic tasks is determined; for example, "grabbing" must be executed after "localization." Simultaneously, resource consumption is checked (e.g., two tasks simultaneously occupy the robotic arm), and the task order is adjusted using a Constraint Compatibility (CSP) algorithm. At this point, each node in the generated atomic task execution tree includes: task ID, input parameters, constraints, and execution priority (e.g., obstacle avoidance task has higher priority than energy consumption optimization).

[0089] Step S154: Determine the atomic task sequence based on the leaf nodes of the atomic task execution tree.

[0090] Specifically, by arranging the leaf nodes of the constructed atomic task execution tree in execution order, an atomic task sequence can be obtained. For example, the atomic task sequence is: [Task1: Locate, Task2: Grab, Task3: Move, Task4: Release].

[0091] This embodiment upgrades the "hard-coded rules" relied upon in traditional methods to "data-driven decomposition" by using a pre-set knowledge base and recursive algorithms. This supports task adjustments in dynamic environments (such as automatically inserting an "obstacle avoidance" subtask when an obstacle is detected), increasing the success rate of multi-step task decomposition from 60% to 92%. Simultaneously, a checking mechanism is used to avoid task execution conflicts. For example, under a "30-second time limit" constraint, decomposition paths that exceed the time limit are automatically eliminated, ensuring that task planning meets potential user constraints and improving task execution reliability.

[0092] In one embodiment, such as Figure 6 As shown, step S160, which generates a spatiotemporally continuous action planning candidate set based on the atomic task sequence, visual spatial features, and robot kinematic features, includes:

[0093] Step S161: For each atomic task in the atomic task sequence, multiple candidate trajectories that satisfy dynamic constraints are generated by combining visual spatial features and robot kinematic features through a motion planning algorithm.

[0094] Specifically, when determining the motion planning candidate set, multiple candidate trajectories that satisfy dynamic constraints are first determined for each atomic task in the atomic task sequence. For example, for the "grasp" atomic task, the RRT* algorithm is used to search for a collision-free path in the configuration space (C-space). Its initial state is the robot's current pose, and the target state is the pose of the object grasping point (calculated by combining visual spatial features). Its constraints include joint velocity limits (≤180° / s) and acceleration limits (≤100° / s²). By optimizing different objectives (such as trajectory 1: shortest path; trajectory 2: minimum joint torque; trajectory 3: maximum obstacle avoidance safe distance), multiple candidate trajectories that satisfy dynamic constraints can be generated. Each candidate trajectory includes: a joint angle sequence (time resolution 10ms) and an end-effector pose sequence.

[0095] Step S162: Spatiotemporally and spatially stitch together the candidate trajectories of adjacent atomic tasks to form a candidate set of action planning that covers the entire task process.

[0096] Specifically, for adjacent atomic tasks in an atomic task sequence, their candidate trajectories need to be spatiotemporally continuously spliced ​​to form multiple motion planning candidates covering the entire task flow, thus obtaining a motion planning candidate set. For example, the end pose endpoint of the previous task (e.g., "grab") is used as the starting point of the next task (e.g., "move"), and the connection point is smoothed using Bézier curve interpolation to ensure continuous velocity and acceleration (jerk ≤ 50° / s³). Simultaneously, user constraints (e.g., "accuracy ± 0.5 mm") are converted into a trajectory endpoint position error threshold (≤ 0.5 mm), and accumulated errors are checked during splicing to eliminate trajectories exceeding the limit. At least 5 differentiated candidate trajectories (e.g., different obstacle avoidance paths, different end poses) are generated for each atomic task, ultimately forming a motion planning candidate set containing 10 to 20 complete task flows. Each motion planning candidate in the set includes: joint space trajectory, Cartesian space path, and execution time parameters. Candidate trajectories for the same atomic task have different obstacle avoidance strategies or energy consumption optimization objectives.

[0097] This embodiment generates candidate trajectories by combining real-time visual spatial features (such as the location of dynamic obstacles). Compared with traditional offline planning, the obstacle avoidance success rate in dynamic scenarios is increased from 70% to 90%, and the trajectory smoothness is improved by 60% (jerk value is reduced by 40%), reducing robotic arm vibration. Through diverse candidate trajectories, it supports flexible trade-offs among multiple objectives such as time, accuracy, and energy consumption. For example, in medical scenarios, the trajectory with the highest accuracy is prioritized, while in industrial scenarios, the trajectory with the lowest energy consumption is prioritized to meet the needs of different applications.

[0098] In one embodiment, such as Figure 7 As shown, step S170, which involves outputting cross-modal association features based on user command text, environmental visual images, and action planning candidate sets in the joint embedding space, includes:

[0099] Step S171: Semantically encode the user command text to obtain a language embedding vector.

[0100] Specifically, the T5 model can be used for semantic encoding, and prefix-tuning can be used to adapt to the robot task scenario, outputting language embedding vectors that preserve the syntax and semantics of the instructions. The dimension of the language embedding vectors is 1024.

[0101] Step S172: Extract features from the environmental visual image to obtain a visual embedding vector.

[0102] Specifically, Faster R-CNN can be used to extract local area features (ROI Pooling) and combined with position encoding (sine and cosine functions) to generate visual embedding vectors containing spatial locations. The dimensions of the visual embedding vectors are 1024.

[0103] Step S173: Encode motion features in the action planning candidate set to obtain action embedding vectors.

[0104] Specifically, the trajectory parameters (such as joint angle sequences, end pose sequences, etc.) of the motion trajectory in the motion planning candidate set can be converted into time series data, local motion patterns can be extracted by 1D-CNN, and then global average pooling can be used to generate motion embedding vectors with a dimension of 1024.

[0105] Step S174: Through the cross-modal contrastive learning algorithm, the language embedding vector, visual embedding vector, and action embedding vector are semantically aligned in the joint embedding space to obtain cross-modal association features.

[0106] Specifically, when performing semantic alignment using a cross-modal contrastive learning algorithm, positive sample pairs consist of language-visual-action data from the same task (e.g., the instruction "grab the red cup," the corresponding image, and the successful trajectory); negative sample pairs consist of randomly matched cross-modal data from different tasks; the loss function is the NT-Xent loss function.

[0107]

[0108] Among them, s i Let τ be the similarity of positive sample pairs, and τ be the temperature parameter (set to 0.1). Contrastive learning forces similar modalities to be close together in the embedding space, while dissimilar ones are kept away. In some embodiments, the joint embedding space can be checked periodically using visualization tools (such as t-SNE) to ensure that the embeddings of "grabbing" related language, vision (object grasping points), and action (gripper closure trajectory) form clusters, with cross-modal semantic consistency exceeding 90%.

[0109] In this embodiment, a contrastive learning algorithm enables semantic correspondences between different modal features in the joint embedding space (e.g., the visual obstacle region corresponding to the "obstacle avoidance" command is highly correlated with the action obstacle avoidance trajectory embedding). This solves the decision-making fragmentation problem caused by independent modal processing in traditional methods, improving the cross-modal reasoning accuracy in complex scenarios from 50% to 88%. Furthermore, the joint embedding space supports zero-shot learning (e.g., in the untrained "place in a specified color region" task, visual color features can be semantically correlated with the action localization trajectory), reducing reliance on massive training data and improving the ability to adapt to new tasks by 40%.

[0110] In one embodiment, such as Figure 8 As shown, step S180, which calculates the contextual similarity between cross-modal association features and historical interaction data features, includes:

[0111] Step S181: Extract historical successful interaction data of the same type of task from the historical interaction database and use it as the feature of historical interaction data.

[0112] Specifically, the historical interaction database stores multiple successful historical interaction data. First, based on the task type (such as the core intent of the instruction), the historical successful interaction data of the same type of task is extracted from the historical interaction database and used as the characteristics of the historical interaction data.

[0113] Step S182: Calculate the weighted fusion value of the semantic matching degree, environmental feature matching degree, and action trajectory smoothness of the cross-modal association features and historical interaction data features based on preset weights, and use it as the context similarity.

[0114] Specifically, the historical interaction data features also include corresponding language embedding vectors, visual embedding vectors, and action embedding vectors. Semantic matching is the cosine similarity between the language embedding vectors in the cross-modal association features and the language embedding vectors in the historical interaction data features. Environmental feature matching can be achieved by comparing object positions, obstacle distributions, etc., in two visual embedding vectors; Earth Mover's Distance can be used to calculate the spatial difference as the environmental feature matching. Action trajectory smoothness can be calculated using Dynamic Time Warping (DTW) to determine the similarity of the trajectory curves represented by two action embedding vectors. Preset weights are set for semantic matching, environmental feature matching, and action trajectory smoothness. These preset weights can be dynamically adjusted based on key constraints; for example, if the key constraint is "time priority," the weight of the action trajectory time parameter is increased to 0.6. In some embodiments, the weights of each dimension can be adjusted in real time through reinforcement learning. For example, the weight vector α = [α1, α2, α3], Σα = 1. After weighted calculation, a weighted fusion value is obtained and used as the context similarity.

[0115] In this embodiment, the traditional "planning from scratch" is transformed into "matching similar cases" by using historical successful interaction data stored in the historical interaction database, which improves the planning speed by 50% (the planning time for complex tasks is reduced from 20s to 10s) and can inherit historical optimization experience (such as mature obstacle avoidance strategies).

[0116] In some embodiments, step S190, which involves selecting the optimal action planning candidate with the highest similarity from the action planning candidate set based on contextual similarity, includes: sorting the action planning candidates in the action planning candidate set based on contextual similarity to determine the action planning candidate with the highest similarity, and using it as the optimal action planning candidate. In some embodiments, if the contextual similarity of the optimal action planning candidate is less than a preset threshold (e.g., 0.7), planning is retried. Simultaneously, historical failure cases (e.g., adding penalties for trajectories that previously failed due to collisions) are incorporated to avoid repeating errors.

[0117] In some embodiments, such as Figure 9 As shown, after step S210, which involves controlling the robot's movement based on the optimal motion planning candidate, the robot control method further includes:

[0118] Step S220: Collect execution feedback data in real time.

[0119] Specifically, the robot collects execution feedback data in real time during its movement. This execution feedback data includes process data and result data. Process data includes real-time collected joint current (reflecting load), force sensor data (indicating abnormal contact force), and visual repositioning results (whether the object has shifted). Result data includes task completion status (success / failure), user feedback (satisfaction evaluation via touchscreen or voice input), and error statistics (such as target position error and time deviation).

[0120] Step S230: Dynamically update the joint embedding space based on the execution feedback data to improve the generalized execution capability of complex instructions.

[0121] Specifically, if the execution feedback data indicates three consecutive failures of the same task, joint embedding space fine-tuning is triggered. The execution feedback data is used as training samples, the underlying encoder is frozen, and only the cross-modal attention layer parameters are updated. In some embodiments, the training batch size for updating the cross-modal attention layer parameters is set to 32, and the learning rate is 1e. -4 .

[0122] This embodiment corrects system deviations in real time through feedback. For example, after detecting that "glass objects are easily slipped during grasping," it automatically adjusts the force threshold corresponding to the "handle gently" constraint (from 5N to 3N), increasing the success rate of similar tasks from 60% to 85%, forming a complete intelligent closed loop of "perception-execution-learning." The dynamic update mechanism allows the system to adapt to new environments (such as changing the robotic arm model) or user habits (such as specific operating preferences) without redeployment. Task execution capabilities continuously improve throughout the system's lifecycle, and maintenance costs are reduced by more than 30%.

[0123] In some embodiments, if the task is determined to be successful based on the result data in the execution feedback data, the successful cases (with optimized trajectory parameters) are stored in the historical interaction database; otherwise, the failed cases are marked with their error type (such as "collision", "constraint not satisfied", etc.) for the penalty mechanism in subsequent similarity calculations.

[0124] In some embodiments, virtual training data (simulating the visual features of transparent objects through data augmentation) is generated for high-frequency failure scenarios (such as "transparent object grasping") and injected into the joint embedding space for training to improve the model's ability to handle rare scenarios.

[0125] The robot control method based on the above multimodal large model has at least the following beneficial effects:

[0126] 1. Multimodal deep fusion improves interaction accuracy: By spatiotemporal alignment and feature fusion of text, vision and robot state data, a comprehensive perception of user intent and environmental state is achieved, avoiding comprehension bias caused by the lack of information from a single modality;

[0127] 2. Hierarchical recursive decomposition enhances task adaptability: Based on a large language model and task knowledge base, the complex instruction decomposition strategy supports flexible task decomposition in dynamic environments and can handle multi-level complex tasks that are difficult for traditional systems to handle.

[0128] 3. Cross-modal association optimization of action planning: The joint embedding space achieves semantic alignment of visual, linguistic, and action features, and the similarity assessment of historical interaction data enables action planning to simultaneously meet semantic intent, environmental constraints, and historical success experience, thereby improving execution reliability;

[0129] 4. Closed-loop feedback mechanism enhances generalization ability: By dynamically updating the joint embedding space through execution feedback, the system can autonomously optimize the cross-modal association model, adapt to untrained scenarios, and enhance generalization performance in long-term interactions;

[0130] 5. Systematic Intelligent Decision-Making Framework: It organically integrates context awareness, large model reasoning, task decomposition, motion planning, and cross-modal screening to form a full-link intelligent closed loop from perception to execution, providing a general solution for complex robot interactions.

[0131] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0132] In one embodiment, a robot is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0133] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0134] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot control method based on a multimodal large model, characterized in that, The method includes: Acquire multimodal input data; wherein, the multimodal input data includes: user command text, environmental visual images, and robot body state data; The multimodal input data is preprocessed to extract contextual features; wherein, the contextual features include: semantic features, visual spatial features, and robot kinematic features; The semantic features are analyzed based on a large language model to extract the core intent of the instruction; The user instruction text is subjected to constraint extraction to obtain key constraints; Based on the core intent of the instruction and the key constraints, and combined with a preset task knowledge base, the complex instruction is recursively decomposed into a sequence of atomic tasks. Based on the atomic task sequence, the visual spatial features, and the robot kinematic features, a spatiotemporally continuous set of action planning candidates is generated. In the joint embedding space, cross-modal association features are output based on the user instruction text, the environmental visual image, and the action planning candidate set; wherein, the joint embedding space is a pre-constructed embedding space for vision, language, and action; Calculate the contextual similarity between the cross-modal association features and the historical interaction data features; Based on the contextual similarity, the optimal action planning candidate with the highest similarity is selected from the action planning candidate set; The robot's movement is controlled based on the aforementioned optimal motion planning candidate.

2. The method according to claim 1, characterized in that, The step of acquiring multimodal input data includes: The user command text is acquired in real time via a microphone array or text input interface; The environment is visually captured by cameras deployed in the environment or by visual sensors mounted on the robot. The robot body's state data is collected through an inertial measurement unit, joint encoder, and force sensor integrated into the robot body. The multimodal input data is subjected to timestamp alignment and noise filtering.

3. The method according to claim 1, characterized in that, The step of preprocessing the multimodal input data and extracting contextual features includes: The user instruction text is segmented, part-of-speech tagging is performed, and named entity recognition is performed. The semantic features are then generated by combining the pre-trained language model. The environmental visual image is subjected to target detection, semantic segmentation and depth estimation, and the visual spatial features containing object category, spatial location and geometric size are extracted; The inverse kinematics calculation is performed on the joint angles in the robot body state data, and the extracted dynamic features are combined to obtain the robot kinematic features; wherein, the dynamic features include: joint angular velocity, joint acceleration and motor torque; The semantic features, the visual spatial features, and the robot kinematic features are aligned in a spatiotemporal dimension to form the contextual features of multimodal fusion.

4. The method according to claim 1, characterized in that, The step of parsing the semantic features based on a large language model and extracting the core intent of the instruction includes: The semantic features are contextually modeled using a multi-layer Transformer encoder of a large language model, and the intent is classified using an intent classifier to obtain the core intent of the instruction; wherein, the core intent of the instruction includes: task execution, path planning, and status query.

5. The method according to claim 1, characterized in that, The step of extracting constraints from the user instruction text to obtain key constraints includes: Constraint keywords are extracted from the user instruction text based on rule matching and sequence labeling; Based on a preset constraint rule base and common sense reasoning, the fuzzy constraints in the constraint keywords are logically completed to obtain the key constraint conditions; wherein, the key constraint conditions include: time limit, spatial forbidden zone, and operation precision.

6. The method according to claim 1, characterized in that, The step of recursively decomposing complex instructions into atomic task sequences based on the core intent of the instruction and the key constraints, combined with a preset task knowledge base, includes: Based on the core intent of the instruction, a corresponding complex task template is matched from a preset task knowledge base; The complex task template is decomposed into indivisible atomic tasks layer by layer based on the recursive decomposition algorithm. The execution order, resource consumption, and constraint compatibility of the atomic tasks are checked for dependencies, and an atomic task execution tree containing task priorities and timing constraints is generated. The atomic task sequence is determined based on the leaf nodes of the atomic task execution tree.

7. The method according to claim 1, characterized in that, The step of generating a spatiotemporally continuous action planning candidate set based on the atomic task sequence, the visual spatial features, and the robot kinematic features includes: For each atomic task in the atomic task sequence, multiple candidate trajectories that satisfy dynamic constraints are generated by combining the visual spatial features and the robot kinematic features through a motion planning algorithm. The candidate trajectories of adjacent atomic tasks are spatiotemporally continuous to form the action planning candidate set covering the entire task process; wherein, each action planning candidate in the action planning candidate set includes: joint space trajectory, Cartesian space path and execution time parameters, and the candidate trajectories of the same atomic task have different obstacle avoidance strategies or energy consumption optimization objectives.

8. The method according to claim 1, characterized in that, The step of outputting cross-modal association features in the joint embedding space based on the user instruction text, the environmental visual image, and the action planning candidate set includes: The user command text is semantically encoded to obtain a language embedding vector; Feature extraction is performed on the environmental visual image to obtain a visual embedding vector; Motion feature encoding is performed on the action planning candidate set to obtain action embedding vectors; By using a cross-modal contrastive learning algorithm, the language embedding vector, the visual embedding vector, and the action embedding vector are semantically aligned in the joint embedding space to obtain the cross-modal association features.

9. The method according to claim 1, characterized in that, The step of calculating the contextual similarity between the cross-modal association features and the historical interaction data features includes: Extract successful historical interaction data of similar tasks from the historical interaction database and use it as a feature of the historical interaction data. The weighted fusion value of the semantic matching degree, environmental feature matching degree, and action trajectory smoothness of the cross-modal association features and the historical interaction data features is calculated based on preset weights and used as the context similarity; wherein, the preset weights are dynamically adjusted according to the key constraints.

10. The method according to claim 1, characterized in that, Following the step of controlling the robot's movement based on the optimal motion planning candidate, the method further includes: Real-time collection of execution feedback data; The joint embedding space is dynamically updated based on the execution feedback data to improve the generalized execution capability of complex instructions.

Images