A semantic constraint instant distillation humanoid robot control method based on a large language model and related equipment
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 广州里工实业有限公司
- Filing Date
- 2025-09-04
- Publication Date
- 2026-06-02
Smart Images
Figure CN121340233B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robot control technology, and in particular to a control method and related equipment for a humanoid robot based on semantic constraints of a large language model for instantaneous distillation. Background Technology
[0002] In related technologies, traditional humanoid robot control systems mainly rely on fixed policy libraries, rule systems, or deep reinforcement learning models, achieving initial results in motion control, path planning, and task execution. However, these systems typically have two significant limitations: first, task parameters and environmental constraints must be preset in a structured instruction manner, lacking flexibility and semantic understanding capabilities; second, the coupling between perception and control is weak, making it difficult to achieve rapid response to complex semantic instructions or motion adjustments under changing scene conditions.
[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention
[0004] The main objective of this application is to propose a semantically constrained real-time distillation humanoid robot control method and related equipment based on a large language model, which can meet the dual requirements of real-time performance and dynamic consistency for humanoid robots.
[0005] To achieve the above objectives, one aspect of this application proposes a semantically constrained real-time distillation humanoid robot control method based on a large language model, the method comprising the following steps:
[0006] Obtain user natural language commands;
[0007] The user's natural language instructions are semantically parsed using a large language model to obtain a structured semantic intent tensor;
[0008] The semantic intent tensor is compressed and translated using a local lightweight model to obtain control parameters;
[0009] The humanoid robot is controlled to complete the target action according to the control parameters, and the execution status feedback information is collected.
[0010] Based on the execution status feedback information, semantic offset detection is performed, and the local lightweight model is adjusted online to form a closed-loop control path.
[0011] In some embodiments, the step of semantically parsing the user's natural language instructions using a large language model to obtain a structured semantic intent tensor includes:
[0012] The user's natural language instructions are reasoned through multiple rounds using a multi-round chained reasoning mechanism through a large language model. The instructions are decomposed into action logic units, and a structured semantic intent tensor is generated by combining environmental knowledge. The large language model is a pre-trained language model with instruction following capability, used to perform structured parsing of action targets, object attributes, temporal logic, and environmental constraints. The fields of the semantic intent tensor include action type, object features, spatial constraints, action sequence, interaction direction, and security parameters. The fields of the semantic intent tensor are represented using vector or sparse coding methods.
[0013] In some embodiments, the compression and translation of the semantic intent tensor using a local lightweight model to obtain control parameters includes:
[0014] Configure a local lightweight model; the local lightweight model includes a miniature Transformer model or a weighted distillation network using LoRA adjustment techniques;
[0015] The semantic intent tensor is transmitted to the robot's local control terminal through compression encoding, and is parsed in real time by the local lightweight model. The semantic-control mapping capability is obtained through distillation. Compression and translation are performed once at a preset period during runtime to transform the high-dimensional description in the semantic intent tensor into the parameter form received by the controller, thus obtaining control parameters. The control parameters include the target joint angle sequence, trajectory constraint window, impedance adjustment coefficient, and motion termination condition.
[0016] In some embodiments, controlling the humanoid robot to complete the target action according to the control parameters and collecting execution status feedback information includes:
[0017] Based on the control parameters, and using a model predictive control framework, the desired parameter trajectory is mapped to a high-frequency driving signal. At the same time, the current sensor data of the humanoid robot is corrected and interpolated to drive the humanoid robot to complete the target action.
[0018] During the humanoid robot's task execution, execution status feedback information is continuously collected; the execution status feedback information includes position error, task completion degree, control delay, and energy consumption.
[0019] In some embodiments, the step of performing semantic offset detection based on the execution state feedback information and adjusting the local lightweight model online to form a closed-loop control path includes:
[0020] Based on the execution status feedback information, semantic offset detection is performed, and combined with trajectory comparison, temporal deviation analysis and feedback confidence calculation, execution deviation information is determined.
[0021] Based on the execution deviation information, the failed samples are analyzed and their labels are traced back to extract the abnormal triggering factors;
[0022] The local lightweight model is adjusted online using the aforementioned abnormal triggering factors to obtain an optimized strategy and form a closed-loop control path.
[0023] To achieve the above objectives, another aspect of this application proposes a semantically constrained real-time distillation humanoid robot control system based on a large language model, used to implement the aforementioned method. The system includes a natural language task parsing module, a semantic intent encoding and distillation module, a control parameter generation and execution module, and a state feedback and dynamic optimization module.
[0024] The natural language task parsing module is used to parse natural language instructions into structured semantic intent tensors based on a large language model;
[0025] The semantic intent encoding and distillation module is used to compress and translate the semantic intent tensor based on a local lightweight model to obtain control parameters;
[0026] The control parameter generation and execution module is used to convert the control parameters into continuous control signals and drive the humanoid robot to move, while collecting execution status feedback information.
[0027] The state feedback and dynamic tuning module is used to perform semantic offset detection based on the execution state feedback information and to adjust the local lightweight model online.
[0028] In some embodiments, the natural language task parsing module is deployed in the cloud or at an edge node; the semantic intent encoding and distillation module is deployed on the robot's local control board; and the control parameter generation and execution module is deployed on the robot's execution layer.
[0029] In some embodiments, the semantic intent encoding and distillation module has a semantic-dynamic nested contrastive loss function, which is trained and optimized through an online weight adjustment mechanism.
[0030] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.
[0031] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.
[0032] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above.
[0033] The embodiments of this application include at least the following beneficial effects: This application provides a method, system, electronic device, storage medium, and program product for controlling a humanoid robot based on a large language model with semantic constraints for real-time distillation. The invention includes: acquiring user natural language commands; performing semantic parsing of the user natural language commands using a large language model to obtain a structured semantic intent tensor; compressing and translating the semantic intent tensor using a local lightweight model to obtain control parameters; controlling the humanoid robot to complete target actions according to the control parameters and collecting execution status feedback information; performing semantic offset detection based on the execution status feedback information and adjusting the local lightweight model online to form a closed-loop control path. This application can improve the semantic understanding ability, action generation efficiency, feedback adaptability, and cross-scene generalization ability of humanoid robots. Attached Figure Description
[0034] Figure 1 This is a flowchart of the semantically constrained instantaneous distillation humanoid robot control method based on a large language model provided in the embodiments of this application;
[0035] Figure 2 This is an overall operation flowchart provided in the embodiments of this application;
[0036] Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.
[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0039] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.
[0040] 1) LoRA (Low-Rank Adaptation) is an efficient fine-tuning technique for large models (such as large language models).
[0041] 2) Transformer, a deep learning model architecture based on the self-attention mechanism;
[0042] 3) LLM (Large Language Model);
[0043] 4) Chain-of-Thought: A prompting technique that guides large language models in reasoning.
[0044] 5) MPC (Model Predictive Control), a control strategy;
[0045] 6) IMU (Inertial Measurement Unit), a type of sensor;
[0046] 7) NPU (Neural Processing Unit);
[0047] 8) RTOS (Real-Time Operating System);
[0048] 9) Qwen2-7B, Tongyi Qianwen 2nd Generation 7B Model;
[0049] 10) WebSocket, a network communication protocol;
[0050] 11) TinyTransformer, a micro-transformer.
[0051] As the hardware capabilities of humanoid robots continue to improve, they are gradually becoming practically deployable in service, healthcare, and manufacturing scenarios. Traditional humanoid robot control systems primarily rely on fixed policy libraries, rule systems, or deep reinforcement learning models, achieving initial success in motion control, path planning, and task execution. However, these systems typically suffer from two significant limitations: first, task parameters and environmental constraints must be preset using structured instructions, lacking flexibility and semantic understanding capabilities; second, the coupling between perception and control is weak, making it difficult to achieve rapid response to complex semantic instructions or motion adjustments under varying scenario conditions.
[0052] In recent years, Large Language Models (LLMs) have demonstrated powerful capabilities in semantic understanding and contextual reasoning, sparking a research boom into their integration into robot control chains. Some initial explorations have attempted to parse natural language using LLMs and then invoke existing skill modules to complete actions. However, these methods generally suffer from problems such as long inference chains, high response times, and disconnection from the underlying dynamic controllers, failing to meet the dual requirements of real-time performance and dynamic consistency for humanoid robots.
[0053] Therefore, a novel mechanism is urgently needed to efficiently integrate the high-level semantic understanding capabilities of large language models with the robot's underlying control logic, enabling natural language commands to be instantly converted into dynamic parameters and operate stably within a closed-loop control cycle, thereby truly achieving flexible autonomous control that integrates semantics and dynamics. This is precisely the key problem that this invention aims to solve.
[0054] This invention provides a semantic constraint real-time distillation humanoid robot control method and related equipment based on a large language model. It parses natural language tasks using a cloud-based large model to generate structured control intentions; then, a local lightweight model performs semantic compression and real-time distillation to quickly convert these into joint-level control parameters, achieving efficient alignment between semantic information and underlying control. This method constructs a three-layer linkage mechanism of "semantic understanding—parameter distillation—real-time control," significantly improving the response speed and control stability of humanoid robots for complex tasks.
[0055] This invention considers that humanoid robot control systems in related technologies typically employ two mainstream approaches: one based on offline-trained deep reinforcement learning models, and the other based on skill trees or hand-crafted policy libraries, invoked through rule-based logic. While these methods perform well in specific tasks, they suffer from the following key problems:
[0056] 1) Semantic access fragmentation: Traditional control systems have difficulty directly processing natural language or complex semantic inputs, and often require manual conversion of tasks into parameterized instructions, which severely limits the naturalness and versatility of human-computer interaction.
[0057] High control latency: Even with the introduction of large language models for task parsing, the complexity of LLM inference structures, coupled with their typical deployment in the cloud and latency bottlenecks between the model and the local robot, makes them unsuitable for high-frequency control loops.
[0058] 2) Semantic and dynamic decoupling: Current methods have difficulty directly mapping the semantic reasoning results of LLM to the target parameters or optimization constraints available to the underlying dynamic controller, resulting in unstable, uninterpretable or unrobust control strategies.
[0059] 3) Lack of adaptive compression mechanism: During execution, the robot cannot quickly compress and reconstruct semantic intent according to the actual state, and cannot support the real-time control requirements in dynamic environments.
[0060] Based on the aforementioned technical bottlenecks, this invention proposes an intelligent control method that integrates cloud-based large language model semantic understanding with a local control parameter instant distillation mechanism (a humanoid robot control method based on semantic constraint instant distillation of a large language model). The goal is to achieve efficient bridging between natural language tasks and the underlying control system of the humanoid robot, meet the response requirements at the control cycle level, and simultaneously possess high interpretability, strong adaptability, and good control stability.
[0061] Figure 1 This is an optional flowchart of the semantically constrained instantaneous distillation humanoid robot control method based on a large language model provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.
[0062] Step S101: Obtain user's natural language instructions;
[0063] Step S102: Semantic parsing of the user's natural language instructions is performed using a large language model to obtain a structured semantic intent tensor;
[0064] Step S103: Compress and translate the semantic intent tensor using a local lightweight model to obtain control parameters;
[0065] Step S104: Control the humanoid robot to complete the target action according to the control parameters, and collect the execution status feedback information;
[0066] Step S105: Based on the execution status feedback information, perform semantic offset detection, adjust the local lightweight model online, and form a closed-loop control path.
[0067] Steps S101 to S105 as illustrated in the embodiments of this application achieve an optimal balance between performance and efficiency by leveraging the powerful understanding capabilities of the large language model and relying on the real-time response capabilities of the local lightweight model, thereby improving the real-time performance (low latency) and stability of the control.
[0068] In step S101 of some embodiments, an automatic speech recognition module can be integrated to collect the user's speech signal through a microphone array, perform noise reduction, enhancement, and recognition, and transcribe the speech into the user's natural language commands in real time. Alternatively, a graphical user interface or command-line interface can be provided in other ways to allow users to directly input text commands. This method is suitable for noisy environments or scenarios requiring precise command input, and is not limited to these methods.
[0069] In some embodiments, step S102 may include, but is not limited to, step S201:
[0070] Step S201: Using a large language model and a multi-round chained reasoning mechanism, the user's natural language instruction is subjected to multiple rounds of reasoning. The user's natural language instruction is decomposed into action logic units, and a structured semantic intent tensor is generated by combining environmental knowledge. The large language model is a pre-trained language model with instruction following capability, used to perform structured parsing of action targets, object attributes, temporal logic, and environmental constraints. The fields of the semantic intent tensor include action type, object features, spatial constraints, action sequence, interaction direction, and security parameters. The fields of the semantic intent tensor are represented by vector or sparse coding.
[0071] In step S201 of some embodiments, the unstructured language is converted into a structured tensor containing explicit fields such as action type and spatial constraints, providing a highly standardized and unambiguous input for the downstream control system and laying the foundation for subsequent compression and distillation.
[0072] In some embodiments, step S103 may include, but is not limited to, steps S301 to S302:
[0073] Step S301: Set up a local lightweight model; the local lightweight model includes a miniature Transformer model or a weighted distillation network using LoRA adjustment technology.
[0074] Step S302: The semantic intent tensor is transmitted to the robot's local control terminal through compression encoding. The local lightweight model performs real-time parsing and obtains semantic-control mapping capability through distillation. During runtime, compression and translation are performed once at a preset period to convert the high-dimensional description in the semantic intent tensor into parameter form received by the controller, thereby obtaining control parameters. The control parameters include target joint angle sequence, trajectory constraint window, impedance adjustment coefficient, and motion termination condition.
[0075] In step S301 of some embodiments, a micro Transformer or weighted distillation network is used to ensure that the model has a very small number of parameters and computational cost, so that it can be deployed on an embedded chip with limited computing power to meet the real-time requirements of robot control.
[0076] In step S302 of some embodiments, the high-dimensional, abstract "intention" is distilled into low-dimensional, specific, controller-executable motion parameters (such as joint angles and impedance coefficients), and translated at high speed and periodically at a preset period (such as 30-50ms). The output frequency is matched with the operating frequency of the robot controller, thereby generating smooth and continuous control commands to drive the robot to complete smooth and stable movements.
[0077] In some embodiments, step S104 may include, but is not limited to, steps S401 to S402:
[0078] Step S401: Based on the control parameters and the model predictive control framework, the desired parameter trajectory is mapped to a high-frequency driving signal. At the same time, the current sensor data of the humanoid robot is corrected and interpolated to drive the humanoid robot to complete the target action.
[0079] Step S402: During the process of the humanoid robot performing the task, continuous collection of execution status feedback information is carried out; the execution status feedback information includes position error, task achievement degree, control delay and energy consumption.
[0080] In step S401 of some embodiments, a model predictive control (MPC) framework is adopted. Instead of simply tracking the current target, the controller predicts the state over a future period based on the model and optimizes and calculates the optimal control sequence.
[0081] In step S402 of some embodiments, by monitoring indicators such as task completion rate and delay, it can not only be used for closed-loop control, but also provide data logs for system performance evaluation, fault diagnosis and long-term evolution.
[0082] In some embodiments, step S105 may include, but is not limited to, steps S501 to S503:
[0083] Step S501: Based on the execution status feedback information, semantic offset detection is performed, and combined with trajectory comparison, temporal deviation analysis and feedback confidence calculation, execution deviation information is determined.
[0084] Step S502: Based on the execution deviation information, analyze and label backtrack the failed samples to extract the abnormal triggering factors;
[0085] Step S503: The local lightweight model is adjusted online using the abnormal triggering factors to obtain an optimization strategy and form a closed-loop control path.
[0086] In step S501 of some embodiments, a comprehensive judgment is made by combining multiple dimensions such as trajectory, time sequence, and confidence level, which avoids the limitations of single indicator detection and can improve the accuracy and reliability of semantic offset detection.
[0087] In step S502 of some embodiments, the extracted abnormal triggering factors constitute samples with clear error labels. Referring to these samples for model fine-tuning can improve the model's performance.
[0088] In step S503 of some embodiments, the three links of detection, analysis, and adjustment are linked together to form a complete closed-loop control path of "perception-decision-execution-learning".
[0089] This application also provides a semantically constrained real-time distillation humanoid robot control system based on a large language model, used to implement the aforementioned method. The system includes a natural language task parsing module, a semantic intent encoding and distillation module, a control parameter generation and execution module, and a state feedback and dynamic optimization module.
[0090] The natural language task parsing module is used to parse natural language instructions into structured semantic intent tensors based on a large language model;
[0091] The semantic intent encoding and distillation module is used to compress and translate the semantic intent tensor based on a local lightweight model to obtain control parameters;
[0092] The control parameter generation and execution module is used to convert the control parameters into continuous control signals and drive the humanoid robot to move, while collecting execution status feedback information.
[0093] The state feedback and dynamic tuning module is used to perform semantic offset detection based on the execution state feedback information and to adjust the local lightweight model online.
[0094] Furthermore, these four modules form a complete closed loop of "perception-cognition-decision-execution-learning", achieving end-to-end alignment of semantics and control, and efficiently and accurately linking the fuzzy natural language commands of humans with the precise joint motion control of robots.
[0095] In some embodiments, the natural language task parsing module is deployed in the cloud or at an edge node; the semantic intent encoding and distillation module is deployed on the robot's local control board; and the control parameter generation and execution module is deployed on the robot's execution layer.
[0096] Furthermore, the modules are deployed in the cloud, at edge nodes, and locally on the robot. The large cloud model can provide services to multiple robots simultaneously, making it easy to expand and maintain. The local distillation model can be individually fine-tuned according to the specific dynamic parameters of each robot body (such as weight, arm length, and joint torque).
[0097] In some embodiments, the semantic intent encoding and distillation module has a semantic-dynamic nested contrastive loss function, which is trained and optimized through an online weight adjustment mechanism.
[0098] Furthermore, the use of semantic-dynamic nested contrastive loss functions can enable the model to gain a deeper understanding of the subtle correspondence between semantic intent and control outcome, thereby learning a more accurate and robust mapping model and avoiding information distortion during compression translation.
[0099] As an optional implementation, the system architecture of this embodiment mainly includes the following four core modules: a natural language task parsing module, a semantic intent encoding and distillation module, a control parameter generation and execution module, and a state feedback and dynamic optimization module. These components work together to form a real-time feedback closed loop from natural language understanding to humanoid robot motion control.
[0100] 1. Natural Language Task Parsing Module
[0101] This module, deployed in the cloud or on edge nodes, is built upon a Large Language Model (LLM) and is primarily used to understand user-input natural language task instructions. The model can not only identify task objectives (such as "hand me the cup"), but also infer complex semantic structures such as relevant environmental entities (e.g., where the "cup" is located), the order of actions, and potential safety constraints. Through contextual modeling and a chain-of-thought mechanism, this module can parse unstructured language information into structured intermediate representations with controllable meaning.
[0102] The parsing results are represented as a "semantic intent tensor," which includes elements such as the action target, expected joint pose, motion constraint boundaries, contact sequence, and interaction safety domain. This tensor is sent to downstream modules for compression and reconstruction for real-time input control. Compared to traditional semantic label classification, this parsing method is more versatile and expressive, capable of covering multi-step tasks and abstract goals.
[0103] 2. Semantic Intent Encoding and Distillation Module
[0104] This module consists of a lightweight model deployed on the local control board, primarily responsible for compressing and translating semantic intent tensors and outputting target parameters usable by the robot controller within milliseconds. This process employs a micro-distillation network or a structure-aware small model (such as a finely tuned version of LoRA) to instantly map complex semantic tensors into several dimensional control variables, such as target joint angles, desired trajectory curvature, and impedance coefficients.
[0105] To ensure a high-fidelity mapping from semantics to dynamic control, this module introduces a semantic-dynamic nested contrastive loss function and designs an online weight adjustment mechanism. This allows the module to learn while executing during robot tasks, achieving a dual optimization of semantic mapping efficiency and accuracy. The distillation process possesses context-memory capabilities, retaining task sequence information and adapting to the generation of coherent actions and conditional branching behaviors.
[0106] 3. Control Parameter Generation and Execution Module
[0107] This module is the robot's execution layer, built upon a traditional Model Predictive Controller (MPC) or deep dynamics control framework. It receives the target parameters output from the distillation module and rapidly interpolates them into a continuous control curve to meet high-frequency (e.g., every 30ms) closed-loop control requirements. Parameters include desired joint pose, velocity constraints, environmental contact priority, and motion sequence length.
[0108] The control module also integrates a tracking mechanism for the deviation between control constraints and the actual state, and uses feedback control laws for dynamic adjustments, such as gain adjustment or local correction strategies, to ensure the robot's robustness and stability in real-world physical scenarios. Furthermore, the execution layer can combine contact sensors, IMU data, and visual position feedback to achieve real-time correction of posture, balance, and path, constructing a complete physical interaction loop.
[0109] 4. Status Feedback and Dynamic Optimization Module
[0110] This module monitors the robot's current state data and control performance, using this data as feedback signals for model weight adjustment and dynamic reconstruction of the intent tensor. Its core functions include semantic drift detection, controller bias compensation, and online policy fine-tuning. By evaluating the consistency between semantic intent and execution results, issues such as semantic instruction misunderstanding or control distortion can be identified.
[0111] The system records execution data through a sliding window and uses temporal difference modeling to determine if there is a semantic-dynamic mapping offset. Once an offset is detected, a lightweight fine-tuning process is triggered on the local model, and the data is simultaneously fed back to the cloud for semantic update suggestions for the large model, thereby achieving closed-loop collaborative evolution between the semantic model and the physical controller.
[0112] refer to Figure 2 The system operation flow of this invention embodiment is as follows:
[0113] 1. User semantic task input and intent parsing
[0114] Users issue commands to the robot via voice, text, or natural language interfaces, such as "Please bring me the book" or "Help me stand up." The system first receives this semantic input and then uses a large language model deployed in the cloud to perform contextual analysis on the command, identifying key elements such as the task objective, the type of action involved, object information, and environmental constraints.
[0115] During the parsing process, the language model employs a multi-round chain-of-thought mechanism to break down instructions into controllable action logic units and generate intermediate control intentions by combining environmental knowledge (such as item type and spatial structure). These intentions are no longer general text outputs but rather structured semantic tensors containing elements such as action targets, spatial parameters, time steps, and pose targets, providing a highly usable semantic input format for downstream modules.
[0116] 2. Control semantic tensor construction and local distillation mapping
[0117] The semantic tensors generated in the cloud are transmitted to the robot's local control unit through compression encoding, where they are parsed in real time by a local lightweight model (such as a simplified version of Transformer or a weighted distillation network). This lightweight model obtains semantic-control mapping capabilities in advance through distillation and performs compression translation once every 30-50ms during runtime.
[0118] Specifically, the local model transforms the high-dimensional description in the semantic tensor into a parameter form that the controller can directly receive, including: target joint angle sequence, trajectory constraint window, impedance adjustment coefficient, motion termination condition, etc. This compression mapping process is achieved through structure-preserving distillation training, ensuring minimal information loss of semantic intent during translation. This module supports task decomposition, state memory, and context association processing, and can be used to handle multi-turn or compound instructions.
[0119] 3. Real-time generation of control parameters and execution of actions
[0120] The control parameters of the distillation output are input to the humanoid robot controller in real time. The controller can map the desired parameter trajectory into a high-frequency drive signal based on the model predictive control (MPC) framework, and make corrections and interpolations according to the current body state (sensor data) to ensure the smoothness and responsiveness of continuous control commands.
[0121] The system execution layer combines information such as the robot's current joint state, external contact forces, and IMU attitude data to compensate for errors during execution. If the task target is a dynamic object or an interactive operation (such as handing over or supporting), the controller will adjust the contact strategy and gait rhythm based on real-time perception during movement to achieve dynamic control stability under continuous semantics.
[0122] 4. State-aware feedback and semantic consistency detection
[0123] During the robot's task execution, the system continuously collects execution feedback information, including dimensions such as position error, task completion rate, control latency, and energy consumption. The system constructs a semantic-execution consistency discriminant function to determine whether the current control trajectory is consistent with the semantic intent.
[0124] Once semantic drift or control deviation is detected (such as actions deviating from the original task objective), the system triggers a semantic rollback mechanism to reconstruct the semantic tensor and initiate a new distillation process. This detection mechanism uses two methods—time window difference modeling and semantic reprojection of the execution trajectory—to ensure continuity and fault tolerance in the control process.
[0125] 5. Adaptive Adjustment and Semantic Reconstruction of Distillation Model
[0126] Based on the feedback above, the system fine-tunes the local lightweight model to optimize semantic mapping efficiency. This process uses incremental training with local data, weight fine-tuning caching, and drift recovery strategies to dynamically correct the model structure without affecting control stability. Furthermore, significant semantic drift events can be synchronized to the large cloud model via a relay mechanism, driving the evolution of long-term semantic paradigms.
[0127] During continuous task interactions, the system automatically constructs a database corresponding to semantic memory and control results, forming a task-behavior mapping experience base, which provides a foundation for the continuous evolution of the distillation model and adaptation to new scenarios. This dynamic collaborative mechanism of "optimizing while executing" is the key innovation of this invention, distinguishing it from traditional controller invocation strategies.
[0128] The following section provides a detailed introduction and explanation of the solution in this invention, using a specific application scenario of a semantic task-driven object retrieval and delivery control process on a humanoid robot platform:
[0129] 1. Experimental Environment
[0130] In this embodiment, the humanoid robot platform used is a bipedal robot with 30 degrees of freedom, equipped with a complete upper limb grasping module and a multimodal perception system including vision, touch, and IMU. The robot's local controller runs on an embedded chip with a quad-core ARM CPU and a 1T NPU, supporting local model deployment and high-frequency closed-loop control (control frequency up to 50Hz). The robot's operating system is a lightweight RTOS system, supporting parallel operation of task scheduling and the model inference engine.
[0131] The large language model is deployed on an edge server within a local area network, using the Qwen2-7B model for lightweight inference, and combined with a context memory caching module to complete the structured parsing and intent generation of natural language task instructions. The large model and the robot's local system transmit semantic tensors via a WebSocket channel, with an average latency controlled within 40ms.
[0132] The lightweight distillation model is deployed on the robot, using LoRA+TinyTransformer as its backbone architecture. The model's parameter count is kept below 5M, supporting millisecond-level response on the embedded control chip. Training data sources include a pre-built semantic-action alignment dataset and task reprojection feedback samples from the robot's operation. The distillation model employs a sliding fine-tuning strategy for online incremental learning, ensuring the system possesses a certain degree of adaptability.
[0133] The test environment was designed with a physical space including a workbench, a shelf, and a user operation area. The robot was required to complete the interactive action of "retrieving an object from the table and delivering it to the user" based on the user's natural language commands. During the experiment, metrics such as semantic parsing time, control response latency, execution accuracy, and action completion rate were recorded to evaluate the system's stability and practicality in a real-world environment.
[0134] 2. Experimental Procedure
[0135] Step 1: Natural Language Task Input and Semantic Parsing
[0136] First, the user issues natural language commands to the humanoid robot via a voice interface, such as "Please hand me the red cup on the table." The speech recognition system transcribes the speech into text input and sends the text to a large language model server deployed on edge computing nodes. The large language model uses contextual understanding and command deconstruction mechanisms to analyze the semantic content.
[0137] The language model identifies the core action "handing over," the target object "red cup," and the spatial constraint "on the table" in the task. Based on the environment model, it performs reasoning to confirm the uniqueness of the "red cup" and the user's relative position in the current scene. At the same time, the model determines whether multi-step reasoning is involved (such as whether movement or grasping is required) and generates structured action flows for these reasoning chains.
[0138] Finally, the system generates a multi-field "semantic intent tensor" which includes information such as object description, expected path logic, grasping action priority, and interaction direction (user / location). This tensor is then sent to the robot's local distillation model module via a low-latency interface, providing a precise semantic input foundation for subsequent motion control.
[0139] Step 2: Semantic Tensor Compression and Control Parameter Generation
[0140] After receiving the semantic intent tensor, the locally deployed distillation model first decomposes its fields to extract core control factors (such as target location, action category, path constraints, etc.). Then, the model performs a non-linear mapping from semantics to control parameters and outputs a series of control instruction templates.
[0141] These control parameters include, but are not limited to, the three-dimensional coordinates of the target grasping position, the parameterized form of the desired motion curve, path switching points (such as walking → grasping → turning → delivery), and tension and impedance adjustment strategies. Simultaneously, the system performs secondary filtering of the parameters based on the robot's body configuration and dynamic constraints to ensure that the control curve does not violate mechanical or posture stability.
[0142] The distillation model runs on a low-computing-power chip, with inference latency controlled within 20ms, and possesses a certain degree of anti-interference capability. If redundant or conflicting information exists in the intent tensor, the model will perform fuzzy merging and simplified reconstruction based on historical task execution experience to ensure the accuracy and real-time performance of parameter output.
[0143] Step 3: Robot Action Execution and Online Feedback
[0144] After receiving complete control parameters, the controller immediately initiates the motion execution process. The humanoid robot first adjusts its posture to face the target tabletop, uses the visual recognition module to pinpoint the exact location of the red cup, and constructs a grasping path based on its spatial coordinates. During path planning, the system automatically avoids obstacles and performs dynamic posture balance adjustments.
[0145] Once the arm reaches the designated position, the robot performs a precise gripping operation, then smoothly turns and moves towards the user. The controller continuously monitors gait stability, center of gravity drift, and torso torque to prevent falls or slips due to rapid movements. Upon approaching the user, the robot extends its arm to the designated delivery area and slowly releases the item.
[0146] Throughout the entire process, the system collects signals such as robot posture data, execution error, stress changes, and gripping feedback, and inputs these signals into the state perception module for processing in real time. If target grasping failure or execution path deviation is detected, the control flow will pause and trigger a semantic reconstruction mechanism to prevent the task from going out of control.
[0147] Step 4: State-aware feedback and semantic consistency detection
[0148] Various state data generated by the execution layer are sent to the local feedback module, which compares the current task execution path with the preset logic of the semantic intent to determine consistency. For example, if the semantic goal is "to hand it to the user," but the robot is detected not approaching the user's coordinate area, the system will determine that there is a task deviation.
[0149] The consistency detection mechanism combines trajectory comparison, timing deviation analysis, and feedback confidence calculation to determine the degree of execution deviation. If the deviation is within the allowable range, the controller will autonomously adjust the attitude parameters for micro-correction; if it exceeds the set threshold, the system will terminate the current control and restart the distillation process, triggering a new round of parameter generation.
[0150] In this embodiment of the invention, the degree of execution deviation mainly includes:
[0151] Timing deviation analysis: If a task is preset to be completed within 15 seconds, and the robot executes it beyond that time limit, it is judged as a timing deviation.
[0152] Trajectory comparison: If the task requires the cup to be delivered to the user's location [0.8, -0.3, 1.1], and the final position deviation of the end effector exceeds the set threshold (e.g., 10cm), it is determined to be trajectory deviation;
[0153] Feedback confidence level: refers to the degree of confidence the perception module has in the task execution status, for example:
[0154] 1) The vision module has a low probability of successfully recognizing the delivery action (<80%);
[0155] 2) The force sensor did not detect the release of resistance during the "lowering" process;
[0156] 3) The user's hand position does not overlap with the delivery area.
[0157] If the above deviation exceeds the set threshold, the system will trigger a control interruption and semantic reconstruction to ensure accurate task execution and the stability of the control system.
[0158] This dynamic repair mechanism based on semantic-execution consistency ensures that the system has high fault tolerance and task continuity in changing scenarios, and is especially suitable for application environments such as humanoid robots with complex posture interactions and physical feedback delays.
[0159] Step 5: Adaptive Adjustment and Semantic Reconstruction of the Distillation Model
[0160] After receiving feedback from the consistency detection module, the local distillation module analyzes and backtracks the labels of failed samples, automatically extracting anomaly triggers such as semantic understanding errors, control response errors, or perception errors. Based on these labels, the system performs fine-tuning training and updates the distillation mapping strategy.
[0161] Furthermore, if systematic deviations occur in consecutive tasks, the system will automatically synchronize these "typical deviation scenarios" to the cloud, implement semantic paradigm adjustment suggestions through a large model, and update the semantic tensor generation method, thereby improving task understanding and intent deconstruction capabilities. The cloud-based return optimization strategy can be selectively deployed locally on the robot.
[0162] Ultimately, this process completes a closed-loop learning process of "natural language task → semantic intent generation → distillation and compression → action execution → feedback correction → experience accumulation", providing humanoid robots with highly robust, highly interpretable, and highly adaptable control path support.
[0163] This invention's embodiments combine scenario-based description semantic intent tensor generation:
[0164] I. Scene Setting:
[0165] Environmental information:
[0166] The robot is in a standard home environment, with a coffee table 1.2 meters in front of it and a red mug on it.
[0167] The user stands about 0.8 meters to the right front of the robot, facing the robot, with their hands in a space 1.1 meters above the ground, where a delivery contact area has been reserved.
[0168] Other non-target items exist in the environment, such as a remote control and a kettle.
[0169] Voice command: The user issues a natural language command through the voice interface: "Hand me the cup."
[0170] Perception system recognition results:
[0171] Cup color: Red.
[0172] Cup position: Center coordinates of coffee table [x=1.2,y=0.0,z=0.4] (relative to robot's chest).
[0173] User interaction point location: [x=0.8,y=-0.3,z=1.1] (low hand raised to the right front).
[0174] II. Example of semantic intent tensor generation:
[0175]
[0176]
[0177] This invention provides a semantic intent tensor from natural language task input and semantic parsing. This semantic intent tensor is used to clarify the target task and is the semantic basis for re-distillation. The main fields are shown in Table 1.
[0178] Table 1. Main fields of the semantic intent tensor
[0179]
[0180]
[0181] This invention provides robot current state perception data to guide distillation model adjustments. Referring to Table 2, various sensor data from the robot are used to provide feedback on actual execution deviations and assist the distillation model in real-time adjustments.
[0182] Table 2. Data from various sensors of the robot
[0183]
[0184] Compared to traditional humanoid robot control methods, this invention significantly improves semantic understanding, action generation efficiency, feedback adaptability, and cross-scenario generalization capabilities. By introducing a large language model to generate structured semantic tensors, it solves problems such as significant discrepancies between natural language and control signals, and ambiguous task objectives, enabling the robot to understand user commands in a more human-like manner. Simultaneously, by leveraging a local distillation model to achieve rapid parsing of control intentions, it effectively avoids the high latency and heavy computational burden associated with directly using large models, ensuring real-time and stable control responses.
[0185] The proposed "semantic intent tensor + instant distillation mapping" structure breaks away from the rigid orchestration patterns of traditional action templates or finite state machines, enabling humanoid robots to flexibly complete combined actions and interactive tasks in unknown environments, and providing semantic-level reverse correction for failed executions. Furthermore, through a feedback-driven distillation model adaptive training mechanism, the system possesses continuous evolution capabilities, iteratively optimizing the semantic-to-control mapping link under different users and scenarios, achieving dynamic enhancement through "learning while executing."
[0186] This invention is highly versatile and deployable, applicable to a variety of humanoid robot platforms with autonomous control requirements. It has broad application prospects, especially in service robots, rehabilitation assistance, educational companionship, and hazardous job replacement, providing a new solution for promoting semantic-driven human-machine collaboration to a higher level of intelligence.
[0187] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.
[0188] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0189] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:
[0190] The processor 301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.
[0191] The memory 302 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302 and is called and executed by the processor 301 using the methods described in the embodiments of this application.
[0192] Input / output interface 303 is used to implement information input and output;
[0193] The communication interface 304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0194] Bus 305 transmits information between various components of the device (e.g., processor 301, memory 302, input / output interface 303, and communication interface 304);
[0195] The processor 301, memory 302, input / output interface 303, and communication interface 304 are connected to each other within the device via bus 305.
[0196] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0197] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0198] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0199] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0200] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0201] The semantically constrained real-time distillation humanoid robot control method, system, electronic device, storage medium, and program product based on a large language model provided in this application embodiment enable a high-degree-of-freedom humanoid robot to reliably execute complex operational tasks given by humans in natural language through a closed-loop learning cycle of "perception-decision-control-feedback". The robot is no longer limited to performing preset, fixed tasks (such as tightening screws). For unknown, combinatorial new instructions (such as "first bring the cup, then mop the floor"), the system can attempt to complete them through an understanding-distillation-execution process, and continuously learn and optimize through a feedback mechanism, possessing adaptability to unknown scenarios and the potential for continuous evolution.
[0202] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0203] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0204] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0205] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0206] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0207] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0208] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.
[0209] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0210] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0211] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0212] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A semantically constrained real-time distillation humanoid robot control method based on a large language model, characterized in that, The method includes the following steps: Obtain user natural language commands; The user's natural language instructions are semantically parsed using a large language model to obtain a structured semantic intent tensor; The semantic intent tensor is compressed and translated using a local lightweight model to obtain control parameters; The humanoid robot is controlled to complete the target action according to the control parameters, and the execution status feedback information is collected. Based on the execution status feedback information, semantic offset detection is performed, and the local lightweight model is adjusted online to form a closed-loop control path; The step of semantically parsing the user's natural language instructions using a large language model to obtain a structured semantic intent tensor includes: The user's natural language instructions are reasoned through multiple rounds using a multi-round chained reasoning mechanism through a large language model. These instructions are decomposed into action logic units, and combined with environmental knowledge to generate a structured semantic intent tensor. The large language model is a pre-trained language model with instruction-following capabilities, used for structured parsing of action targets, object attributes, temporal logic, and environmental constraints. The fields of the semantic intent tensor include action type, object features, spatial constraints, action sequence, interaction direction, and security parameters. The fields of the semantic intent tensor are represented using vector or sparse coding methods. The process of compressing and translating the semantic intent tensor using a local lightweight model to obtain control parameters includes: Configure a local lightweight model; the local lightweight model includes a weighted distillation network; The semantic intent tensor is transmitted to the robot's local control terminal through compression encoding, and is parsed in real time by the local lightweight model. The semantic-control mapping capability is obtained through distillation. Compression and translation are performed once at a preset period during runtime to transform the high-dimensional description in the semantic intent tensor into the parameter form received by the controller, thus obtaining control parameters. The control parameters include the target joint angle sequence, trajectory constraint window, impedance adjustment coefficient, and motion termination condition.
2. The method according to claim 1, characterized in that, The process of controlling the humanoid robot to complete the target action according to the control parameters and collecting execution status feedback information includes: Based on the control parameters, and using a model predictive control framework, the desired parameter trajectory is mapped to a high-frequency driving signal. At the same time, the current sensor data of the humanoid robot is corrected and interpolated to drive the humanoid robot to complete the target action. During the humanoid robot's task execution, execution status feedback information is continuously collected; the execution status feedback information includes position error, task completion degree, control delay, and energy consumption.
3. The method according to claim 1, characterized in that, The step of performing semantic offset detection based on the execution status feedback information and adjusting the local lightweight model online to form a closed-loop control path includes: Based on the execution status feedback information, semantic offset detection is performed, and combined with trajectory comparison, temporal deviation analysis and feedback confidence calculation, execution deviation information is determined. Based on the execution deviation information, the failed samples are analyzed and their labels are traced back to extract the abnormal triggering factors; The local lightweight model is adjusted online using the aforementioned abnormal triggering factors to obtain an optimized strategy and form a closed-loop control path.
4. A semantically constrained real-time distillation humanoid robot control system based on a large language model, used to implement the method as described in any one of claims 1 to 3, characterized in that, The system includes a natural language task parsing module, a semantic intent encoding and distillation module, a control parameter generation and execution module, and a state feedback and dynamic optimization module. The natural language task parsing module is used to parse natural language instructions into structured semantic intent tensors based on a large language model; The semantic intent encoding and distillation module is used to compress and translate the semantic intent tensor based on a local lightweight model to obtain control parameters; The control parameter generation and execution module is used to convert the control parameters into continuous control signals and drive the humanoid robot to move, while collecting execution status feedback information. The state feedback and dynamic tuning module is used to perform semantic offset detection based on the execution state feedback information and to adjust the local lightweight model online.
5. The system according to claim 4, characterized in that, The natural language task parsing module is deployed in the cloud or at the edge node; the semantic intent encoding and distillation module is deployed on the robot's local control board; and the control parameter generation and execution module is deployed in the robot's execution layer.
6. The system according to claim 4, characterized in that, The semantic intent encoding and distillation module is equipped with a semantic-dynamic nested contrastive loss function, and is trained and optimized through an online weight adjustment mechanism.
7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 3.
8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 3.