A natural language control method for humanoid robots

Abstract

The present invention proposes a natural language control method for a humanoid robot. By constructing a simple skill library and a complex skill library for the robot, the present invention effectively reduces the dependence on a large amount of operational data in specific fields. The present invention utilizes a large language model as the base of a task planner, giving full play to its capabilities of situational learning, semantic understanding, and common sense reasoning. By introducing a visual model in the field of computer vision, the present invention can obtain high-level semantic information and low-level spatial information of the environment, and construct an accurate scene graph as a representation of the environmental state. The voice interaction interface constructed by the present invention improves the naturalness of human-computer interaction, allowing users to communicate and input commands with the robot more conveniently, effectively improving the user experience, and is conducive to the promotion of humanoid robots in practical application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a natural language control method for a humanoid robot. Background Art

[0002] Humanoid robots are robots with a humanoid appearance that mimic human functions and intelligence. They possess a head, torso, and limbs, making them more versatile and adaptable to human work and living environments. Humanoid robots not only enhance the naturalness and efficiency of human-robot interaction, enabling robots to collaborate with humans in a more humane manner, but also assist humans in completing dangerous, arduous, or delicate tasks in industrial production, home services, healthcare, and other fields, significantly improving work efficiency and safety. In particular, in addressing the global aging population, humanoid robots have the potential to provide companionship and daily support, alleviating the workload of caregivers.

[0003] The rapid development of artificial intelligence, particularly deep learning and reinforcement learning, has significantly boosted research in humanoid robots. Deep learning enables robots to learn complex patterns and behaviors from vast amounts of data, enabling more intelligent perception and decision-making. Reinforcement learning helps robots optimize their action strategies through continuous experimentation and feedback, enabling them to more effectively complete tasks in their environments. Furthermore, advances in natural language processing and computer vision technologies are enabling humanoid robots to interact with humans more naturally and efficiently.

[0004] Current research on humanoid robot manipulation primarily focuses on end-to-end control methods, a control approach that directly generates output data (e.g., robot motions) based on raw input data (e.g., sensor information, user commands, etc.). This approach significantly reduces programming and code, avoiding the complex intermediate steps and manual feature engineering required in traditional methods, enabling humanoid robots to demonstrate greater flexibility and adaptability across a variety of tasks. End-to-end control methods rely on large amounts of data for learning and training. However, in practical applications, obtaining sufficient training data of sufficient quantity and quality is often a daunting task. This is especially true given the high cost of collecting robot manipulation data, which further exacerbates the scarcity and imbalance of training data, leading to inadequate model training or weak generalization, thus compromising robot performance and stability. Furthermore, to improve the multi-task execution capabilities of manipulation strategies, end-to-end methods require the construction of large-scale neural network models. Training these models requires significant computational resources and time, and the high computational effort also limits the control rate of the system. Therefore, how to reduce dependence on large amounts of operational data, improve the efficiency of model training and reasoning, while maintaining the flexibility and efficiency of humanoid robot operations so that it can adapt to the complex needs of actual application scenarios has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to solve the problems in the prior art to meet the needs of actual situations. A natural language control method for a humanoid robot is proposed to improve the problems of the existing humanoid robot operation strategy being highly dependent on specific field data and having low efficiency in model training and reasoning.

[0006] The present invention is achieved through the following technical solutions. The present invention proposes a natural language control method for a humanoid robot, which includes the following steps:

[0007] S1. Use robotics technology to build a simple robot skill library, encapsulating it into an API that can be called by a program to complete simple manipulation tasks with a wide variety of objects and low perception accuracy requirements;

[0008] S2. Use end-to-end robot manipulation strategies fine-tuned for specific tasks to build a complex robot skill library to complete complex manipulation tasks that are difficult to decompose into simple skill sequences;

[0009] S3. Use the visual model to obtain high-level semantic information and low-level spatial information of the environment and construct a scene graph as the representation of the environment state;

[0010] S4. Use voice services to build voice interaction interfaces to improve the naturalness of human-computer interaction;

[0011] S5. Based on a large language model, a task planner is constructed using the scenario learning, semantic understanding, and common sense reasoning capabilities of the large language model. The action sequence is planned based on natural language instructions and environmental state representations. The action instructions in the robot's simple skill library and complex skill library are then called to control the robot to perform actions.

[0012] Furthermore, in step S1,

[0013] S11. Build a mobile navigation API: This provides map-based path planning, allowing the robot to automatically calculate and move along the optimal path based on its current and target locations. It also supports obstacle avoidance, using integrated sensors to monitor the surrounding environment in real time and avoid collisions. Users can set multiple navigation points to implement complex movement sequences.

[0014] S12. Build a robotic arm motion planning API: This provides inverse kinematics solving capabilities to calculate the angles of each robotic arm joint based on the target position and posture of the end effector. It also supports smooth motion trajectory planning to ensure that the robotic arm does not generate excessive acceleration or jitter during movement. It also allows users to customize motion modes to suit different operational requirements.

[0015] S13. Build an object grasping API: Provide grasping solutions that adapt to objects of various shapes, sizes, and materials; provide a variety of configurable grasping strategies, including precise grasping based on friction control and stable grasping based on inclusiveness, to adapt to the characteristics of different objects; the API allows users to set grasping parameters and provides a closed-loop feedback mechanism to cope with grasping tasks in dynamic environments.

[0016] Furthermore, in step S2,

[0017] S21. Establish a pre-training dataset and a fine-tuning dataset. The pre-training dataset includes open-source robot multi-task operation data and a synthetic dataset created by writing task supervision code in a simulation environment using a large language model. The fine-tuning dataset is collected directly on the personalized robot and task scenario.

[0018] S22. Build a robot operation policy network based on a multimodal pre-trained model and train the policy network using the loss function of the diffusion model.

[0019] S23. Collect robot operation data for various mission scenarios in real environments through teleoperation or data gloves, fine-tune the policy network, and deploy it on the robot;

[0020] S24. For reinforcement learning of complex manipulation task strategies for dexterous hands, a reward function design based on a large language model and a domain randomization method are used to improve the effect of virtual-to-real migration and reduce manual workload.

[0021] Furthermore, in step S22, the diffusion strategy and 3D scene representation are combined to achieve generalization of robot operations; a 3D feature point cloud representation of the scene is obtained by aggregating depth maps and 2D image features from single or multiple camera perspectives, and a conditional diffusion model is used to learn the action distribution conditioned on the robot and environment states, and the 3D pose trajectory of the robot end effector is iteratively denoised given language instructions;

[0022] The multimodal pre-trained model CLIP is used as a 2D image encoder and a natural speech command encoder. The farthest point sampling algorithm is used to downsample the 3D feature point cloud of the scene. The FiLM module is used to fuse the denoising time step and the robot body information to adjust the network activation value. Cross-attention is used to fuse language information and visual features, and the self-attention layer predicts the error of the end-effector pose trajectory under the current environment state and time step.

[0023] Furthermore, in step S24, in order to achieve the migration of operation strategies from the virtual environment to the real environment, a domain randomization method needs to be adopted: multiple sets of physical parameters are sampled in the simulation environment to learn a more stable strategy; first, the physical parameters are independently sampled in the simulation environment to train the operation strategy, and the feasible parameter value range is roughly determined. Then, a large language model is used to infer reasonable parameter combinations based on the operation tasks and physical common sense, and multiple candidate operation strategies are trained again. Finally, they are deployed and tested in the real environment.

[0024] Furthermore, in step S3, a method is adopted in which the robot autonomously explores the local environment through a large language model and a visual model to construct a scene graph that captures the structure of the environment; in order to obtain multiple attributes of a single entity in the scene graph and the spatial relationship between different entities, it is regarded as a visual query problem, and a perception API is constructed using multiple pre-trained models, including an object recognition model, an open vocabulary target detection model, a semantic segmentation model, a pose estimation model, and a large multimodal model; and an executable API call code is generated based on the relational query statement using the large language model.

[0025] Furthermore, in step S4, a large language model is used to manage the human-computer dialogue, infer the user's intention and parse the operation task instructions issued by the user, and summarize the human-computer dialogue context to update the task planning database.

[0026] Furthermore, in step S5,

[0027] S51 Hierarchical task planning based on thought chain prompts;

[0028] S52 builds a personalized mission planning database;

[0029] S53 integrates external text feedback.

[0030] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the natural language control method for a humanoid robot when executing the computer program.

[0031] The present invention also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the natural language manipulation method for a humanoid robot.

[0032] Beneficial effects of the present invention:

[0033] 1. This invention effectively reduces reliance on large amounts of domain-specific operational data by constructing simple and complex robot skill libraries. The simple skill library encapsulates basic robot operation APIs, enabling it to handle a variety of common tasks that do not require high sensory accuracy. The complex skill library, on the other hand, enables efficient processing of complex operational tasks through task-specific fine-tuning. The combination of these two significantly enhances the operational flexibility and adaptability of humanoid robots.

[0034] 2. This invention utilizes a large language model as the foundation of the task planner, fully leveraging its capabilities for contextual learning, semantic understanding, and commonsense reasoning. This enables the robot to intelligently plan reasonable action sequences based on natural language instructions and environmental state representations, and execute the corresponding action instructions from the corresponding skill library, thereby improving the efficiency of model training and reasoning and reducing the consumption of computing resources.

[0035] 3. By incorporating visual models from computer vision, this invention can capture both high-level semantic and low-level spatial information about the environment, constructing a precise scene graph as a representation of the environment's state. This not only enhances the robot's environmental perception but also provides it with a richer basis for decision-making, enabling it to perform operations with greater precision and efficiency.

[0036] 4. The voice interaction interface constructed by the present invention improves the naturalness of human-computer interaction, allowing users to communicate and input commands with the robot more conveniently, effectively improving the user experience and facilitating the promotion of humanoid robots in practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 1 is a flow chart of a method for natural language control of a humanoid robot according to an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a process for building a robot complex skill library according to an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a process for constructing a scene graph environment representation according to an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of a process for constructing a voice interaction interface according to an embodiment of the present invention;

[0042] Figure 5 A flowchart of a task planner based on a large language model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] Combine Figure 1-Figure 5 The present invention provides a natural language manipulation method for a humanoid robot, the method comprising the following steps:

[0045] S1. Build a simple robot skill library using robotic technologies such as mobile navigation, robotic arm motion planning, and object grasping. This library is encapsulated as an API that can be called by a program to complete simple manipulation tasks that have a wide variety of objects and require low perception accuracy.

[0046] In step S1,

[0047] S11. Build a mobile navigation API: Provide map-based path planning capabilities, allowing the robot to automatically calculate and move along the optimal path based on its current and target locations; support obstacle avoidance, using integrated sensors (such as lidar and ultrasonic sensors) to detect the surrounding environment in real time and avoid collisions; and allow users to set multiple navigation points to implement complex movement sequences.

[0048] S12. Build a robotic arm motion planning API: This provides inverse kinematics solving capabilities to calculate the angles of each robotic arm joint based on the target position and posture of the end effector. It supports smooth motion trajectory planning to ensure that the robotic arm does not generate excessive acceleration or jitter during movement. It also allows users to customize motion modes, such as linear interpolation and circular interpolation, to accommodate different operational requirements.

[0049] S13. Build an object grasping API: Provide grasping solutions that adapt to objects of various shapes, sizes, and materials; provide a variety of configurable grasping strategies, including precise grasping based on friction control and stable grasping based on inclusiveness, to adapt to the characteristics of different objects; the API allows users to set grasping parameters (such as clamping force, clamping speed, etc.) and provides a closed-loop feedback mechanism to cope with grasping tasks in dynamic environments.

[0050] S2. Use end-to-end robotic manipulation strategies (imitation learning and reinforcement learning of manipulation skills) with task-specific fine-tuning to build a complex robotic skill library for completing complex manipulation tasks that are difficult to decompose into simple skill sequences.

[0051] In step S2,

[0052] S21. Establish a pre-training dataset and a fine-tuning dataset. The pre-training dataset includes open-source robot multi-task operation data and a synthetic dataset created by writing task supervision code in a simulation environment using a large language model. The fine-tuning dataset is collected directly on the personalized robot and task scenario.

[0053] Large, high-capacity models trained on diverse datasets have demonstrated significant success in effectively processing downstream applications. To increase the scale and diversity of robotic manipulation task data while minimizing data collection costs, imitation learning datasets are divided into pre-training datasets and fine-tuning datasets. Pre-training datasets include existing open-source data (multi-task manipulation data collected on a variety of robots and real-world environments) and simulated environment data, which are large in scale but less targeted. Fine-tuning datasets are collected directly on personalized robots and task scenarios, which are smaller in scale but higher in quality.

[0054] When training the policy network, we first use pre-training data containing multiple tasks and scenarios for large-scale pre-training, so that the policy network can learn the general representation and knowledge of various tasks and scenarios, and improve its generalization to new tasks and scenarios; then use fine-tuning data for specific tasks and scenarios to fine-tune the policy network, so that the policy network can further learn and optimize the execution strategy of specific tasks based on the general knowledge it has mastered.

[0055] For simulated environment data, we leverage the code generation capabilities of large language models to automatically generate rich simulated environments and expert demonstrations, improving the scene diversity (object instances and poses) and task diversity of synthetic data. We primarily use two data generation methods: goal-oriented generation, where we provide a target task to a large language model and it proposes a task curriculum to solve the target task; and exploratory generation, where the large language model is guided by previous tasks and iteratively proposes novel tasks that help solve more complex tasks.

[0056] S22. Build a robot operation policy network based on a multimodal pre-trained model and train the policy network using the loss function of the diffusion model.

[0057] Many robotic manipulation tasks are inherently multimodal; at any given moment during a task, multiple actions may produce the optimal behavior. Human demonstrations often include multiple ways of completing a task. Rather than representing a policy as a deterministic function, it's more natural to consider policy learning as a probability distribution learning problem over continuous random variables. Reconstructing 3D scenes from 2D images can achieve better generalization because it can adapt to observations from new camera perspectives at test time.

[0058] Combining diffusion strategies with 3D scene representations enables generalization of robotic manipulation. Depth maps and 2D image features are aggregated from single or multiple camera viewpoints to obtain a 3D feature point cloud representation of the scene. A conditional diffusion model is used to learn an action distribution conditioned on the robot and environment states. Given language commands, the 3D pose trajectory of the robot end-effector is iteratively denoised.

[0059] The multimodal pre-trained model CLIP (Contrastive Language-Image Pre-training) serves as both a 2D image encoder and a natural speech command encoder. The farthest point sampling algorithm is used to downsample the scene's 3D feature point cloud. The FiLM (Feature-wise Linear Modulation) module fuses denoising time steps and robotic arm body information to adjust network activations. Cross-attention is used to fuse language information with visual features, and the self-attention layer predicts the error in the end-effector pose trajectory given the current environment state and time step.

[0060] S23. Collect robot operation data for various mission scenarios in real environments through teleoperation or data gloves, fine-tune the policy network, and deploy it on the robot;

[0061] Teleoperation technology allows experts to precisely control robots from a distance, completing complex tasks. By recording the expert's manipulation trajectory, force, speed, and decision-making process, high-quality operational data can be obtained. Data gloves can capture the operator's hand movements and posture, translating this detailed motion information into commands that the robot can recognize. Data from data gloves is particularly important in scenarios requiring high-precision hand manipulation.

[0062] S24. For reinforcement learning of complex manipulation task strategies for dexterous hands, a reward function design based on a large language model and a domain randomization method are used to improve the effect of virtual-to-real migration and reduce manual workload.

[0063] Reinforcement learning is widely used for policy learning in complex robotic manipulation tasks due to its advantages, such as learning from interactions without requiring large amounts of labeled data, fully exploring the action space, and high stability. Natural language commands tend to describe the final effect of an action rather than the specific details of the execution process, which is consistent with the role of reward functions in encoding semantic information about the target state. To reduce the workload of manually designing reward functions, this paper leverages the capabilities of large language models for zero-shot generation, code writing, and contextual refinement. Reward function code is generated based on tasks expressed in natural language. The code is then trained using the standard PPO algorithm in a simulation environment. The reward function's effectiveness is then fed back to the large language model, thereby improving its quality.

[0064] To transfer operational strategies from virtual environments to real-world environments, a domain randomization approach is employed: This involves sampling multiple sets of physical parameters in a simulated environment to learn a more stable strategy. First, the operational strategy is trained by independently sampling physical parameters in the simulated environment to roughly determine the range of feasible parameter values. A large language model is then used to infer reasonable parameter combinations based on the operational task and common physical knowledge. Multiple candidate operational strategies are then trained again, and finally deployed and tested in a real-world environment.

[0065] S3. Use the visual model to obtain high-level semantic information and low-level spatial information of the environment and construct a scene graph as the representation of the environment state;

[0066] Robots need to explore their surroundings to adapt to and handle tasks in unknown environments. Previous work has mostly employed methods that construct scene graphs of the environment. However, these approaches often assume that the environment is static and fully observable, ignoring areas that require active interaction and exploration, thus limiting the robot's ability to handle more complex tasks. To address this issue, a method is proposed in which robots autonomously explore the local environment using large language and vision models to construct a scene graph that captures the environment's structure. A scene graph incorporates both low-level information, such as geometry and semantics, and high-level information, such as the spatial relationships between different entities in the scene and the order in which objects are explored.

[0067] In order to obtain multiple attributes of a single entity in the scene graph and the spatial relationships between different entities, it is regarded as a visual query problem. A variety of pre-trained models are used to build a perception API, including object recognition models, open vocabulary target detection models, semantic segmentation models, pose estimation models, and large multimodal models. A large language model is used to generate executable API call code based on relational query statements.

[0068] S4. Use voice services to build voice interaction interfaces to improve the naturalness of human-computer interaction;

[0069] Text to Speech (TTS) technology is used to synthesize text into audio. To improve speech recognition accuracy and conserve computing resources on mobile devices, voice wake-up and command word recognition tasks, which require relatively low recognition difficulty, are deployed offline on an embedded voice signal acquisition card to reduce response latency and CPU usage. For the more challenging voice dictation function, streaming voice dictation technology is used, enabling real-time upload of voice signal streams and return of text results via the WebSockets protocol, maximizing recognition accuracy and speed. To make voice interaction more natural and flexible, an Audio Activity Detection (VAD) model is used to determine in real time whether the user is speaking and to control the start and end of voice dictation.

[0070] To improve system reliability, a large language model is used to manage human-computer dialogue, infer user intentions and parse the operational task instructions issued by the user, while summarizing the human-computer dialogue context to update the task planning database.

[0071] S5. Based on a large language model, a task planner is constructed using the scenario learning, semantic understanding, and common sense reasoning capabilities of the large language model. The action sequence is planned based on natural language instructions and environmental state representations. The action instructions in the robot's simple skill library and complex skill library are then called to control the robot to perform actions.

[0072] S51 Hierarchical task planning based on thought chain prompts

[0073] Manipulation tasks exhibit a highly hierarchical structure. To efficiently plan across a vast state space and skill set, the planning process is divided into two stages: high-level and low-level. During high-level planning, a large language model autoregressively infers the high-level actions required at the current moment based on the inference context. High-level actions can be either operational actions that change the state of an object or perceptual actions that query object attributes and update the action-scene graph. During low-level planning, for the high-level actions decomposed from the task plan, the large language model generates corresponding executable code based on the current fine-grained environment state and defined skill APIs. The generated code is checked for legality using an abstract syntax tree. If undefined function references appear in the code, the low-level planning module is recursively used to define function references based on known APIs. This not only resolves the references, but also automatically enriches the available skill API set.

[0074] S52 builds a personalized mission planning database

[0075] To parse open-domain natural language commands and dialogues and learn to adapt to user preferences while avoiding exceeding context window limits, a key-value database consisting of dialogue context and planning results is designed. A large language model is used to embed the current dialogue context, environment state, or execution result feedback into a vector space, and relevant information is retrieved from the database based on cosine similarity. The retrieved memories are passed as context examples to the large language model to generate more accurate and coherent responses. During deployment, the database is continuously updated, expanded, and enriched to fully leverage the contextual learning capabilities of the large language model to accomplish new tasks.

[0076] S53 Fusion External Text Feedback

[0077] In order to reliably perform tasks in dynamic environments and under external interference, the anti-interference and self-correction capabilities of large-scale language model planning are improved by utilizing external text feedback (such as user descriptions, successful detection modules, function call execution results, etc.).

[0078] Example

[0079] See Figures 1 to 5 In an embodiment of the present invention, a natural language control method for a humanoid robot includes the following contents:

[0080] 1. Use robotic technologies such as mobile navigation, robotic arm motion planning, and object grasping to build a simple robot skill library and encapsulate it into an API that can be called by the program.

[0081] Build a mobile navigation API using ORB-SLAM. This provides map-based path planning, allowing the robot to automatically calculate and move along the optimal path based on its current and target locations. It also supports obstacle avoidance, using integrated sensors (such as lidar and ultrasonic sensors) to monitor the surrounding environment in real time and avoid collisions. It also allows users to set multiple navigation points to implement complex movement sequences.

[0082] Use MoveIt! to build a robotic arm motion planning API. It provides inverse kinematics solutions, calculating the angles of each robotic arm joint based on the end effector's target position and posture. It supports smooth motion trajectory planning, ensuring the robotic arm does not experience excessive acceleration or jitter during movement. It also allows users to customize motion modes, such as linear and circular interpolation, to accommodate different operational needs.

[0083] Use GraspIt! to build an object grasping API. It provides grasping solutions for objects of various shapes, sizes, and materials. It offers a variety of configurable grasping strategies, including friction-controlled precision grasping and inclusion-based robust grasping, to adapt to different object characteristics. The API allows users to set grasping parameters (such as gripping force and gripping speed) and provides a closed-loop feedback mechanism to handle grasping tasks in dynamic environments.

[0084] 2. Build a complex robot skill library using end-to-end robot operation strategies fine-tuned for specific tasks.

[0085] Imitation learning datasets are divided into pre-training datasets and fine-tuning datasets. The pre-training datasets include existing open source data (Open-X-Embodiment) and simulation environment data (RLbench). The data scale is large but the targeting is low. The fine-tuning datasets are collected directly on personalized robots and task scenarios. The data scale is smaller but the data quality is higher.

[0086] When training the policy network, we first use pre-training data containing multiple tasks and scenarios for large-scale pre-training, so that the policy network can learn the general representation and knowledge of various tasks and scenarios, and improve its generalization to new tasks and scenarios; then use fine-tuning data for specific tasks and scenarios to fine-tune the policy network, so that the policy network can further learn and optimize the execution strategy of specific tasks based on the general knowledge it has mastered.

[0087] Combining diffusion strategies with 3D scene representations enables generalization of robotic manipulation. Depth maps and 2D image features are aggregated from single or multiple camera viewpoints to obtain a 3D feature point cloud representation of the scene. A conditional diffusion model is used to learn an action distribution conditioned on the robot and environment states. Given language commands, the 3D pose trajectory of the robot end-effector is iteratively denoised.

[0088] The multimodal pre-trained model CLIP (Contrastive Language-Image Pre-training) serves as both a 2D image encoder and a natural speech command encoder. The farthest point sampling algorithm is used to downsample the scene's 3D feature point cloud. The FiLM (Feature-wise Linear Modulation) module fuses denoising time steps and robotic arm body information to adjust network activations. Cross-attention is used to fuse language information with visual features, and the self-attention layer predicts the error in the end-effector pose trajectory given the current environment state and time step.

[0089] 3. Use visual models in the field of computer vision to obtain high-level semantic information and low-level spatial information of the environment, and construct a scene graph as a representation of the environment state.

[0090] In order to obtain multiple attributes of a single entity in the scene graph and the spatial relationships between different entities, it is regarded as a visual query problem. A perception API is built using multiple pre-trained models, including an object recognition model (DETR), an open vocabulary target detection model (OWL-ViT), a semantic segmentation model (Segment-Anything), a pose estimation model (FoundationPose), and a large multimodal model (GPT-4V). A large language model (GPT-4-turbo) is used to generate executable API call code based on relational query statements.

[0091] 4. Use existing voice services to build voice interaction interfaces to improve the naturalness of human-computer interaction.

[0092] To improve speech recognition accuracy and save computing resources on mobile devices, the Picovoice service is used for voice wake-up and command word recognition tasks, which are less difficult to recognize. For the voice dictation function, which is more difficult to process, the streaming voice dictation cloud service provided by iFlytek is used. The websockets protocol is used to achieve real-time upload of voice signal streams and real-time return of text results, maximizing recognition accuracy and speed.

[0093] Use ALSA to manage audio devices, use PulseAudio to implement audio sharing, use Pyaudio to read and write audio streams to audio devices, adopt non-blocking callback mode audio stream control, and store audio data in global variables by defining callback functions.

[0094] To make voice interactions more natural and flexible, Silero VAD was deployed on the robot. Silero VAD is a voice activity detection model based on a deep neural network. It was trained on a large corpus of over 100 languages. It performs well on audio from different domains, can handle various background noise levels, and has excellent detection accuracy and recognition speed.

[0095] IO concurrent processing is implemented based on threads and coroutines. Coroutines are used to control asynchronous websockets communication, and threads are used to manage various modules of voice interaction.

[0096] 5. Based on a large language model (GPT-4-turbo), the task planner is built using the contextual learning, semantic understanding, and common sense reasoning capabilities of the large language model. The action sequence is planned based on natural language instructions and environmental state representations, and then the action instructions in the robot's simple skill library and complex skill library are called to control the robot to perform the action.

[0097] Manipulation tasks exhibit a highly hierarchical structure. To efficiently plan across a vast state space and skill set, the planning process is divided into two stages: high-level and low-level. During high-level planning, a large language model autoregressively infers the high-level actions required at the current moment based on the inference context. High-level actions can be either operational actions that change the state of an object or perceptual actions that query object attributes and update the action-scene graph. During low-level planning, for the high-level actions decomposed from the task plan, the large language model generates corresponding executable code based on the current fine-grained environment state and defined skill APIs. The generated code is checked for legality using an abstract syntax tree. If undefined function references appear in the code, the low-level planning module is recursively used to define function references based on known APIs. This not only resolves the references, but also automatically enriches the available skill API set.

[0098] To parse open-domain natural language commands and dialogues and learn to adapt to user preferences while avoiding exceeding context window limits, a key-value database consisting of dialogue context and planning results is designed. A large language model is used to embed the current dialogue context, environment state, or execution result feedback into a vector space, and relevant information is retrieved from the database based on cosine similarity. The retrieved memories are passed as context examples to the large language model to generate more accurate and coherent responses. During deployment, the database is continuously updated, expanded, and enriched to fully leverage the contextual learning capabilities of the large language model to accomplish new tasks.

[0099] In order to reliably perform tasks in dynamic environments and under external interference, the anti-interference and self-correction capabilities of large-scale language model planning are improved by utilizing external text feedback (such as user descriptions, successful detection modules, function call execution results, etc.).

[0100] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the natural language control method for a humanoid robot when executing the computer program.

[0101] The present invention also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the natural language manipulation method for a humanoid robot.

[0102] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0103] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0104] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0105] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0106] The above is a detailed introduction to the natural language control method of a humanoid robot proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A natural language control method for a humanoid robot, characterized in that: The method comprises the following steps: S1. Use robotics technology to build a simple robot skill library, encapsulating it into an API that can be called by a program to complete simple manipulation tasks with a wide variety of objects and low perception accuracy requirements; S2. Use end-to-end robot manipulation strategies fine-tuned for specific tasks to build a complex robot skill library to complete complex manipulation tasks that are difficult to decompose into simple skill sequences; S3. Use the visual model to obtain high-level semantic information and low-level spatial information of the environment and construct a scene graph as the representation of the environment state; S4. Use voice services to build voice interaction interfaces to improve the naturalness of human-computer interaction; S5. Based on a large language model, a task planner is constructed using the scenario learning, semantic understanding, and common sense reasoning capabilities of the large language model. The action sequence is planned based on natural language instructions and environmental state representations. The action instructions in the robot's simple skill library and complex skill library are then called to control the robot to perform actions.

2. The method according to claim 1, characterized in that In step S1, S11. Build a mobile navigation API: This provides map-based path planning, allowing the robot to automatically calculate and move along the optimal path based on its current and target locations. It also supports obstacle avoidance, using integrated sensors to monitor the surrounding environment in real time and avoid collisions. Users can set multiple navigation points to implement complex movement sequences. S12. Build a robotic arm motion planning API: This provides inverse kinematics solving capabilities to calculate the angles of each robotic arm joint based on the target position and posture of the end effector. It also supports smooth motion trajectory planning to ensure that the robotic arm does not generate excessive acceleration or jitter during movement. It also allows users to customize motion modes to suit different operational requirements. S13. Build an object grasping API: Provide grasping solutions that adapt to objects of various shapes, sizes, and materials; provide a variety of configurable grasping strategies, including precise grasping based on friction control and stable grasping based on inclusiveness, to adapt to the characteristics of different objects; the API allows users to set grasping parameters and provides a closed-loop feedback mechanism to cope with grasping tasks in dynamic environments.

3. The method according to claim 2, characterized in that In step S2, S21. Establish a pre-training dataset and a fine-tuning dataset. The pre-training dataset includes open-source robot multi-task operation data and a synthetic dataset created by writing task supervision code in a simulation environment using a large language model. The fine-tuning dataset is collected directly on the personalized robot and task scenario. S22. Build a robot operation policy network based on a multimodal pre-trained model and train the policy network using the loss function of the diffusion model. S23. Collect robot operation data for various mission scenarios in real environments through teleoperation or data gloves, fine-tune the policy network, and deploy it on the robot; S24. For reinforcement learning of complex manipulation task strategies for dexterous hands, a reward function design based on a large language model and a domain randomization method are used to improve the effect of virtual-to-real migration and reduce manual workload.

4. The method according to claim 3, characterized in that In step S22, the diffusion strategy and 3D scene representation are combined to achieve generalization of robot operations. A 3D feature point cloud representation of the scene is obtained by aggregating depth maps and 2D image features from single or multiple camera views. A conditional diffusion model is used to learn the action distribution conditioned on the robot and environment states. The 3D pose trajectory of the robot end effector is iteratively denoised given language instructions. The multimodal pre-trained model CLIP is used as a 2D image encoder and a natural speech command encoder. The farthest point sampling algorithm is used to downsample the 3D feature point cloud of the scene. The FiLM module is used to fuse the denoising time step and the robot body information to adjust the network activation value. Using cross-attention to fuse language information with visual features, the self-attention layer predicts the error of the end-effector pose trajectory under the current environment state and time step.

5. The method according to claim 4, characterized in that In step S24, in order to achieve the migration of operation strategies from the virtual environment to the real environment, a domain randomization method needs to be adopted: multiple sets of physical parameters are sampled in the simulation environment to learn a more stable strategy; first, the physical parameters are independently sampled in the simulation environment to train the operation strategy, and the feasible parameter value range is roughly determined. Then, a large language model is used to infer reasonable parameter combinations based on the operation tasks and physical common sense, and multiple candidate operation strategies are trained again. Finally, they are deployed and tested in the real environment.

6. The method according to claim 5, characterized in that In step S3, a method is adopted in which the robot autonomously explores the local environment through a large language model and a visual model to construct a scene graph that captures the structure of the environment. To obtain multiple attributes of a single entity in the scene graph and the spatial relationships between different entities, this is regarded as a visual query problem, and a perception API is constructed using multiple pre-trained models, including an object recognition model, an open vocabulary target detection model, a semantic segmentation model, a pose estimation model, and a large multimodal model. The large language model is used to generate executable API call code based on the relational query statement.

7. The method according to claim 6, characterized in that In step S4, a large language model is used to manage the human-computer dialogue, infer the user's intention and parse the operation task instructions issued by the user, and summarize the human-computer dialogue context to update the task planning database.

8. The method according to claim 7, characterized in that In step S5, S51 Hierarchical task planning based on thought chain prompts; S52 builds a personalized mission planning database; S53 integrates external text feedback.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images