Natural language control method for humanoid robot

Abstract

The invention provides a natural language control method for a humanoid robot. By constructing the simple skill library and the complex skill library of the robot, the dependence on a large amount of operation data in a specific field is effectively reduced. According to the method, a large language model is used as a base of a task planner, and the capabilities of scene learning, semantic understanding and common sense reasoning of the task planner are brought into full play. By introducing a visual model in the field of computer vision, high-level semantic information and low-level spatial information of the environment can be obtained, and an accurate scene graph is constructed to serve as environment state representation. According to the voice interaction interface constructed by the invention, the naturalness of man-machine interaction is improved, so that a user can more conveniently communicate with the robot and input instructions, the user experience is effectively improved, and popularization of the humanoid robot in an actual application scene is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a natural language control method for a humanoid robot. Background Art

[0002] Humanoid robots are robots that have a human appearance and imitate human functions and intelligence. They have a head, a torso, and limbs, which makes them more versatile and adaptable in human work and living environments. Humanoid robots can not only improve the naturalness and efficiency of human-machine interaction, allowing robots to collaborate with humans in a more humane way, but also assist humans in completing dangerous, heavy or delicate work in industrial production, home services, medical care and other fields, significantly improving work efficiency and safety. In particular, in response to the problem of an aging society around the world, humanoid robots are expected to provide companionship and daily life support, reducing the work pressure of caregivers.

[0003] The rapid development of artificial intelligence, especially deep learning and reinforcement learning, has greatly promoted the research of humanoid robots. Deep learning enables robots to learn complex patterns and behaviors from large amounts of data, thereby achieving more intelligent perception and decision-making capabilities. Reinforcement learning helps robots optimize their action strategies through continuous experimentation and feedback, so as to complete tasks more effectively in the environment. In addition, advances in natural language processing and computer vision technologies have made humanoid robots more natural and efficient when interacting with humans.

[0004] At present, the research on humanoid robot manipulation mainly focuses on end-to-end control methods, which is a control method that directly generates output data (such as robot movements) based on raw input data (such as sensor information, user instructions, etc.). This method can significantly reduce the amount of programming and program code, avoid the complex intermediate steps and manual feature engineering in traditional methods, and enable humanoid robots to show higher flexibility and adaptability in various tasks. End-to-end control methods rely on a large amount of data for learning and training. However, in practical applications, obtaining sufficient quantity and quality of training data is often a difficult task, especially the cost of collecting robot operation data is very high, which further exacerbates the scarcity and imbalance of training data, resulting in insufficient model training or weak generalization ability, thus affecting the performance and stability of the robot. On the other hand, in order to improve the multi-task execution capability of the manipulation strategy, the end-to-end method needs to build a large-scale neural network model. The training of these models requires a lot of computing resources and time. At the same time, the huge amount of computing also limits the control rate of the system. Therefore, how to reduce the dependence on large amounts of operational data, improve the efficiency of model training and reasoning, while maintaining the flexibility and efficiency of humanoid robot operations so that it can adapt to the complex needs in actual application scenarios has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to solve the problems in the prior art to meet the needs of actual situations. A natural language control method for a humanoid robot is proposed to improve the problems that the existing humanoid robot operation strategies are highly dependent on specific field data and have low model training and reasoning efficiency.

[0006] The present invention is implemented by the following technical solution. The present invention proposes a natural language control method for a humanoid robot, and the method comprises the following steps:

[0007] S1. Use robotics technology to build a simple robot skill library, encapsulate it into an API that can be called by a program, and use it to complete simple operation tasks with a wide variety of operation objects and low perception accuracy requirements;

[0008] S2. Use end-to-end robot manipulation strategies fine-tuned for specific tasks to build a robot complex skill library to complete complex manipulation tasks that are difficult to decompose into simple skill sequences;

[0009] S3, using the visual model to obtain high-level semantic information and low-level spatial information of the environment, and constructing a scene graph as a representation of the environment state;

[0010] S4. Use voice services to build voice interaction interfaces to improve the naturalness of human-computer interaction;

[0011] S5. Based on a large language model, a task planner is built using the scenario learning, semantic understanding, and common sense reasoning capabilities of the large language model. The action sequence is planned according to natural language instructions and environmental state representations, and then the action instructions in the robot's simple skill library and complex skill library are called to control the robot to perform actions.

[0012] Further, in step S1,

[0013] S11. Build mobile navigation API: Provide map-based path planning function. The robot automatically calculates and moves along the optimal path based on the current position and target position. Support obstacle avoidance function. The integrated sensor detects the surrounding environment in real time to avoid collision. Users can set multiple navigation points to achieve complex movement sequences.

[0014] S12. Build robot motion planning API: Provide inverse kinematics solution function to calculate the angles of each joint of the robot according to the target position and posture of the end effector; support smooth motion trajectory planning to ensure that the robot does not produce excessive acceleration or jitter during movement; allow users to customize motion modes to adapt to different operation requirements;

[0015] S13. Build an object grasping API: Provide grasping solutions that can adapt to objects of various shapes, sizes and materials; provide a variety of configurable grasping strategies, including precise grasping based on friction control and stable grasping based on inclusiveness, to adapt to the characteristics of different objects; the API allows users to set grasping parameters and provides a closed-loop feedback mechanism to cope with grasping tasks in dynamic environments.

[0016] Further, in step S2,

[0017] S21. Establish a pre-training dataset and a fine-tuning dataset, wherein the pre-training dataset includes open source robot multi-task operation data and a synthetic dataset produced by writing task supervision code in a simulation environment using a large language model; the fine-tuning dataset is directly collected on a personalized robot and task scenario;

[0018] S22. Construct a robot operation strategy network based on a multimodal pre-trained model, and train the strategy network using the loss function of the diffusion model;

[0019] S23. Collect robot operation data in various mission scenarios in real environments through teleoperation or data gloves, fine-tune the policy network and deploy it on the robot;

[0020] S24. For the reinforcement learning of complex manipulation task strategies for dexterous hands, a reward function design based on a large language model and a domain randomization method are used to improve the effect of virtual-to-real migration and reduce manual workload.

[0021] Further, in step S22, the diffusion strategy and 3D scene representation are combined to realize the generalization of robot operation; the 3D feature point cloud representation of the scene is obtained by aggregating the depth map and 2D image features from a single or multiple camera perspectives, and the action distribution conditioned on the robot and environment states is learned using a conditional diffusion model, and the 3D pose trajectory of the robot end effector is iteratively denoised given the language instructions;

[0022] The multimodal pre-trained model CLIP is used as a 2D image encoder and natural speech command encoder. The farthest point sampling algorithm is used to downsample the 3D feature point cloud of the scene. The FiLM module is used to fuse the denoising time step and the robot body information to adjust the network activation value. Cross-attention is used to fuse language information and visual features, and the self-attention layer predicts the error of the end-effector pose trajectory under the current environment state and time step.

[0023] Furthermore, in step S24, in order to realize the migration of operation strategies from the virtual environment to the real environment, a domain randomization method needs to be adopted: sampling multiple groups of physical parameters in the simulation environment to learn a more stable strategy; firstly, independently sampling physical parameters in the simulation environment to train the operation strategy, roughly determining the feasible parameter value range, and then using a large language model to infer reasonable parameter combinations based on the operation tasks and physical common sense, and then training multiple candidate operation strategies again, and finally deploying and testing them in the real environment.

[0024] Furthermore, in step S3, a method is adopted in which the robot autonomously explores the local environment through a large language model and a visual model to construct a scene graph that captures the structure of the environment; in order to obtain multiple attributes of a single entity in the scene graph and the spatial relationship between different entities, it is regarded as a visual query problem, and a perception API is constructed using multiple pre-trained models, including an object recognition model, an open vocabulary target detection model, a semantic segmentation model, a pose estimation model, and a large multimodal model; a large language model is used to generate executable API call code based on the relational query statement.

[0025] Furthermore, in step S4, a large language model is used to manage the human-computer dialogue, infer the user's intention and parse the operation task instructions issued by the user, and summarize the human-computer dialogue context to update the task planning database.

[0026] Further, in step S5,

[0027] S51 Hierarchical task planning based on thought chain prompts;

[0028] S52 builds a personalized mission planning database;

[0029] S53 integrates external text feedback.

[0030] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the natural language control method for a humanoid robot when executing the computer program.

[0031] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the natural language control method for a humanoid robot.

[0032] Beneficial effects of the present invention:

[0033] 1. The present invention effectively reduces the reliance on a large amount of specific field operation data by constructing a simple skill library and a complex skill library for robots. The simple skill library encapsulates the basic robot operation API and can handle a variety of common tasks that do not require high perception accuracy, while the complex skill library achieves efficient processing of complex operation tasks through fine-tuning for specific tasks. The combination of the two significantly improves the operational flexibility and adaptability of humanoid robots.

[0034] 2. The present invention uses a large language model as the foundation of the task planner, giving full play to its ability of situational learning, semantic understanding and common sense reasoning. This enables the robot to intelligently plan a reasonable action sequence based on natural language instructions and environmental state representation, and call the action instructions in the corresponding skill library for execution, thereby improving the efficiency of model training and reasoning and reducing the consumption of computing resources.

[0035] 3. By introducing the visual model in the field of computer vision, the present invention can obtain high-level semantic information and low-level spatial information of the environment and construct an accurate scene graph as a representation of the environmental state. This not only enhances the robot's ability to perceive the environment, but also provides it with a richer basis for decision-making, making the robot more accurate and efficient when performing operations.

[0036] 4. The voice interaction interface constructed by the present invention improves the naturalness of human-computer interaction, allowing users to communicate and input commands with the robot more conveniently, effectively improving the user experience and facilitating the promotion of humanoid robots in practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A schematic flow chart of a natural language control method for a humanoid robot according to an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a process for building a robot complex skill library according to an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a process for constructing a scene graph environment representation according to an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of a process for constructing a voice interaction interface according to an embodiment of the present invention;

[0042] Figure 5 A flowchart of a task planner based on a large language model is shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Combination Figure 1-Figure 5 The present invention provides a natural language control method for a humanoid robot, the method comprising the following steps:

[0045] S1. Use robot technologies such as mobile navigation, robotic arm motion planning, and object grasping to build a simple robot skill library, which is encapsulated into an API that can be called by a program to complete simple operation tasks with a wide variety of operation objects and low perception accuracy requirements;

[0046] In step S1,

[0047] S11. Build mobile navigation API: Provide map-based path planning function, allowing the robot to automatically calculate and move along the optimal path based on the current position and target position; support obstacle avoidance function, through integrated sensors (such as lidar, ultrasonic sensor) to detect the surrounding environment in real time to avoid collision; allow users to set multiple navigation points to achieve complex movement sequences;

[0048] S12. Build robot motion planning API: Provide inverse kinematics solution function to calculate the angles of each joint of the robot according to the target position and posture of the end effector; support smooth motion trajectory planning to ensure that the robot does not produce excessive acceleration or jitter during movement; allow users to customize motion modes, such as linear interpolation, circular interpolation, etc., to meet different operation requirements;

[0049] S13. Build an object grasping API: Provide grasping solutions that can adapt to objects of various shapes, sizes and materials; provide a variety of configurable grasping strategies, including precise grasping based on friction control and stable grasping based on inclusiveness, to adapt to the characteristics of different objects; the API allows users to set grasping parameters (such as clamping force, clamping speed, etc.), and provides a closed-loop feedback mechanism to cope with grasping tasks in dynamic environments.

[0050] S2. Use end-to-end robot manipulation strategies (imitation learning and reinforcement learning of manipulation skills) for specific task fine-tuning to build a robot complex skill library to complete complex manipulation tasks that are difficult to decompose into simple skill sequences;

[0051] In step S2,

[0052] S21. Establish a pre-training dataset and a fine-tuning dataset, wherein the pre-training dataset includes open source robot multi-task operation data and a synthetic dataset produced by writing task supervision code in a simulation environment using a large language model; the fine-tuning dataset is directly collected on a personalized robot and task scenario;

[0053] Large-scale, high-capacity models trained on different datasets have shown significant success in effectively processing downstream applications. In order to increase the scale and diversity of robot operation task data while minimizing data collection costs, the imitation learning dataset is divided into a pre-training dataset and a fine-tuning dataset. The pre-training dataset includes existing open source data (multi-task operation data collected in a variety of robots and real environments) and simulated environment data, which has a large data scale but low pertinence; the fine-tuning dataset is directly collected on personalized robots and task scenarios, with a smaller data scale but higher data quality.

[0054] When training the policy network, we first use pre-training data containing a variety of tasks and scenarios to perform large-scale pre-training, so that the policy network can learn the general representation and knowledge of various tasks and scenarios, and improve its generalization to new tasks and scenarios; then use fine-tuning data for specific tasks and scenarios to fine-tune the policy network, so that the policy network can further learn and optimize the execution strategy of specific tasks based on the general knowledge it has mastered.

[0055] For simulated environment data, we automatically generate rich simulated environments and expert demonstrations by leveraging the code generation capabilities of large language models, improving the scene diversity (object instances and poses) and task diversity of synthetic data. We mainly use two data generation methods: goal-oriented generation, providing target tasks to large language models, and large language models propose task courses to solve target tasks; and exploratory generation, where large language models guide from previous tasks and iteratively propose novel tasks that help solve more complex tasks.

[0056] S22. Construct a robot operation strategy network based on a multimodal pre-trained model, and train the strategy network using the loss function of the diffusion model;

[0057] Many robotic manipulation tasks are inherently multimodal, with multiple actions that may produce the best behavior at any moment during the task, and human demonstrations often include multiple ways to complete a task. Rather than representing the policy as a deterministic function, it is more natural to consider policy learning as a probability distribution learning problem for continuous random variables. Reconstructing 3D scenes from 2D images can achieve better generalization capabilities because it can adapt to observations from new camera perspectives at test time.

[0058] The diffusion strategy and 3D scene representation are combined to achieve the generalization of robot operation. The 3D feature point cloud representation of the scene is obtained by aggregating depth maps and 2D image features from single or multiple camera perspectives, and the action distribution conditioned on the robot and environment states is learned using a conditional diffusion model. The 3D pose trajectory of the robot end effector is iteratively denoised given language instructions.

[0059] The multimodal pre-training model CLIP (Contrastive Language-Image Pre-training) is used as a 2D image encoder and a natural voice command encoder. The farthest point sampling algorithm is used to downsample the 3D feature point cloud of the scene. The FiLM (Feature-wise Linear Modulation) module is used to fuse the denoising time step and the robot body information to adjust the network activation value. The cross-attention is used to fuse the language information and visual features, and the self-attention layer predicts the error of the end effector posture trajectory under the current environment state and time step.

[0060] S23. Collect robot operation data in various mission scenarios in real environments through teleoperation or data gloves, fine-tune the policy network and deploy it on the robot;

[0061] Using teleoperation technology, experts can precisely control robots remotely to complete various complex tasks. By recording the expert's operation trajectory, force, speed, and decision-making process, high-quality operation data can be obtained; data gloves can capture the operator's hand movements and postures, and convert these fine movement information into instructions that the robot can recognize. In scenarios that require high-precision hand operations, the data of data gloves is particularly important.

[0062] S24. For the reinforcement learning of complex manipulation task strategies for dexterous hands, a reward function design based on a large language model and a domain randomization method are used to improve the effect of virtual-to-real migration and reduce manual workload.

[0063] Reinforcement learning is widely used in strategy learning for complex robot operation tasks because it does not require a large amount of labeled data to learn from interactions, fully explores the action space, and has strong stability. Natural language commands tend to describe the final effect of an action rather than the specific details of the action execution process, which is consistent with the role of the reward function that encodes the semantic information of the target state. In order to reduce the workload of manually designing the reward function, the zero-shot generation, code writing, and context improvement capabilities of large language models are used to generate reward function codes based on tasks expressed in natural language, and the standard PPO algorithm is used for training in a simulation environment. The effect of the reward function is then fed back to the large language model to improve the quality of the reward function.

[0064] In order to achieve the migration of operation strategies from virtual environments to real environments, a domain randomization method is needed: sampling multiple sets of physical parameters in the simulation environment to learn more stable strategies. First, the physical parameters are independently sampled in the simulation environment to train the operation strategy, and the feasible parameter value range is roughly determined. Then, a large language model is used to infer reasonable parameter combinations based on the operation tasks and physical common sense, and multiple candidate operation strategies are trained again. Finally, they are deployed and tested in the real environment.

[0065] S3, using the visual model to obtain high-level semantic information and low-level spatial information of the environment, and constructing a scene graph as a representation of the environment state;

[0066] Robots need to explore their surroundings to adapt to and handle tasks in unknown environments. Most previous works have adopted the method of constructing a scene graph of the environment, but it is usually assumed that the environment is static and fully observable, ignoring the areas that require active interaction and exploration, which limits the robot's ability to handle more complex tasks. To address this problem, a method is adopted in which the robot autonomously explores the local environment through a large language model and a visual model to construct a scene graph that captures the structure of the environment. The scene graph contains both low-level information such as geometry and semantics, and also considers high-level information such as the spatial relationship between different entities in the scene and the order of object exploration.

[0067] In order to obtain multiple attributes of a single entity in the scene graph and the spatial relations between different entities, it is regarded as a visual query problem, and a perception API is built using a variety of pre-trained models, including object recognition models, open vocabulary target detection models, semantic segmentation models, pose estimation models, and large multimodal models. A large language model is used to generate executable API call code based on relational query statements.

[0068] S4. Use voice services to build voice interaction interfaces to improve the naturalness of human-computer interaction;

[0069] Text to Speech (TTS) technology is used to synthesize text into audio. To improve speech recognition accuracy and save computing resources on mobile devices, voice wake-up and command word recognition tasks with lower recognition difficulty are deployed offline on embedded voice signal acquisition cards to reduce response delays and CPU usage; for voice dictation functions with higher recognition difficulty, streaming voice dictation technology is used to achieve real-time upload of voice signal streams and real-time return of text results through the websockets protocol, maximizing recognition accuracy and speed. To make voice interaction more natural and flexible, the Voice Activity Detection (VAD) model is used to determine in real time whether the user is speaking and control the start and end of voice dictation.

[0070] To improve system reliability, a large language model is used to manage human-computer dialogue, infer user intentions and parse the operational task instructions issued by the user, and summarize the human-computer dialogue context to update the task planning database.

[0071] S5. Based on a large language model, a task planner is built using the scenario learning, semantic understanding, and common sense reasoning capabilities of the large language model. The action sequence is planned according to natural language instructions and environmental state representations, and then the action instructions in the robot's simple skill library and complex skill library are called to control the robot to perform actions.

[0072] S51 Hierarchical task planning based on thought chain prompts

[0073] Operation tasks exhibit a highly hierarchical structure. In order to efficiently plan on a large state space and skill set, the planning process is divided into two stages: high-level and low-level. In high-level planning, the large language model autoregressively infers the high-level actions that need to be performed at the current moment based on the reasoning context. The high-level actions can be either operational actions that change the state of an object or perceptual actions that query the object's attributes and then update the action-scene graph. In low-level planning, for the high-level actions decomposed from the task planning, the large language model generates the corresponding executable code based on the current fine-grained environment state and the defined skill API. The abstract syntax tree is used to check whether the generated code is legal. If there is an undefined function reference in the code, the low-level planning module is recursively used to define the function reference based on the known API. On the one hand, the reference can be resolved, and on the other hand, the available skill API set can be automatically enriched.

[0074] S52 builds a personalized mission planning database

[0075] In order to parse open-domain natural language instructions and dialogues and learn to adapt to user preferences while avoiding exceeding the context window limit, a key-value database consisting of dialogue context-planning results is designed. A large language model is used to embed the current dialogue context, environment state, or execution result feedback into the vector space, and relevant information is retrieved from the database based on cosine similarity. The retrieved memory is passed as a context example to the large language model to generate more accurate and coherent responses. During the deployment process, the database is continuously updated, expanded, and enriched to fully utilize the scenario learning capabilities of the large language model to complete new tasks.

[0076] S53 Fusion External Text Feedback

[0077] In order to reliably perform tasks in dynamic environments and under external interference, the anti-interference and self-correction capabilities of large language model planning are improved by utilizing external text feedback (such as user descriptions, successful detection modules, function call execution results, etc.).

[0078] Example

[0079] See also Figure 1 to Figure 5 In an embodiment of the present invention, a natural language control method for a humanoid robot includes the following contents:

[0080] 1. Use robotic technologies such as mobile navigation, robotic arm motion planning, and object grasping to build a simple robot skill library, and encapsulate it into an API that can be called by the program.

[0081] Use ORB-SLAM to build a mobile navigation API. Provide map-based path planning function, allowing the robot to automatically calculate and move along the optimal path based on the current position and target position; support obstacle avoidance function, through integrated sensors (such as lidar, ultrasonic sensor) to detect the surrounding environment in real time to avoid collision; allow users to set multiple navigation points to achieve complex movement sequences.

[0082] Use MoveIt! to build a robotic arm motion planning API. It provides inverse kinematics solving function to calculate the angles of each joint of the robotic arm based on the target position and posture of the end effector; supports smooth motion trajectory planning to ensure that the robotic arm does not produce excessive acceleration or jitter during movement; and allows users to customize motion modes, such as linear interpolation, circular interpolation, etc., to meet different operation requirements.

[0083] Use GraspIt! to build an object grasping API. It provides grasping solutions for objects of various shapes, sizes and materials; it provides a variety of configurable grasping strategies, including precise grasping based on friction control and stable grasping based on inclusion, to adapt to the characteristics of different objects; the API allows users to set grasping parameters (such as gripping force, gripping speed, etc.), and provides a closed-loop feedback mechanism to cope with grasping tasks in dynamic environments.

[0084] 2. Build a library of complex robot skills using end-to-end robot action strategies fine-tuned for specific tasks.

[0085] Imitation learning datasets are divided into pre-training datasets and fine-tuning datasets. The pre-training datasets include existing open source data (Open-X-Embodiment) and simulation environment data (RLbench). The data scale is large but the targeting is low. The fine-tuning datasets are collected directly on personalized robots and task scenarios. The data scale is smaller but the data quality is higher.

[0086] When training the policy network, we first use pre-training data containing a variety of tasks and scenarios to perform large-scale pre-training, so that the policy network can learn the general representation and knowledge of various tasks and scenarios, and improve its generalization to new tasks and scenarios; then use fine-tuning data for specific tasks and scenarios to fine-tune the policy network, so that the policy network can further learn and optimize the execution strategy of specific tasks based on the general knowledge it has mastered.

[0087] The diffusion strategy and 3D scene representation are combined to achieve the generalization of robot operation. The 3D feature point cloud representation of the scene is obtained by aggregating depth maps and 2D image features from single or multiple camera perspectives, and the action distribution conditioned on the robot and environment states is learned using a conditional diffusion model. The 3D pose trajectory of the robot end effector is iteratively denoised given language instructions.

[0088] The multimodal pre-training model CLIP (Contrastive Language-Image Pre-training) is used as a 2D image encoder and a natural voice command encoder. The farthest point sampling algorithm is used to downsample the 3D feature point cloud of the scene. The FiLM (Feature-wise Linear Modulation) module is used to fuse the denoising time step and the robot body information to adjust the network activation value. The cross-attention is used to fuse the language information and visual features, and the self-attention layer predicts the error of the end effector posture trajectory under the current environment state and time step.

[0089] 3. Use visual models in the field of computer vision to obtain high-level semantic information and low-level spatial information of the environment, and construct a scene graph as a representation of the environment state.

[0090] In order to obtain multiple attributes of a single entity in the scene graph and the spatial relations between different entities, it is regarded as a visual query problem, and a perception API is built using a variety of pre-trained models, including an object recognition model (DETR), an open vocabulary target detection model (OWL-ViT), a semantic segmentation model (Segment-Anything), a pose estimation model (FoundationPose), and a large multimodal model (GPT-4V), etc. A large language model (GPT-4-turbo) is used to generate executable API call code based on relational query statements.

[0091] 4. Use existing voice services to build voice interaction interfaces to improve the naturalness of human-computer interaction.

[0092] In order to improve the accuracy of speech recognition and save the computing resources of mobile devices, Picovoice service is used for voice wake-up and command word recognition tasks with lower recognition difficulty. For the voice dictation function with higher processing difficulty, the streaming voice dictation cloud service provided by iFlytek is adopted. The websockets protocol is used to realize real-time upload of voice signal stream and real-time return of text results, thereby maximizing recognition accuracy and speed.

[0093] Use ALSA to manage audio devices, use PulseAudio to implement audio sharing, use Pyaudio to read and write audio streams to audio devices, adopt non-blocking callback mode audio stream control, and store audio data in global variables by defining callback functions.

[0094] In order to make voice interaction more natural and flexible, Silero VAD is deployed on the robot. Silero VAD is a voice activity detection model based on deep neural networks. It is trained on a large corpus of more than 100 languages. It performs well on audio in different fields and can cope with various background noises and quality levels. It has excellent detection accuracy and recognition speed.

[0095] IO concurrent processing is implemented based on threads and coroutines. Coroutines are used to control asynchronous websockets communication, and threads are used to manage various modules of voice interaction.

[0096] 5. Based on a large language model (GPT-4-turbo), a task planner is built using the scenario learning, semantic understanding, and common sense reasoning capabilities of the large language model. The action sequence is planned according to natural language instructions and environmental state representations, and then the action instructions in the robot's simple skill library and complex skill library are called to control the robot to perform actions.

[0097] Operation tasks exhibit a highly hierarchical structure. In order to efficiently plan on a large state space and skill set, the planning process is divided into two stages: high-level and low-level. In high-level planning, the large language model autoregressively infers the high-level actions that need to be performed at the current moment based on the reasoning context. The high-level actions can be either operational actions that change the state of an object or perceptual actions that query the object's attributes and then update the action-scene graph. In low-level planning, for the high-level actions decomposed from the task planning, the large language model generates the corresponding executable code based on the current fine-grained environment state and the defined skill API. The abstract syntax tree is used to check whether the generated code is legal. If there is an undefined function reference in the code, the low-level planning module is recursively used to define the function reference based on the known API. On the one hand, the reference can be resolved, and on the other hand, the available skill API set can be automatically enriched.

[0098] In order to parse open-domain natural language instructions and dialogues and learn to adapt to user preferences while avoiding exceeding the context window limit, a key-value database consisting of dialogue context-planning results is designed. A large language model is used to embed the current dialogue context, environment state, or execution result feedback into the vector space, and relevant information is retrieved from the database based on cosine similarity. The retrieved memory is passed as a context example to the large language model to generate more accurate and coherent responses. During the deployment process, the database is continuously updated, expanded, and enriched to fully utilize the scenario learning capabilities of the large language model to complete new tasks.

[0099] In order to reliably perform tasks in dynamic environments and under external interference, the anti-interference and self-correction capabilities of large language model planning are improved by utilizing external text feedback (such as user descriptions, successful detection modules, function call execution results, etc.).

[0100] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the natural language control method for a humanoid robot when executing the computer program.

[0101] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the natural language control method for a humanoid robot.

[0102] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the method described in the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0103] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0104] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0105] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0106] The above is a detailed introduction to the natural language control method of a humanoid robot proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A natural language control method for a humanoid robot, characterized in that: The method comprises the following steps: S1. Use robotics technology to build a simple robot skill library, encapsulate it into an API that can be called by a program, and use it to complete simple operation tasks with a wide variety of operation objects and low perception accuracy requirements; S2. Use end-to-end robot manipulation strategies fine-tuned for specific tasks to build a robot complex skill library to complete complex manipulation tasks that are difficult to decompose into simple skill sequences; S3, using the visual model to obtain high-level semantic information and low-level spatial information of the environment, and constructing a scene graph as a representation of the environment state; S4. Use voice services to build voice interaction interfaces to improve the naturalness of human-computer interaction; S5. Based on a large language model, a task planner is built using the scenario learning, semantic understanding, and common sense reasoning capabilities of the large language model. The action sequence is planned according to natural language instructions and environmental state representations, and then the action instructions in the robot's simple skill library and complex skill library are called to control the robot to perform actions.

2. The method according to claim 1, characterized in that In step S1, S11. Build mobile navigation API: Provide map-based path planning function. The robot automatically calculates and moves along the optimal path based on the current position and target position. Support obstacle avoidance function. The integrated sensor detects the surrounding environment in real time to avoid collision. Users can set multiple navigation points to achieve complex movement sequences. S12. Build robot motion planning API: Provide inverse kinematics solution function to calculate the angles of each joint of the robot according to the target position and posture of the end effector; support smooth motion trajectory planning to ensure that the robot does not produce excessive acceleration or jitter during movement; allow users to customize motion modes to adapt to different operation requirements; S13. Build an object grasping API: Provide grasping solutions that can adapt to objects of various shapes, sizes and materials; provide a variety of configurable grasping strategies, including precise grasping based on friction control and stable grasping based on inclusiveness, to adapt to the characteristics of different objects; the API allows users to set grasping parameters and provides a closed-loop feedback mechanism to cope with grasping tasks in dynamic environments.

3. The method according to claim 2, characterized in that In step S2, S21. Establish a pre-training dataset and a fine-tuning dataset, wherein the pre-training dataset includes open source robot multi-task operation data and a synthetic dataset produced by writing task supervision code in a simulation environment using a large language model; the fine-tuning dataset is directly collected on a personalized robot and task scenario; S22. Construct a robot operation strategy network based on a multimodal pre-trained model, and train the strategy network using the loss function of the diffusion model; S23. Collect robot operation data in various task scenarios in real environments through teleoperation or data gloves, fine-tune the policy network and deploy it on the robot; S24. For the reinforcement learning of complex manipulation task strategies for dexterous hands, a reward function design based on a large language model and a domain randomization method are used to improve the effect of virtual-to-real migration and reduce manual workload.

4. The method according to claim 3, characterized in that In step S22, the diffusion strategy and 3D scene representation are combined to realize the generalization of robot operation; the 3D feature point cloud representation of the scene is obtained by aggregating the depth map and 2D image features from a single or multiple camera perspectives, and the action distribution conditioned on the robot and environment states is learned using a conditional diffusion model, and the 3D pose trajectory of the robot end effector is iteratively denoised given the language instructions; The multimodal pre-trained model CLIP is used as a 2D image encoder and a natural voice command encoder. The farthest point sampling algorithm is used to downsample the 3D feature point cloud of the scene. The FiLM module is used to fuse the denoising time step and the robot body information to adjust the network activation value. Using cross-attention to fuse language information with visual features, the self-attention layer predicts the error of the end-effector pose trajectory under the current environment state and time step.

5. The method according to claim 4, characterized in that In step S24, in order to realize the migration of operation strategies from the virtual environment to the real environment, a domain randomization method needs to be adopted: multiple groups of physical parameters are sampled in the simulation environment to learn a more stable strategy; first, the physical parameters are independently sampled in the simulation environment to train the operation strategy, and the feasible parameter value range is roughly determined. Then, a large language model is used to infer reasonable parameter combinations based on the operation tasks and physical common sense, and multiple candidate operation strategies are trained again, and finally deployed and tested in the real environment.

6. The method according to claim 5, characterized in that In step S3, a method is adopted in which the robot autonomously explores the local environment through a large language model and a visual model to construct a scene graph that captures the structure of the environment; in order to obtain multiple attributes of a single entity in the scene graph and the spatial relationship between different entities, it is regarded as a visual query problem, and a perception API is constructed using multiple pre-trained models, including an object recognition model, an open vocabulary target detection model, a semantic segmentation model, a pose estimation model, and a large multimodal model; a large language model is used to generate executable API call code based on the relational query statement.

7. The method according to claim 6, characterized in that In step S4, a large language model is used to manage the human-computer dialogue, infer the user's intention and parse the operation task instructions issued by the user, and summarize the human-computer dialogue context to update the task planning database.

8. The method according to claim 7, characterized in that In step S5, S51 Hierarchical task planning based on thought chain prompts; S52 builds a personalized mission planning database; S53 integrates external text feedback.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images