Task execution method, device, robot and product
Patent Information
- Application Number
- CN202510421751.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-03
- Publication Date
- 2026-10-16
Smart Images
Figure CN122914111A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of robotics, and more specifically to methods, apparatus, robots and products for performing tasks. Background Technology
[0002] With the rapid development of artificial intelligence and robotics, intelligent robots with autonomous movement and operation capabilities are gradually becoming more common in homes, businesses, and industries. Users expect to be able to issue commands to robots via voice, text, gestures, etc., and expect the robots to accurately understand the content of the commands in order to complete the corresponding tasks in the current environment. Summary of the Invention
[0003] In a first aspect of the embodiments of this disclosure, a task execution method is provided, applicable to an intelligent robot. The method includes acquiring a user instruction represented in natural language, the user instruction indicating an object to be grasped by the intelligent robot. The method further includes generating a movement instruction indicating the location of the object based on the user instruction, using a language model, so that the intelligent robot moves to that location. The method also includes acquiring an image at that location. Furthermore, the method includes generating a grasping instruction based on the image, so that the intelligent robot grasps the object.
[0004] In a second aspect of the embodiments of this disclosure, a task execution apparatus suitable for an intelligent robot is provided. The apparatus includes a user instruction acquisition module configured to acquire user instructions represented in natural language, the user instructions indicating an object to be grasped by the intelligent robot. The apparatus also includes a movement instruction generation module configured to generate, based on the user instructions and using a language model, a movement instruction indicating the location of the object, so that the intelligent robot moves to that location. The apparatus further includes an image acquisition module configured to acquire an image at that location. Additionally, the apparatus includes a grasping instruction generation module configured to generate a grasping instruction based on the image, so that the intelligent robot grasps the object.
[0005] In a third aspect of the embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a task execution method suitable for an intelligent robot. The method includes acquiring user instructions represented in natural language, the user instructions instructing the intelligent robot to grasp an object. The method further includes generating, based on the user instructions and utilizing a language model, a movement instruction indicating the location of the object, so as to cause the intelligent robot to move to that location. The method also includes acquiring an image at that location. Furthermore, the method includes generating a grasping instruction based on the image, so as to cause the intelligent robot to grasp the object.
[0006] In a fourth aspect of embodiments of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to implement a task execution method suitable for an intelligent robot. The method includes acquiring user instructions represented in natural language, the user instructions instructing the intelligent robot to grasp an object. The method further includes generating, based on the user instructions and utilizing a language model, a movement instruction indicating the location of the object, causing the intelligent robot to move to that location. The method also includes acquiring an image at that location. Furthermore, the method includes generating a grasping instruction based on the image, causing the intelligent robot to grasp the object.
[0007] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description
[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0009] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;
[0010] Figure 2 A flowchart of a task execution method according to some embodiments of the present disclosure is shown;
[0011] Figure 3 A schematic diagram illustrating an example of generating movement commands based on user instructions using a language model, according to some embodiments of the present disclosure;
[0012] Figure 4 A schematic diagram illustrates an example of generating multiple movement commands based on user instructions and utilizing a language model, according to some embodiments of the present disclosure;
[0013] Figure 5 A schematic diagram illustrates an example of generating multiple movement commands and multiple corresponding action commands based on user instructions and utilizing a language model, according to some embodiments of the present disclosure.
[0014] Figure 6 A schematic diagram illustrating an example of generating grabbing instructions based on user instructions and images according to some embodiments of the present disclosure;
[0015] Figure 7A schematic diagram illustrating an example of generating a grabbing instruction based on a user instruction and an image including multiple candidate objects, according to some embodiments of the present disclosure;
[0016] Figure 8 A block diagram of a task execution apparatus according to some embodiments of the present disclosure is shown; and
[0017] Figure 9 A block diagram of an intelligent robot capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation
[0018] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.
[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0020] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.
[0021] In the field of intelligent robot technology, users expect to issue natural language commands to robots via voice or text, such as "Get me a bottle of water" or "Get me my phone." To accomplish these tasks, robots need to understand user intentions and be able to perceive and manipulate their real-world environment. Some related technologies pre-set semantic rules or command templates for specific tasks, thereby converting natural language into concrete operational instructions. For example, rule matching is a method based on predefined grammatical rules or templates. A robot can be configured with a pre-defined set of templates and use these templates to identify key elements in the user's statement. For instance, a sample template might be "Give me the [item] at [location]." When a user says a command such as "Give me the water glass from the kitchen," the robot can compare the command with the template, thus extracting "kitchen" as the location and the water glass as the object to be grasped.
[0022] However, these related technologies cannot handle vague or irregular user instructions. For example, when certain key elements are missing from a user instruction, the robot will be unable to complete the task successfully. For instance, when the user instruction is "Give me the water glass," the instruction lacks a location element. Therefore, the robot does not know where to grab the water glass. In related technologies, to solve this problem, experts need to manually supplement new rules or templates to cover a wide variety of user instructions. Furthermore, experts also need to manually configure missing information; for example, the location of the water glass might be on a restaurant table or in a kitchen cabinet. This increases the resources and time spent maintaining the grammatical rules or templates, leading to increased costs for the robot.
[0023] In recent years, with the development of deep learning and natural language processing technologies, Large Language Models (LLMs, also referred to as language models in this paper) have become one of the core technologies for realizing human-computer interaction based on natural language. Large language models are neural network models with extremely large parameter scales trained on massive corpora of data. Some examples of large language models include GPT and DeepSeek. These models, through unsupervised pre-training on large-scale text data, can learn the grammatical structure, semantic rules, and contextual relationships of language, possessing powerful language understanding and generation capabilities. Unlike traditional natural language processing methods that rely on rules or shallow semantic parsing, large language models can automatically infer user intent, understand complex sentences, and handle context-dependent information without explicit rules. Furthermore, large language models also possess a certain degree of logical reasoning and knowledge generalization capabilities.
[0024] The introduction of large language models enables robotic systems to leap from "keyword recognition" to "semantic understanding and reasoning" in language comprehension, greatly enhancing the system's ability to process natural language and laying the foundation for language-driven task execution systems.
[0025] Therefore, embodiments of this disclosure provide a task execution scheme applicable to intelligent robots. In this scheme, the robot can acquire user instructions represented in natural language, which instruct the robot to grasp an object. Then, based on the user instructions, the robot can use a language model to generate movement instructions indicating the location of the object, causing the robot to move to that location. The robot can then acquire an image at that location. Finally, based on the image, the robot can generate grasping instructions to grasp the object.
[0026] In this way, leveraging the understanding capabilities of language models enhances the robot's natural language processing abilities, enabling it to handle user commands expressed in various ways. This allows users to express their needs more naturally. Furthermore, by utilizing the reasoning capabilities of language models, the robot can process user commands lacking essential elements, allowing users to communicate with the robot in a manner similar to human communication, rather than having to wait until all necessary elements are present before speaking the command. Thus, this approach not only reduces the cost of enabling the robot to understand user commands but also improves the user experience.
[0027] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of this disclosure may be implemented is shown. For example... Figure 1 As shown, environment 100 includes robot 102, which may be equipped with various hardware or sensors. For example, robot 102 may have a language interaction module, a motion control and navigation module, a visual perception and recognition module, a grasping and execution module, and a control and communication module. The control and communication module may include a controller and a wireless communication module. The controller may be any control unit with processing or computing capabilities. The controller can be used to control the various subsystems of robot 102, schedule tasks, fuse sensor data, and make behavioral decisions. The wireless communication module may include, for example, a Wi-Fi module or a Bluetooth module. The wireless communication module can be used to communicate with user equipment, cloud servers, or other devices.
[0028] The language interaction module may include a microphone array or voice receiver and a speaker. The microphone array or voice receiver can be used to capture the user's voice commands and can also support functions such as speech recognition and noise suppression. The speaker can be used to interact with the user via voice (e.g., asking clarifying questions).
[0029] The motion control and navigation module may include a mobile chassis, wheel encoders, an inertial measurement unit (IMU), a lidar, a depth camera, and a navigation control unit. The mobile chassis may be, for example, a differential drive chassis or an omnidirectional wheel chassis. When robot 102 is a humanoid robot, the mobile chassis may be the robot's legs and feet. The mobile chassis enables robot 102 to navigate autonomously on the ground. The wheel encoders and IMU can be used to achieve motion state perception and attitude estimation. The lidar and depth camera can support simultaneous localization and mapping (SLAM) functions for mapping and obstacle avoidance. The navigation control unit can translate movement commands into executable path commands to move robot 102 to the location indicated by the movement commands.
[0030] The visual perception and recognition module may include an RGB camera or an RGB-D camera, an image processing unit, and a depth map perception module. The RGB or RGB-D camera can be used to acquire environmental images after the robot 102 reaches the target position. These environmental images can be, for example, color images or depth images. The image processing unit can use object detection models, image segmentation models, etc., to identify the position of the target object in the image. The depth map perception module can be used to calculate the three-dimensional position and pose of the target object to determine the grasping action.
[0031] The grasping execution module may include a robotic arm, an end effector, and tactile sensors. The robotic arm and end effector can support spatial motion for performing grasping actions. The tactile sensors can be used to determine whether an object has been successfully grasped and to monitor the contact state between the end effector and the target object.
[0032] In environment 100, user 104 and robot 102 can be located at location 106 (e.g., living room, bedroom, etc.). User 104 can input user commands 112 in natural language to robot 102, where user commands 112 instruct robot 102 to grasp an object. For example, user command 112 could be "Bring me the water glass," where the object to be grasped is the water glass. User command 112 could also be "Bring me my history book," where the object to be grasped is the book. User commands 112 can be voice or text. In some embodiments, user 104 can speak user command 112 to the robot. In some embodiments, user 104 can input user command 112 through a user device or the user interface of robot 102. In some embodiments, user 104 can input user command 112 by selecting controls displayed on the user interface of robot 102.
[0033] In environment 100, robot 102 can invoke language model 114. Language model 114 can be deployed locally on robot 102. For example, language model 114 can be deployed in the controller of robot 102. Language model 114 can also be deployed on a cloud server. Robot 102 can invoke language model 114 by communicating with the cloud server using a wireless communication module. Language model 104 is a deep learning-based natural language processing model that learns from massive amounts of text data, captures the syntax, semantics, and contextual relationships of language, and is able to understand natural language and generate high-quality coherent text. For example, a technique for implementing language model 104 could be based on a Transformer architecture, using a self-attention mechanism to analyze the relationships between words and sentences in the input text to identify deep-level structures in the language. Language model 104 can be a large language model, such as GPT, DeepSeek, etc.
[0034] In environment 100, robot 102 can generate a movement instruction 116 indicating the location of an object to be grasped, based on user instruction 112 and using language model 114. For example, movement instruction 116 could indicate moving to location 110. In the example where user instruction 112 is "Please bring me the water glass," the object to be grasped is a water glass. However, user instruction 112 does not explicitly indicate the location of the water glass. By utilizing the understanding and reasoning capabilities of language model 114, robot 102 can infer that the object to be grasped 108 can be found at location 110 (e.g., the dining room). Therefore, robot 102 can input the user instruction into language model 114. Language model 114 can generate movement instruction 116 based on user instruction 112. Then, robot 102 can move from location 106 to location 110 (e.g., from the bedroom to the dining room) based on movement instruction 116.
[0035] In environment 100, after moving to location 110, robot 102 can use a camera at location 110 to take pictures of the surrounding environment to obtain image 118, which includes the object 108 to be grasped. For example, after moving to a location in a restaurant, robot 102 can use a camera to take pictures of the restaurant to obtain an image of the restaurant. In this image, a water glass is placed on a table.
[0036] In environment 100, robot 102 can generate grasping instructions 120 based on image 118 and grasp object 108 based on grasping instructions 120. In some embodiments, robot 102 can utilize an object detection model to identify object 108 in image 118 and determine the position of object 108. Then, robot 102 can generate corresponding grasping instructions 120 based on the determined position of object 108. In some embodiments, robot 102 can utilize language model 114 (or another language model) to generate grasping instructions 120. For example, robot 102 can input image 118 into language model 114 so that language model 114 generates grasping instructions 120 based on image 118. Then, robot 102 can grasp object 108 based on the generated grasping instructions 120, return to position 106, and deliver object 108 to user 104.
[0037] In this way, leveraging the understanding capabilities of language model 114 enhances the natural language processing capabilities of robot 102, enabling it to handle user instructions 112 expressed in various ways. This allows user 104 to express their needs more naturally. Furthermore, by utilizing the reasoning capabilities of language model 114, robot 102 can handle user instructions lacking necessary elements, allowing user 104 to communicate with robot 102 in a manner similar to human communication, without needing to confirm that the instruction contains all necessary elements before speaking it to robot 102. Thus, this approach not only reduces the cost of enabling robot 102 to understand the meaning of user instructions but also improves the user experience.
[0038] Figure 2 A flowchart of a task execution method 200 according to some embodiments of the present disclosure is shown. Method 200 can be executed by an intelligent robot or a robot controller. For example, method 200 can be executed by… Figure 1 The robot 102 or its controller executes the commands. For example... Figure 2 As shown in box 202, the robot can obtain user instructions expressed in natural language, which instruct the robot to grasp the object. For example, in... Figure 1In the environment 100 shown, user 104 and robot 102 can be located at position 106 (e.g., living room, bedroom, etc.). User 104 can input user commands 112 in natural language to robot 102, where user commands 112 instruct robot 102 to grasp an object. For example, user command 112 could be "Bring me the water glass," where the object to be grasped is the water glass. User command 112 could also be "Bring me my history book," where the object to be grasped is the book. User commands 112 can be voice or text. In some embodiments, user 104 can speak user command 112 to the robot. In some embodiments, user 104 can input user command 112 through a user device or the user interface of robot 102. In some embodiments, user 104 can input user command 112 by selecting controls displayed on the user interface of robot 102.
[0039] In box 204, the robot can use a language model to generate movement commands indicating the location of an object based on user instructions, so that the robot can move to that location. For example, in... Figure 1 In the environment 100 shown, robot 102 can invoke language model 114. Language model 114 can be deployed locally on robot 102. For example, language model 114 can be deployed in the controller of robot 102. Language model 114 can also be deployed on a cloud server. Robot 102 can invoke language model 114 by communicating with the cloud server using a wireless communication module. Language model 114 is a deep learning-based natural language processing model that learns from massive amounts of text data, captures the syntax, semantics, and contextual relationships of language, and is able to understand natural language and generate high-quality coherent text. Language model 114 can be, for example, a large language model such as GPT or DeepSeek. Robot 102 can use language model 114 to generate movement command 116 indicating the location of the object to be grasped based on user command 112. For example, movement command 116 can indicate moving to location 110. In the example where user command 112 is "Please bring me the water cup," the object to be grasped is the water cup. However, user command 112 does not explicitly indicate the location of the water cup. By utilizing the understanding and reasoning capabilities of language model 114, robot 102 can infer that the object 108 to be grasped can be found at location 110 (e.g., a dining room). Therefore, robot 102 can input user commands into language model 114. Language model 114 can generate movement commands 116 based on user commands 112. Then, robot 102 can move from location 106 to location 110 (e.g., from the bedroom to the dining room) based on movement commands 116.
[0040] In box 206, the robot can acquire an image at that location. For example, in... Figure 1 In the environment 100 shown, after moving to position 110, robot 102 can use a camera at position 110 to take pictures of the surrounding environment to obtain image 118, which includes the object 108 to be grasped. For example, after moving to a location in a restaurant, robot 102 can use a camera to take pictures of the restaurant to obtain an image of the restaurant. In this image, a water glass is placed on a table.
[0041] In box 208, the robot can generate grasping instructions based on the image to enable the robot to grasp the object. For example, in... Figure 1 In the illustrated environment 100, robot 102 can generate grasping instructions 120 based on image 118 and grasp object 108 based on grasping instructions 120. In some embodiments, robot 102 can utilize an object detection model to identify object 108 in image 118 and determine the position of object 108. Then, robot 102 can generate corresponding grasping instructions 120 based on the determined position of object 108. In some embodiments, robot 102 can utilize language model 114 (or another language model) to generate grasping instructions 120. For example, robot 102 can input image 118 into language model 114 so that language model 114 generates grasping instructions 120 based on image 118. Then, robot 102 can grasp object 108 based on the generated grasping instructions 120, return to position 106, and deliver object 108 to user 104.
[0042] In this way, leveraging the understanding capabilities of language models enhances the robot's natural language processing abilities, enabling it to handle user commands expressed in various ways. This allows users to express their needs more naturally. Furthermore, by utilizing the reasoning capabilities of language models, the robot can process user commands lacking essential elements, allowing users to communicate with the robot in a manner similar to human communication, rather than having to wait until all necessary elements are present before speaking the command. Thus, this approach not only reduces the cost of enabling the robot to understand user commands but also improves the user experience.
[0043] In some embodiments, when using a language model to generate movement commands, the robot can generate prompts for the language model based on user instructions. The robot can then input these prompts into the language model to generate movement commands. In some embodiments, the robot can obtain a task description and an output template. The robot can then generate prompts based on the task description, user instructions, and the output template. In some embodiments, movement commands include coordinates of a location. Constructing prompts improves the controllability and stability of the language model's output. Furthermore, by including task intent and template structure in the prompts, the format consistency and accuracy of the language model's output can be improved, ensuring that the generated movement commands are structurally standardized, easy to parse and execute, and reducing the possibility of errors in downstream modules.
[0044] Figure 3 A schematic diagram of example 300, based on user instructions and utilizing a language model, is shown according to some embodiments of the present disclosure. For example... Figure 3 As shown, Example 300 includes a prompt word generation module 302 and a language model 304, wherein the language model 304 may be, for example, a prompt word generation module 302 and a language model 304. Figure 1 The language model 114 is used in the example. In example 300, the robot can obtain a pre-determined task description 306 and a pre-determined output template 310. The task description 306 is used to prompt the language model 304 based on user instructions 308 (e.g., ...). Figure 1 The language model 304 generates output using user instructions 112, and this output should include movement instructions instructing the robot to move to the location of the grasped object. An output template 310 is used to define the format of the output of the language model 304. In some embodiments, the output template 310 may also include output examples to enable the language model 304 to better understand the output template 310 and improve the accuracy of the generated output. In some embodiments, the output template 310 may also include a reasoning explanation, requiring the language model 304 to explain the reason for moving to that location. By having the language model 304 output this explanation, the reasoning logic of the language model 304 can be enhanced, thereby improving the accuracy of the output.
[0045] In Example 300, the prompt generation module 302 can generate a prompt 312 based on the task description 306, the user instruction 308, and the output template 310. Below is an example of the prompt 312 generated when the user instruction 308 is "Please bring me the water glass":
[0046]
Task Description
[0047] You are a robot assistant. Your task is to determine which room to move to based on the user's natural language commands in preparation for a subsequent grasping operation. You need to use common sense and the content of the commands to make reasonable inferences and output a structured "movement command".
[0048] User Commands
[0049] Please bring me my water glass.
[0050] Output Template
[0051] Please output the movement command in the following format:
[0052]
[0053] In Example 300, the generated cue word 312 can be input into the language model 304. The language model 304 can output content based on the cue word 312, and the output content can include a movement instruction 314. The movement instruction 314 can indicate the location of the object to be grasped. For example, an example of a movement instruction 314 is shown below:
[0054]
[0055] In some embodiments, the output template 310 may further specify the target location to include coordinates. Thus, the movement command 314 may include specific coordinate values, such as the restaurant (x, y). This allows the robot to navigate to the target location based on the coordinate values in the movement command 314.
[0056] In some embodiments, the movement instruction is a first movement instruction, the location is a first location, and when generating movement instructions indicating the location of an object using a language model, the robot can generate multiple movement instructions based on the user instruction, including a first movement instruction and a second movement instruction. The second movement instruction indicates a second location where the object is located, and is used to move the robot to the second location to grasp the object if the robot fails to grasp it at the first location. This approach avoids the robot having to make additional inquiries with the user due to failing to find the object at the first location, thereby improving the coherence and intelligence of human-computer interaction. Furthermore, this approach increases the success rate of task completion and enhances the robustness of task execution.
[0057] In some embodiments, the task description can instruct the language model to generate multiple movement commands. The robot can acquire a pre-determined sorting strategy. Then, the robot can generate prompts based on the task description, sorting strategy, user commands, and output template. In this way, the language model can generate multiple movement commands to target locations according to a set priority. By embedding the sorting strategy into the prompt construction process, the practicality, controllability, and success rate of the language model's output can be improved, thereby further enhancing the robot's planning ability and interaction performance in complex environments.
[0058] In some embodiments, the sorting strategy can be to sort multiple movement instructions based on the probability that an object will appear at a corresponding location. This improves the success rate of task completion. In some embodiments, the sorting strategy can be to sort multiple movement instructions based on path planning for multiple locations corresponding to the multiple movement instructions. This reduces the average movement time and saves energy.
[0059] Figure 4 A schematic diagram of example 400, based on user instructions and utilizing a language model, is shown according to some embodiments of the present disclosure. For example... Figure 4 As shown, Example 400 includes a prompt word generation module 402 and a language model 404, wherein the language model 404 may be, for example, a prompt word generation module 402 and a language model 404. Figure 1 The language model 114 is used in the example. In example 400, the robot can obtain a predetermined task description 406, a predetermined sorting strategy 410, and a predetermined output template 412. The task description 406 is used to prompt the language model 404 based on user instructions 408 (e.g., ...). Figure 1 The language model 404 generates output using user instructions 112, and this output should include multiple move instructions, which are ordered based on a sorting strategy 410. An output template 412 is used to define the format of the output of the language model 404. In some embodiments, the output template 412 may also include output examples to enable the language model 404 to better understand the output template 412 and improve the accuracy of the generated output. In some embodiments, the output template 412 may also include a reasoning explanation, requiring the language model 404 to explain the reason for moving to that position. By having the language model 404 output this explanation, the reasoning logic of the language model 404 can be enhanced, thereby improving the accuracy of the output.
[0060] In Example 400, the prompt generation module 402 can generate a prompt 414 based on the task description 406, user instruction 408, sorting strategy 410, and output template 412. Below is an example of the prompt 414 generated when the user instruction 408 is "Please bring me the water glass":
[0061]
Task Description
[0062] You are a service robot. Your task is to generate multiple possible movement commands based on the user's natural language instructions to move to the likely location of a target object. You need to sort these locations according to the object's probability of occurrence or likelihood of occurrence. You will output multiple movement commands and prioritize them according to the sorting strategy, trying the most likely locations first.
[0063] User Commands
[0064] Please bring me my water glass.
[0065] Sorting Strategy
[0066] Please sort the multiple target locations according to the following principles:
[0067] - Determine the most likely location of the water cup based on common sense;
[0068] - Sort by "probability of finding the target object from high to low";
[0069] - Please explain your reasoning for ordering the position in each move instruction.
[0070] Output Template
[0071] Please output multiple move commands according to the following JSON structure (2-3 commands are recommended):
[0072]
[0073] In Example 400, the generated prompt word 414 can be input into language model 404. Language model 404 can output content based on prompt word 414, which may include movement instructions 416-1, 416-2, and 416-3. Movement instructions 416-1, 416-2, and 416-3 can indicate the possible location of the object to be grasped. Movement instructions 416-1, 416-2, and 416-3 are ordered according to priority, where movement instruction 416-1 has a higher priority than movement instruction 416-2, and movement instruction 416-2 has a higher priority than movement instruction 416-3. For example, an example of movement instructions 416-1, 416-2, and 416-3 is shown below:
[0074]
[0075] After generating movement commands 416-1, 416-2, and 416-3, the robot can first move to the location indicated by movement command 416-1 (e.g., a restaurant) and then take an image of the surrounding environment. The robot can identify whether there is an object to be grasped in the image. If the object exists, the robot can grasp it and deliver it to the user. If the object does not exist, the robot can continue to move to the location indicated by movement command 416-2 (e.g., a kitchen) and then take an image of the surrounding environment. The robot can identify whether there is an object to be grasped in the image. If the object exists, the robot can grasp it and deliver it to the user. If the object does not exist, the robot can continue to move to the location indicated by movement command 416-3 (e.g., a bedroom) and then take an image of the surrounding environment, and so on.
[0076] In practical applications, robots often need to perform certain auxiliary actions before grasping an object, such as opening cabinet doors, drawers, refrigerators, or moving obstructions. In some embodiments, the robot can use a language model to generate motion instructions associated with movement commands based on user instructions. This allows the robot to execute the specified actions included in the motion instructions after moving to the location and before grasping the object. By using a language model to generate position-associated motion instructions, the robot can complete intermediate steps related to the task context. This enhances the robot's task adaptability in real-world scenarios. Furthermore, this approach can refine implicit steps in user instructions, thereby improving the understanding of indirect user instructions and increasing the success rate of task completion.
[0077] Figure 5 A schematic diagram of example 500, based on user instructions and utilizing a language model, is shown according to some embodiments of the present disclosure to generate multiple movement instructions and multiple corresponding action instructions. For example... Figure 5 As shown, Example 500 includes a prompt word generation module 502 and a language model 504, wherein the language model 504 may be, for example, a prompt word generation module 502 and a language model 504. Figure 1 The language model 114 is used in the example. In example 500, the robot can obtain a predetermined task description 506, a predetermined sorting strategy 510, and a predetermined output template 512. The task description 506 is used to prompt the language model 504 based on user instructions 508 (e.g., ...). Figure 1The output is generated using user instructions 512. This output should include multiple movement instructions and associated action instructions for each movement instruction, wherein the multiple movement instructions are ordered based on a sorting strategy 510, and the action instructions indicate a specified action that the robot needs to perform after moving to the target location and before grasping an object. An output template 512 is used to define the format of the output of the language model 504. In some embodiments, the output template 512 may also include output examples to enable the language model 504 to better understand the output template 512 and improve the accuracy of the generated output. In some embodiments, the output template 512 may also include a reasoning explanation, requiring the language model 504 to explain the reason for moving to that location. By having the language model 504 output this explanation, the reasoning logic of the language model 504 can be enhanced, thereby improving the accuracy of the output.
[0078] In Example 500, the prompt generation module 502 can generate a prompt 514 based on the task description 506, user instruction 508, sorting strategy 510, and output template 512. Below is an example of the prompt 514 generated when the user instruction 508 is "Please bring me the water glass":
[0079]
Task Description
[0080] You are a service robot. Users will give you natural language commands (such as "Bring me my water glass"). Based on these commands, you need to infer multiple possible target locations and generate a movement command for each location. After the robot moves to that location, before grasping the target object, it needs to execute an action command related to that location. For example: open a cabinet door, scan the surroundings, push aside obstacles, etc. Please generate corresponding action commands for each movement command based on task semantics and common sense.
[0081] User Commands
[0082] Please bring me my water glass.
[0083] Sorting Strategy
[0084] Please sort the multiple target locations according to the following principles:
[0085] - Determine the most likely location of the water cup based on common sense;
[0086] - Sort by "probability of finding the target object from high to low";
[0087] - Please explain your reasoning for ordering the position in each move instruction.
[0088] Output Template
[0089] Please output multiple target locations in the following JSON structure, each containing a movement command and several action commands:
[0090]
[0091] In Example 500, the generated prompt word 514 can be input into language model 504. Language model 504 can output content based on prompt word 514, which may include movement instructions 516-1, 516-2, and 516-3. Movement instructions 516-1, 516-2, and 516-3 can indicate the possible location of the object to be grasped. The output content may also include action instructions 518-1, 518-2, and 518-3. Action instruction 518-1 is associated with movement instruction 516-1, action instruction 518-2 is associated with movement instruction 516-2, and action instruction 518-3 is associated with movement instruction 516-3. Movement instructions 516-1, 516-2, and 516-3 are ordered according to priority, where movement instruction 516-1 has a higher priority than movement instruction 516-2, and movement instruction 516-2 has a higher priority than movement instruction 516-3. For example, the following shows an example of move commands 516-1, 516-2, and 516-3:
[0092]
[0093]
[0094]
[0095] After generating movement commands 516-1, 516-2, and 516-3, and action commands 518-1, 518-2, and 518-3, the robot can first move to the location indicated by movement command 516-1 (e.g., the dining room), execute action command 518-1 (e.g., scan the dining table), and then take an image of the surrounding environment. The robot can identify whether there is an object to be grasped in the image. If the object exists, the robot can grasp the object and deliver it to the user. If the object does not exist, the robot can continue to move to the location indicated by movement command 516-2 (e.g., the kitchen), execute action command 518-2 (e.g., open the cabinet), and then take an image of the surrounding environment. The robot can identify whether there is an object to be grasped in the image. If the object exists, the robot can grasp the object and deliver it to the user. If the object does not exist, the robot can continue to move to the location indicated by movement command 516-3 (e.g., the bedroom), execute action command 518-3 (e.g., scan the coffee table), and then take an image of the surrounding environment, and so on.
[0096] In some embodiments, user instructions may include object attributes (e.g., color, shape, size, etc.). For example, a user instruction might be "Bring me the green water glass," where the user specifies the water glass's color attribute as green. In some embodiments, the robot can determine the object's attributes based on the user instruction using a language model. The robot can then generate a grasping instruction based on the image and the object's attributes. In some embodiments, the object is the target object, and the robot can determine the target object's category. The robot can determine candidate objects in the image based on the target object's category. The robot can determine the values of the candidate object's attributes. Then, in response to determining that the value of a candidate object's attribute matches the value of the attribute specified in the user instruction, the robot can generate a grasping instruction by identifying the candidate object as the target object. In this way, the robot can narrow down the recognition range by filtering object attributes, thereby improving target localization accuracy, reducing false grasps, and increasing grasping accuracy.
[0097] Figure 6 A schematic diagram of an example 600 for generating grabbing instructions based on user commands and images according to some embodiments of the present disclosure is shown. Figure 6 As shown, Example 600 includes a language model 602 (e.g., Figure 1 The language model 602 can output the category 608 and attribute 610 of the object 606 to be grasped, as indicated in the user instruction 604, based on the user instruction 604. For example, the user instruction 604 could be "Bring me the green water cup," where the object 606 is a water cup. The language model 602 can receive a prompt word containing the user instruction 604 and output the category (e.g., cup) and attribute (i.e., green) of the water cup. The robot can then move to the target location and use a camera to capture an image 612 of the surrounding environment, where the image 612 contains the object 614 (e.g., a water cup). The robot can identify the object 614 with category 608 in the image 612 and identify the attribute 616 of the object 614. The robot can then compare the value of attribute 610 with the value of attribute 616. When the robot determines that the value of attribute 610 is the same as the value of attribute 616, it can generate a grasping instruction 618 to grasp the object 614.
[0098] In some embodiments, the image of the surrounding environment may contain multiple objects that meet the attribute requirements. To reduce the need for the robot to return to the user and ask which object to grasp, the robot can determine the category of the target object. The robot can identify multiple candidate objects in the image based on the category of the target object and the values of the attributes specified in the user instruction. Then, based on the user instruction and the image, the robot can use a second language model to generate a grasping instruction, which includes the target object determined from the multiple candidate objects. In this way, user interruption can be reduced, task continuity can be improved, and the user experience can be enhanced.
[0099] Figure 7 A schematic diagram of an example 700, illustrating the generation of grabbing instructions based on user instructions and an image including multiple candidate objects, according to some embodiments of the present disclosure, is shown. Figure 7 As shown, Example 700 includes a language model 702 (e.g., Figure 1 The robot uses language model 702 (114) and language model 720, which can be the same language model or different language models. Language model 702 can output the category 708 and attribute 710 of the object 706 to be grasped, as indicated in user instruction 704. For example, user instruction 704 could be "Bring me the green water cup," where object 706 is a water cup. Language model 702 can receive a prompt containing user instruction 704 and output the category (e.g., cup) and attribute (i.e., green) of the water cup. The robot can then move to the target location and use a camera to capture an image 712 of the surrounding environment, where image 712 contains objects 714, 716, and 718 (e.g., three water cups). The robot can identify objects 714, 716, and 718 with category 708 in image 712 and identify the attributes of objects 714, 716, and 718. The robot can then compare the value of attribute 710 with the attribute values of objects 714, 716, and 718. The robot can determine that objects 714, 716, and 718 all match attribute 710 (for example, they are all green water cups).
[0100] To determine the target object to be grasped, the robot can input image 712 and user command 704 into language model 720. Language model 720 can suggest one of objects 714, 716, and 718 as the target object to be grasped (e.g., object 714) based on image 712 and user command 704, and generate grasping command 722 to grasp the target object. The robot can then grasp the target object based on grasping command 722 and deliver it to the user.
[0101] In this way, the robot doesn't need to return to the user and ask which of the three objects (714, 716, and 718) the user wants to grab. This reduces travel time, increases task completion efficiency, and improves the user experience.
[0102] Figure 8 A block diagram of a task execution apparatus 800 according to some embodiments of the present disclosure is shown. Figure 8 As shown, the device 800 includes a user instruction acquisition module 802, configured to acquire user instructions represented in natural language, which instruct the robot to grasp an object. The device 800 also includes a movement instruction generation module 804, configured to generate movement instructions indicating the location of the object based on the user instructions and using a language model, so that the robot moves to that location. The device 800 also includes an image acquisition module 806, configured to acquire an image at that location. Furthermore, the device 800 includes a grasping instruction generation module 808, configured to generate grasping instructions based on the image, so that the robot grasps the object.
[0103] In some embodiments, the movement instruction generation module 804 includes: a prompt word generation module configured to generate prompt words for a language model based on user instructions; and a prompt word usage module configured to generate movement instructions by inputting the prompt words into the language model.
[0104] In some embodiments, the prompt word generation module includes: a task description acquisition module configured to acquire a task description and an output template; and a task description usage module configured to generate prompt words based on the task description, user instructions, and the output template.
[0105] In some embodiments, the movement instruction is a first movement instruction, the position is a first position, and the movement instruction generation module 804 includes: a second movement instruction generation module configured to generate multiple movement instructions based on user instructions and using a language model, wherein the multiple movement instructions include the first movement instruction and the second movement instruction, the second movement instruction indicating a second position where the object is located, and the second movement instruction being used to move the robot to the second position to grasp the object when the robot fails to grasp the object at the first position.
[0106] In some embodiments, the task description instruction language model generates multiple movement instructions, and the task description using module includes: a sorting strategy acquisition module configured to acquire a predetermined sorting strategy; and a sorting strategy using module configured to generate prompt words based on the task description, the sorting strategy, the user instructions, and the output template.
[0107] In some embodiments, the sorting strategy is used to sort multiple move instructions based on the probability that an object will appear in a corresponding position.
[0108] In some embodiments, the sorting strategy is to sort the multiple movement instructions based on path planning for multiple locations corresponding to the multiple movement instructions.
[0109] In some embodiments, the movement command includes the coordinates of the location.
[0110] In some embodiments, the apparatus 800 further includes a motion instruction generation module configured to generate motion instructions associated with movement instructions based on user instructions and using a language model, so that the robot performs a specified action included in the motion instructions after moving to a position and before grasping an object.
[0111] In some embodiments, the grabbing instruction generation module 808 includes: an attribute determination module configured to determine the attributes of an object based on a language model using user instructions; and an attribute usage module configured to generate grabbing instructions based on the attributes of an image and the object.
[0112] In some embodiments, the object is a target object, and the attribute usage module includes: a category determination module configured to determine the category of the target object; a category usage module configured to determine candidate objects in an image based on the category of the target object; a second attribute determination module configured to determine the value of an attribute of the candidate object; and a second attribute usage module configured to generate a grabbing instruction by determining the candidate object as the target object in response to determining that the value of the attribute of the candidate object matches the value of an attribute specified in a user instruction.
[0113] In some embodiments, the language model is a first language model, the object is a target object, and the attribute usage module includes: a second category determination module configured to determine the category of the target object; a second category usage module configured to determine a plurality of candidate objects in the image based on the category of the target object and the value of an attribute specified in a user instruction; and a second language model usage module configured to generate a grabbing instruction based on the user instruction and the image using the second language model, wherein the grabbing instruction contains the target object determined from the plurality of candidate objects.
[0114] It is understood that by utilizing the apparatus 800 of this disclosure, at least one of the many advantages achievable by the methods or processes described above can be realized. For example, in this way, the understanding ability of the language model can be used to improve the robot's natural language processing capabilities, enabling the robot to handle user instructions expressed in various ways. This allows users to express their needs more naturally. Furthermore, by utilizing the reasoning ability of the language model, the robot can handle user instructions lacking necessary elements, allowing users to communicate with the robot in a manner similar to human communication, without having to determine that the instruction contains all necessary elements before speaking it to the robot. Thus, this approach not only reduces the cost of enabling the robot to understand the meaning of user instructions but also improves the user experience.
[0115] Figure 9 A block diagram of a robot 900 capable of implementing various embodiments of the present disclosure is shown. The robot 900 may be, for example, as shown below. Figure 1 Robot 102 is shown. (As shown) Figure 9 As shown, the robot 900 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 901, which can execute various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. The RAM 903 can also store various programs and data required for the operation of the robot 900. The CPU / GPU 901, ROM 902, and RAM 903 are interconnected via bus 904. An input / output (I / O) interface 905 is also connected to bus 904. Although not shown in... Figure 9 As shown, robot 900 may also include a coprocessor.
[0116] Multiple components in robot 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows robot 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0117] The various methods or processes described above can be executed by CPU / GPU 901. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on robot 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU / GPU 901, one or more steps or actions in the methods or processes described above can be performed.
[0118] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.
[0119] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0120] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0121] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.
[0122] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0123] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0125] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A task execution method, characterized in that, Suitable for intelligent robots, including: The robot acquires user instructions in natural language representation, which instruct the intelligent robot to grasp an object. Based on the user instructions, a language model is used to generate movement instructions indicating the location of the object, so that the intelligent robot can move to the location. Acquire an image at the location; and The image is used to generate a grasping command so that the intelligent robot can grasp the object.
2. The method of claim 1, wherein generating the movement command using the language model based on the user command comprises: Based on the user instructions, generate prompt words for the language model; as well as The movement command is generated by inputting the prompt word into the language model.
3. The method of claim 2, wherein generating the prompt word for the language model based on the user instruction comprises: Get the task description and output template; as well as The prompt word is generated based on the task description, the user instruction, and the output template.
4. The method of claim 3, wherein the movement instruction is a first movement instruction, the location is a first location, and generating the movement instruction indicating the location of the object using the language model based on the user instruction comprises: Based on the user instructions, the language model is used to generate multiple movement instructions, wherein the multiple movement instructions include a first movement instruction and a second movement instruction, the second movement instruction indicating a second position where the object is located, and the second movement instruction is used to cause the intelligent robot to move to the second position to grasp the object when the intelligent robot fails to grasp the object at the first position.
5. The method of claim 4, wherein the task description instructs the language model to generate the plurality of movement instructions, and generating the prompt words based on the task description, the user instructions, and the output template comprises: Obtain a pre-determined sorting strategy; as well as The prompt words are generated based on the task description, the sorting strategy, the user instructions, and the output template.
6. The method of claim 5, wherein the sorting strategy sorts the plurality of movement instructions based on the probability that the object appears in the corresponding position.
7. The method of claim 5, wherein the sorting strategy is based on sorting the plurality of movement instructions according to path planning for the plurality of locations corresponding to the plurality of movement instructions.
8. The method of claim 1, wherein the movement command includes the coordinates of the position.
9. The method according to claim 1, further comprising: Based on the user instructions, the language model is used to generate action instructions associated with the movement instructions, so that the intelligent robot performs the specified actions included in the action instructions after moving to the location and before grasping the object.
10. The method of claim 1, wherein generating the grabbing instruction based on the image comprises: Based on the user instructions, the language model is used to determine the attributes of the object; as well as The grabbing instructions are generated based on the attributes of the image and the object.
11. The method of claim 10, wherein the object is a target object, and generating the grabbing instruction based on the image and the attributes of the object comprises: Determine the category of the target object; Candidate objects in the image are determined based on the category of the target object; Determine the value of the attribute of the candidate object; as well as In response to determining that the value of the attribute of the candidate object matches the value of the attribute specified in the user instruction, the fetch instruction is generated by identifying the candidate object as the target object.
12. The method of claim 10, wherein the language model is a first language model, the object is a target object, and generating the grabbing instruction based on the image and the attributes of the object comprises: Determine the category of the target object; Multiple candidate objects in the image are determined based on the category of the target object and the value of the attribute specified in the user instruction; as well as Based on the user instructions and the image, a second language model is used to generate the grabbing instructions, wherein the grabbing instructions include the target object determined from the plurality of candidate objects.
13. A task execution device, characterized in that, Suitable for intelligent robots, including: The user instruction acquisition module is configured to acquire user instructions represented in natural language, which instruct the intelligent robot to grasp an object. The movement instruction generation module is configured to generate movement instructions indicating the location of the object based on the user instruction and using a language model, so that the intelligent robot moves to the location; An image acquisition module is configured to acquire an image at the location; and The grasping instruction generation module is configured to generate grasping instructions based on the image so that the intelligent robot can grasp the object.
14. An intelligent robot, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the intelligent robot to perform the method according to any one of claims 1 to 12.
15. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 12.