Task execution method, device and robot

Abstract

The present invention provides a task execution method, device and robot, the method comprising: identifying object information in the environment based on an environment image corresponding to the task to be executed; decomposing the task to be executed into multiple subtasks based on the execution instructions and object information used to describe the task to be executed, each subtask including an execution object and an execution action of the execution object; identifying the posture of each execution object, and executing the corresponding subtask based on the posture and execution action of each execution object. The task execution method, device and robot provided by the present invention do not rely on a predefined instruction set, and thus can handle more complex and flexible execution instructions and adapt to complex and changing application scenarios. In addition, when faced with long text execution instructions, the present invention can also accurately understand the task requirements.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a task execution method, device and robot. Background Art

[0002] At the intersection of artificial intelligence and robotics, robotic grasping technology, a key component in achieving intelligent robotic operation, has made significant progress in recent years. Traditional robotic grasping methods rely primarily on layered perception models, which identify the target object and estimate its six-dimensional (6D) pose to generate a grasping pose, thereby guiding the robot to perform the grasping task.

[0003] However, robots usually work based on a predefined instruction set, that is, the robot can only recognize and execute instructions within the scope of the predefined instruction set. Once the user gives an instruction out of the range, the robot will not be able to make a correct response. Summary of the Invention

[0004] The present invention provides a task execution method, device and robot to solve the defects in the prior art.

[0005] The present invention provides a task execution method, comprising the following steps:

[0006] Based on the environment image corresponding to the task to be performed, identify the object information in the environment;

[0007] Based on the execution instruction for describing the task to be executed and the item information, decomposing the task to be executed into a plurality of subtasks, each subtask including an execution object and an execution action of the execution object;

[0008] Identify the position and posture of each execution object, and execute the corresponding subtask based on the position and posture of each execution object and the execution action.

[0009] According to a task execution method provided by the present invention, the task to be executed is decomposed into multiple subtasks based on the execution instruction used to describe the task to be executed and the item information, including:

[0010] Converting the execution instruction into an execution text;

[0011] extracting a target text associated with the item information from the execution text based on the item information;

[0012] Based on the target text, the task to be performed is decomposed into multiple subtasks.

[0013] According to a task execution method provided by the present invention, extracting target text associated with the item information from the execution text based on the item information includes:

[0014] Performing entity recognition on the execution text to obtain at least one execution entity;

[0015] determining, based on the item information, a target entity that matches the object in the environment from the at least one execution entity;

[0016] extracting an action entity associated with the target entity from the execution text;

[0017] The target text is constructed based on the action entity and the execution entity.

[0018] According to a task execution method provided by the present invention, extracting a target text associated with the item information from the execution text, and decomposing the task to be executed into multiple subtasks based on the target text, includes:

[0019] Constructing a task decomposition prompt text for prompting task decomposition requirements;

[0020] Based on the task decomposition model, the task decomposition prompt text is applied to extract the target text associated with the item information from the execution text, and based on the target text, the task to be executed is decomposed into multiple subtasks.

[0021] According to a task execution method provided by the present invention, identifying information about objects in the environment based on an environment image corresponding to the task to be executed includes:

[0022] Constructing identification prompt text for prompting item information identification requirements;

[0023] Based on the recognition model, the recognition prompt text is applied to identify and obtain the object information from the environment image.

[0024] According to a task execution method provided by the present invention, the identifying the position and posture of each execution object includes:

[0025] Identify and obtain detection frames of various execution objects from the environment image;

[0026] Based on the detection box of each execution object, the pose of each execution object is determined.

[0027] According to a task execution method provided by the present invention, each execution action exists in a preset action library.

[0028] The present invention also provides a task execution device, comprising the following modules:

[0029] A recognition unit, configured to recognize object information in the environment based on an image of the environment corresponding to the task to be performed;

[0030] a decomposition unit, configured to decompose the task to be executed into a plurality of subtasks based on the execution instruction describing the task to be executed and the item information, each subtask including an execution object and an execution action of the execution object;

[0031] The execution unit is used to identify the posture of each execution object and execute the corresponding subtask based on the posture and execution action of each execution object.

[0032] The present invention also provides a robot, comprising: a camera arranged on the robot body and the task execution device as described above, wherein the camera is used to collect the environment image.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-described task execution methods is implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned task execution methods when executed by a processor.

[0035] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the task execution methods described above.

[0036] The task execution method, device, and robot provided by the present invention can decompose the task to be executed into multiple simple subtasks through object information and execution instructions, and dynamically decompose the task to be executed based on specific environmental information and execution instructions without relying on a predefined instruction set. This allows the robot to handle more complex and flexible execution instructions and adapt to complex and changing application scenarios. In addition, when faced with a long text execution instruction containing multiple execution objects or complex action descriptions, the present invention can also accurately identify multiple execution objects or complex actions through object information and execution instructions, and decompose the task to be executed into multiple subtasks containing a single execution object or simple action. In other words, when faced with a long text execution instruction, the present invention can also accurately understand the task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 It is a flowchart of the task execution method provided by the present invention.

[0039] Figure 2 It is a flowchart of another task execution method provided by the present invention.

[0040] Figure 3 It is a structural diagram of the task execution device provided by the present invention.

[0041] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0043] In the field of robotic grasping technology, grasping operations on objects mainly rely on predefined instruction sets and target detection models. However, traditional grasping systems work based on predefined instruction sets, which means that the robot's grasping operations need to be defined in advance. Only when the robot recognizes a voice instruction that fully matches the predefined instruction set will it perform the corresponding grasping action. For example, if the predefined instruction set only contains the instruction "grab the red object", then when the user says "Please help me pick up the red apple", the robot may not be able to recognize and execute this instruction because it does not fully match the predefined instruction. This limitation makes it difficult to generalize the robot's grasping operations and cannot adapt to complex and changing application scenarios.

[0044] In addition, there are also robot grasping tasks based on target detection models, but target detection models can usually only detect term-level targets. For long text instructions containing multiple objects or complex action descriptions, target detection models are often unable to directly extract the names of the detected objects. For more complex grasping operations, target detection models are also unable to understand complex or abstract user needs. For example, when a user gives the instruction "Please pour me a cup of tea," this instruction not only includes the action of grasping an object, but also involves the operation of moving the object from one location to another. The target detection model cannot accurately understand such complex or abstract user needs and therefore cannot directly perform long-range grasping tasks.

[0045] To address this issue, the present invention provides a task execution method that dynamically decomposes different pending tasks into multiple simple subtasks using item information and execution instructions, demonstrating strong generalization capabilities. Furthermore, even when faced with long text execution instructions containing multiple execution objects or complex action descriptions, the present invention can accurately identify these multiple execution objects or complex actions through item information and execution instructions, and decompose the pending tasks into multiple subtasks containing a single execution object or simple action. In other words, the present invention can accurately understand the task requirements even when faced with long text execution instructions.

[0046] This method can be applied to robots performing grasping tasks, and may also be applied to other intelligent agents performing other tasks (such as material handling tasks). To facilitate understanding of the solutions of the present invention, the following embodiments are described using the application of robots performing grasping tasks as an example.

[0047] Figure 1 It is a flowchart of the task execution method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0048] Step 110: Identify object information in the environment based on the environment image corresponding to the task to be performed.

[0049] Here, environmental images refer to image data of the robot's surroundings, which can be captured by a camera installed on the robot (such as a binocular 3D camera). Object information refers to the object information identified from the environmental images. This object information is used to represent the object's attributes, which may include the object's name, type, specifications, location, color, and purpose.

[0050] Optionally, the attribute information included in the item information can be determined based on the execution instruction sent by the user. That is, if the execution instruction involves a certain attribute, the item information must include the corresponding attribute information. For example, if the execution instruction is "grab the red apple on the table," the "red" in the execution instruction involves the color attribute, so the item information must include the color attribute.

[0051] Since the attribute information included in the item information is associated with the execution instruction, the execution instruction is used to represent the user's true intention, so the identified item information can also accurately match the user's intention.

[0052] Step 120: Based on the execution instruction and item information used to describe the task to be executed, decompose the task to be executed into multiple subtasks, each subtask including an execution object and an execution action of the execution object.

[0053] Specifically, an execution instruction is a user-sent command requesting the robot to perform a specific task, such as "grab the red object" or "pour me a cup of tea." Subtasks break down a task into smaller, more specific units. Each subtask contains a specific execution object and a corresponding execution action. The execution object refers to the object to be manipulated in the subtask, and the execution action refers to the operation performed on the execution object, such as grasping or moving it.

[0054] For example, the task to be executed pointed to by the execution instruction "Please pour me a cup of tea" can be decomposed into multiple subtasks such as "grab the teacup", "grab the tea leaves", "put the tea leaves into the teacup", and "pour hot water".

[0055] It can be seen that the embodiments of the present invention can decompose long-term tasks to be executed into multiple simple sub-tasks through item information and execution instructions, and dynamically decompose tasks to be executed according to specific environmental information and execution instructions without relying on predefined instruction sets, thereby being able to handle more complex and flexible execution instructions and adapt to complex and changing application scenarios.

[0056] In addition, when faced with a long text execution instruction containing multiple execution objects or complex action descriptions, the embodiment of the present invention can also accurately identify multiple execution objects or complex actions through object information and execution instructions, and decompose the task to be executed into multiple subtasks containing a single execution object or simple action. That is, when faced with a long text execution instruction, the embodiment of the present invention can also accurately understand the task requirements.

[0057] The decomposition of tasks to be executed can be expressed by the following formula:

[0058]

[0059] Among them, Task represents the complete task sequence corresponding to the task to be executed (that is, the set of all subtasks), action(i) represents the i-th execution action, and target(i) represents the execution object of the i-th execution action.

[0060] Step 130: Identify the posture of each execution object, and execute the corresponding subtask based on the posture and execution action of each execution object.

[0061] Specifically, pose is used to characterize the position of each object in space. This pose can be a 3D pose (used to describe the position of each object in three-dimensional space) or a 6D pose (used to describe the position and posture of each object in three-dimensional space). Common pose recognition methods include the center of gravity method, the minimum envelope method, the point cloud registration method, and the geometric center method.

[0062] For example, based on the pose information of the object being grasped, a path can be planned from the robot's current position to the object's location. This path must avoid obstacles and ensure the robot can successfully reach the object. After reaching the object, the robot's posture must be adjusted to ensure that the end effector (such as a mechanical gripper) can properly contact the object and grasp it. After the posture adjustment is complete, the robot executes the grasping action using the end effector. After successfully grasping the object, the robot moves the object to a designated location or performs subsequent actions.

[0063] The task execution method provided by the embodiment of the present invention can decompose the task to be executed into multiple simple subtasks through item information and execution instructions, and realize dynamic decomposition of the task to be executed based on specific environmental information and execution instructions without relying on a predefined instruction set, thereby being able to handle more complex and flexible execution instructions and adapt to complex and changing application scenarios. In addition, when faced with a long text execution instruction containing multiple execution objects or complex action descriptions, the embodiment of the present invention can also accurately identify multiple execution objects or complex actions through item information and execution instructions, and decompose the task to be executed into multiple subtasks containing a single execution object or simple action. In other words, when faced with a long text execution instruction, the embodiment of the present invention can also accurately understand the task requirements.

[0064] Based on the above embodiment, based on the execution instructions and item information used to describe the task to be executed, the task to be executed is decomposed into multiple subtasks, including:

[0065] Convert execution instructions into execution text;

[0066] Based on the item information, extracting target text associated with the item information from the execution text;

[0067] Based on the target text, the task to be performed is broken down into multiple subtasks.

[0068] Specifically, the execution instruction is usually sent by the user in the form of voice, that is, the execution instruction is usually a voice instruction. In this case, after receiving the execution instruction, the execution instruction can be voice recognized to obtain the corresponding execution text, that is, the execution text is used to represent the semantic information of the execution instruction.

[0069] Considering that execution instructions are usually sent by users when they need to perform specific operations on the execution objects in the current environment, the execution instructions must contain information about the execution objects. Execution instructions can explicitly indicate the execution object, such as in the execution instruction "Please help me get an apple", the explicit indication is that "apple" is the execution object. Execution instructions can also indirectly indicate the execution object, or it can be understood that the execution object needs to be inferred based on the execution instruction and the object information. For example, in the execution instruction "Please help me get a red thing from the table", the "red thing" in the execution instruction does not specify a specific object, but the object information indicates that the only red thing on the table is an apple. Then, combined with the object information, reasoning is performed and the execution object pointed to by the execution instruction is "apple".

[0070] To this end, embodiments of the present invention extract target text associated with the item information from the execution text based on the item information. This target text includes the execution object targeted by the execution instruction and the execution action performed on the execution object. The execution text can be searched for a matching target entity based on the item information, and the associated text corresponding to the target entity can be used as the target text, such as text describing the execution action performed on the target entity.

[0071] For example, the execution text is "Please ask the robot to grab a red box from the shelf and place it on the conveyor belt." In the execution text, the item information directly related to the grabbing task is "red box." Based on the item information "red box," the target text extracted from the execution text can be "After grabbing the red box on the shelf, move to the conveyor belt and place the red box on the conveyor belt."

[0072] As can be seen from this, the target text clearly indicates the execution object and the execution object's execution action. Based on the target text, the task to be executed can be broken down into multiple subtasks. For example, if the target text is "grab the red box on the shelf and place it on the conveyor belt," the corresponding subtasks are "grab the red box," "move it to the conveyor belt," and "place the red box on the conveyor belt."

[0073] Based on any of the above embodiments, extracting target text associated with the item information from the execution text based on the item information includes:

[0074] Perform entity recognition on the execution text to obtain at least one execution entity;

[0075] Based on the item information, determining a target entity matching the object in the environment from at least one execution entity;

[0076] Extracting action entities associated with target entities from execution text;

[0077] Construct the target text based on the action entity and the execution entity.

[0078] Specifically, the execution entity is typically the execution object, and the execution entity can be understood as a noun phrase in the execution text. If the execution text explicitly indicates the execution object, the execution entity may correspond to the execution object's name, such as the execution entities "apple" and "table." If the execution text indirectly indicates the execution object, the execution entity may correspond to other attributes of the execution object (such as shape and size). For example, the execution entity "red fruit" corresponds to the color and type of the execution object.

[0079] If the execution entity corresponds to the name of the execution object, the execution entity can be matched with the name in the item information to obtain the target entity. If the execution entity corresponds to other attributes of the execution object, the execution entity can be matched with other attributes in the item information to obtain the target entity.

[0080] After determining the target entity, the action entity related to the target entity can be extracted from the execution text. The action entity is usually a verb phrase used to describe the execution action on the target entity, such as "grab", "move", "put in", etc.

[0081] If there are multiple target entities or multiple action entities, the corresponding subtext can be determined based on each target entity and the corresponding action entity, and the subtexts can be spliced ​​together to obtain the target text in the order of executing the actions.

[0082] For example, the execution text is "Please grab a red box from the shelf and place it on the conveyor belt." Based on the semantic information of the execution text, it can be known that before grabbing the red box, you need to move to the front of the shelf, and before placing the red box on the conveyor belt, you also need to move to the conveyor belt. Therefore, the target entities in the execution text include "shelf", "red box", and "conveyor belt." The action entities corresponding to "shelf" include "move", the action entities corresponding to "red box" include "grab", the action entities corresponding to conveyor belt include "move", and the action entity corresponding to "red box" is "place". The target text can be "Move to the front of the shelf, grab the red box, move to the conveyor belt, and place the red box on the conveyor belt."

[0083] Based on any of the above embodiments, a target text associated with the item information is extracted from the execution text, and based on the target text, the task to be executed is decomposed into multiple subtasks, including:

[0084] Constructing a task decomposition prompt text for prompting task decomposition requirements;

[0085] Based on the task decomposition model, the task decomposition prompt text is applied to extract the target text associated with the item information from the execution text. Based on the target text, the task to be executed is decomposed into multiple subtasks.

[0086] Specifically, the task decomposition prompt text is intended to guide how to identify key information from a complex execution text and decompose the task to be executed into multiple subtasks based on this information. The task decomposition prompt text can be prompt text.

[0087] After constructing the task decomposition prompt text, the task decomposition prompt text can be input into the task decomposition model. The task decomposition model extracts the target text associated with the item information from the execution text according to the requirements and instructions in the task decomposition prompt text, and decomposes the task to be executed into multiple subtasks based on the target text.

[0088] The robot's preset action library is represented by "action", each "action" corresponds to a physical behavior of the robot, the execution target of the action is represented by "target", which indicates the execution object of the action, and the robot's interactive response to the task instruction is represented by "response". The task decomposition result of the model is output in the JSON format {actions: [], response: ""}. The task decomposition prompt text example is as follows:

[0089] You are my robot assistant, with built-in action functions. Based on your image recognition results, you will perform corresponding actions according to my instructions to achieve the task. Please output the function to be run and your response to me in JSON format.

[0090] #### Built-in function description

[0091] 1. **Robot body movements:**

[0092] - "stand": When receiving the "stand" command, the robot will adjust from any posture to a standing posture. Example of calling method: {"action":"stand","target":""}.

[0093] - "squat": When receiving the "squat" command, the robot will adjust from any posture to squat. Example of calling method: {"action":"squat","target":""}.

[0094] - "walk": When receiving the "walk" command, the robot will adjust from any posture to standing still. Example of calling the method: {"action":"navigation","target":""}.

[0095] 2. **Robot Movement:**

[0096] - "navigation": When receiving the "navigation" command, the robot moves to the vicinity of the specified target location. Example of calling method: {"action":"navigation","target":"chair"}.

[0097] 3. **Robot crawling:**

[0098] - "grasp": When receiving the "grasp" command, the robot grabs a target object. Example of calling method: {"action":"grasp","target":"bottle"}.

[0099] 3. **Robot grabs and delivers the item:**

[0100] - "give": When receiving the "give" command, the robot gives the item to the customer or places it on an item. Example of calling the method: {"action":"give","target":"customer"}.

[0101] #### **Output format, please strictly follow the order of each symbol and do not change it**

[0102] {"response":"response text"}{"action":"action command name","target":"command target"}... (There is no limit on the length of the reply. If you do not need to execute the action command, you can just reply with {"response":"response text"})

[0103] #### The example output only lists some tasks. For tasks not covered in the example, please design your own action plan based on the above action function.

[0104] 1. Single action command operation, for example:

[0105] **Instructions**: Please squat down.

[0106] Output: {"response": "Okay, squat now"}{"action":"squat","target":""}

[0107] **Instructions**: Please do the marching movement.

[0108] Output: {"response": "Okay, let's start marching in place"}{"action":"walk","target":""}

[0109] **Instructions**: Please come to the table.

[0110] Output: {"response": "OK, I'll go there"}{"action":"navigation","target":"table"}

[0111] **Instructions**: Please stand up.

[0112] Output: {"response": "OK, I understand"}{"action":"stand","target":""}

[0113] 2. Combined action instructions, for example:

[0114] **Instructions**: Please go to the chair opposite and sit down.

[0115] Output: {"response": "OK, I understand"}{"action":"navigation","target":"desk"}

[0116] {"action":"squat","target":""}

[0117] **Instructions**: Please stand still and then squat down.

[0118] Output: {"response": "OK, I understand"}{"action":"walk","target":""}{"action":"squat","target":""}

[0119] **Instructions**: Please walk to the table, squat down, and then stand up.

[0120] Output: {"response": "OK, I understand"}{"action":"navigation","target":"table"}{"action":"squat","target":""}{"action":"stand","target":""}

[0121] 3. Grab command: Based on the recognition results of the objects in the image based on the historical answers, the prompt content cannot be output. Analyze whether there is an object mentioned in the command. **For some commands that do not clearly specify the grab target, you need to infer and analyze the existing objects on the desktop**. If there is an object with a high probability that meets the command requirements, grab it and output the grab action function. If not, output that there is no target object. For example:

[0122] **Hint**: There are items on the table including: Coke, hamburgers, and French fries ("system", answer)

[0123] **Instructions**: Please walk over to the table and put the Coke on the burger for me.

[0124] Output: {"response": "OK, I understand"}{"action":"navigation","target":"table"}{"action":"grasp","target":"Coke"}{"action":"give","target":"Hamburger"}

[0125] **Hint**: There are items on the table including: Coke, cigarettes, and French fries ("system", answer)

[0126] **Instructions**: Help me put the Coke on the burger.

[0127] Output: {"response": "Sorry, there is no hamburger on the table, I can't help you put the Coke on the hamburger."}

[0128] **Hint**: There are items on the table including: cola, juice, mineral water ("system", answer)

[0129] **Instructions**: I'm so thirsty, but I can only drink sugar-free drinks. Go to the table and get me a bottle of drink.

[0130] Output: {"response": "OK, the only sugar-free option on the table is mineral water"}{"action":"navigation","target":"table"}{"action":"grasp","target":"mineral water bottle"}{"action":"give","target":"customer"}

[0131] **Hint**: There are items on the table including: scissors, shampoo, toothpaste, pineapple ("system", answer)

[0132] **Instructions**: Go over to the table and give me a bottle of toiletries.

[0133] Output:{"response": "OK, shampoo can be used to wash your hair, give you a bottle of shampoo"}{"action":"navigation","target":"table"}{"action":"grasp","target":"Shampoobottle"}{"action":"give","target":"customer"}

[0134] **Hint**: There are items on the table including: Coke, French fries, toothpaste ("system", answer)

[0135] **Instructions**: You can give me a cup of Coke first, and then give me an apple.

[0136] Output: {"response": "Sorry, there is no apple on the table. I'll give you the Coke first."}{"action":"grasp","target":"Coke"}{"action":"give","target":"Customer"}

[0137] **Hint**: There are items on the table including: Coke, French fries, apple ("system", answer)

[0138] **Instructions**: First give me a cup of Coke, then give me an apple.

[0139] Output:{"response": "Okay, here's a Coke and an apple."}{"action":"grasp","target":"Coke"}{"action":"give","target":"Customer"}{"action":"grasp","target":"apple"}{"action":"give","target":"Customer"}

[0140] **Hint**: There are items on the table including: vases, plates ("system", answer)

[0141] **Instructions**: Go to the table, squat down, and bring me a bottle of Coke

[0142] Output: {"response": "I'll go to the table, but I'm sorry, there's no Coke on the table, so I can't get it for you."}

[0143] **Hint**: There are items on the table including: vase, plate, cola ("system", answer)

[0144] **Instructions**: Go to the table, squat down, and bring me a bottle of Coke

[0145] Output:{"response": "OK, I see. There's a Coke on the table."}{"action":"navigation","target":"table"}{"action":"squat","target":""}{"action":"grasp","target":"Coke"}{"action":"give","target":"customer"}

[0146] **Hint**: There are items on the table including: apple, banana, plate, basket, cola, bread ("system", answer)

[0147] **Instructions**: Please give me the largest item on the table

[0148] Output: {"response": It's my pleasure to serve you. I looked at the table and the bread is the largest, so I'll give it to you.}{"action":"grasp","target":"bread"}{"action":"give","target":"customer"}

[0149] **Hint**: There are items on the table including: apples, bananas, plates, baskets, cola, bread, and a plastic elephant model ("system", answer)

[0150] **Command**: Please bring me a toy

[0151] Output: {"response": It's a pleasure to help you. The plastic elephant model on the table is most likely a toy.}{"action":"grasp","target":"Plastic toy elephant"}{"action":"give","target":"Customer"}

[0152] **Hint**: There are items on the table including: apples, bananas, plates, baskets, cola, bread, and a plastic elephant model ("system", answer)

[0153] **Command**: Please bring me some food

[0154] Output: {"response": It's a pleasure to serve you. The apples, bananas, and bread on the table are all food.}{"action":"grasp","target":"apples"}{"action":"give","target":"customers"}{"action":"grasp","target":"bananas"}{"action":"give","target":"customers"}{"action":"grasp","target":"bread"}{"action":"give","target":"customers"}

[0155]

Execution text: XXXXXX

[0156] }

[0157] Based on any of the above embodiments, identifying information about objects in the environment based on an environment image corresponding to the task to be performed includes:

[0158] Constructing identification prompt text for prompting item information identification requirements;

[0159] Based on the recognition model, the recognition prompt text is applied to identify the object information from the environment image.

[0160] Specifically, the recognition prompt text is used to clarify the object information that needs to be recognized in the environmental image, that is, the recognition prompt text is used to prompt the recognition model to focus on the key content in the environmental image. For example, the recognition prompt text can clarify the category, color, shape and other information of the object that needs to be recognized in the environmental image.

[0161] The following are examples of recognition prompt text:

[0162] "Please describe what objects are in the environment image, focusing on whether the objects mentioned in the execution instructions exist in the environment image. You are required to tell me every object that appears in the environment image, and the description of the objects should be as rich as possible. The response format is {**Hint**: The objects in the image include: Item 1, Item 2, Item 3}, Example: {**Hint**: The objects in the image include: yellow banana, laptop, cola (drink), red apple (food)} Example: {**Hint**: The objects in the image include: pencil (wooden), lemon (very sour fruit), hamburger, telescope (black)}".

[0163] After constructing the recognition prompt text, the environment image and the recognition prompt text are input into the recognition model. The recognition model accurately identifies the corresponding object information from the environment image based on the prompt content in the recognition prompt text. The recognition model can be a large language model (LLM). An LLM is a natural language processing (NLP) model with a large number of parameters. The number of model parameters and / or the complexity of the model structure exceed a preset threshold. The model processes large amounts of text data during training and has the ability to understand and generate natural language. For example, a large language model can include the Spark large model.

[0164] In addition, before applying the recognition hint text based on the recognition model to identify the object information from the environment image, the size of the environment image is resized to a fixed size (384, 384) and then encoded into 64-bit UTF-8 format, as shown below:

[0165]

[0166]

[0167] Resizing the environment image to a fixed size of (384, 384) ensures that all input images have the same resolution, facilitating subsequent processing and helping to reduce the complexity and time consumption caused by image size differences. Furthermore, encoding the image data in 64-bit UTF-8 format ensures data consistency across different platforms and devices, avoiding errors or garbled characters caused by character encoding differences.

[0168] Based on any of the above embodiments, identifying the posture of each execution object includes:

[0169] Identify and obtain the detection frame of each execution object from the environment image;

[0170] Based on the detection box of each execution object, the pose of each execution object is determined.

[0171] For example, an environment image and the terms for each execution object (e.g., the terms corresponding to the name) can be input into an object detection model. The object detection model identifies the execution objects in the environment image that match the terms for each execution object and outputs a detection box for the execution object. The object detection model can be a Grounding Dino model.

[0172] Next, the environment image and the corner coordinates of each detection frame can be input into an image segmentation model, which segments the area corresponding to the execution object in each detection frame and generates a corresponding mask. The image segmentation model can be an EfficientVIT-SAM model.

[0173] Finally, the mask and the environment image are fed into a pose estimation model, which determines the pose of each object, such as a 6D pose. This allows action planning based on the pose and position of each object to sequentially complete the corresponding subtasks. The pose estimation model can be a Foundation Pose model.

[0174] Based on any of the above embodiments, each execution action exists in a preset action library.

[0175] Specifically, the preset action library is a data set that stores a variety of pre-defined and programmed action instructions or behavior patterns. These action instructions or behavior patterns are usually formulated according to the design purpose, functional requirements and application scenarios of the robot.

[0176] By presetting the execution actions in the action library, the robot can directly call the actions in the preset action library when performing tasks without having to create each action from scratch, thereby shortening the task execution time and improving overall efficiency. Especially in tasks that require repeated execution of certain actions, the use of a preset action library can significantly reduce repetitive work.

[0177] Based on any of the above embodiments, Figure 2 This is a flow chart of another task execution method provided by the present invention, such as Figure 2 As shown, the method includes:

[0178] Construct a recognition prompt text to indicate the object information recognition requirement and obtain the environment image corresponding to the task to be performed. Input the recognition prompt text and the environment image into the recognition model, which uses the recognition prompt text to identify the object information in the environment from the environment image.

[0179] At the same time, the user's execution instructions are received, and based on the speech recognition model, the execution instructions are voice recognized to obtain the corresponding execution text.

[0180] A task decomposition prompt is constructed to indicate task decomposition requirements. This prompt, along with item information and execution text, is then fed into the task decomposition model. The model then decomposes the task into multiple subtasks, each of which includes an execution object and its corresponding execution action. Furthermore, the task decomposition model can also access the user's historical records of previous task decompositions. Based on the records of successful and failed decompositions in the historical records, the task decomposition algorithm can be adjusted and optimized to ultimately arrive at the optimal task decomposition solution.

[0181] The environment image and the entries of each execution object (such as the entries corresponding to the name) are input into the target detection model, and the target detection model identifies the execution objects in the environment image that match the entries of each execution object and outputs the detection box of the execution object.

[0182] The environment image and the coordinates of the corner points of each detection frame are input into the image segmentation model, which segments the area corresponding to the execution object in each detection frame and generates a corresponding mask.

[0183] Finally, the mask and the environment image are input into the posture estimation model, which determines the posture of each execution object, such as the 6D posture. This allows action planning based on the posture and position of each execution object to complete the corresponding subtasks in sequence.

[0184] Furthermore, a test instruction set was constructed to test the aforementioned task execution method. The test instruction set included: action instructions {"Please squat," "Please walk to the table and then mark time,"...}, action + grasping instructions {"Dance for me and then give me the apple," "Walk over to the table and get me a bottle of Coke,"...}, explicit grasping instructions {"Put the burger on the plate for me," "Put the flowers in the bottle for me,"...}, and fuzzy grasping instructions {"Clean up the table for me," "The largest object on the table,"...}. Fifteen instructions of each type were set based on the desktop scenario, for a total of 60 instructions for testing. The recognition model parameters were set to "temperature" = 0.3, "max_tokens" = 150, and the task decomposition model parameters were set to "temperature" = 0.1, "max_tokens" = 200. The model was fed with desktop image information and instruction content, and the output was a task decomposition result in JSON format. Table 1 shows the test results.

[0185] Table 1

[0186]

[0187] The task execution device provided by the present invention is described below. The task execution device described below and the task execution method described above can be referenced to each other.

[0188] Based on any of the above embodiments, Figure 3 It is a structural diagram of the task execution device provided by the present invention, such as Figure 3 As shown, the device includes:

[0189] The recognition unit 310 is used to recognize object information in the environment based on the environment image corresponding to the task to be performed;

[0190] A decomposition unit 320 is configured to decompose the task to be executed into a plurality of subtasks based on the execution instruction and item information used to describe the task to be executed, each subtask including an execution object and an execution action of the execution object;

[0191] The execution unit 330 is used to identify the position and posture of each execution object, and execute the corresponding subtask based on the position and posture of each execution object and the execution action.

[0192] Based on any of the above embodiments, the task to be performed is decomposed into multiple subtasks based on the execution instructions and item information used to describe the task to be performed, including:

[0193] Convert execution instructions into execution text;

[0194] Based on the item information, extracting target text associated with the item information from the execution text;

[0195] Based on the target text, the task to be performed is broken down into multiple subtasks.

[0196] Based on any of the above embodiments, extracting target text associated with the item information from the execution text based on the item information includes:

[0197] Perform entity recognition on the execution text to obtain at least one execution entity;

[0198] Based on the item information, determining a target entity matching the object in the environment from at least one execution entity;

[0199] Extracting action entities associated with target entities from execution text;

[0200] Construct the target text based on the action entity and the execution entity.

[0201] Based on any of the above embodiments, a target text associated with the item information is extracted from the execution text, and based on the target text, the task to be executed is decomposed into multiple subtasks, including:

[0202] Constructing a task decomposition prompt text for prompting task decomposition requirements;

[0203] Based on the task decomposition model, the task decomposition prompt text is applied to extract the target text associated with the item information from the execution text. Based on the target text, the task to be executed is decomposed into multiple subtasks.

[0204] Based on any of the above embodiments, identifying information about objects in the environment based on an environment image corresponding to the task to be performed includes:

[0205] Constructing identification prompt text for prompting item information identification requirements;

[0206] Based on the recognition model, the recognition prompt text is applied to identify the object information from the environment image.

[0207] Based on any of the above embodiments, identifying the posture of each execution object includes:

[0208] Identify and obtain the detection frame of each execution object from the environment image;

[0209] Based on the detection box of each execution object, the pose of each execution object is determined.

[0210] Based on any of the above embodiments, each execution action exists in a preset action library.

[0211] Based on any of the above embodiments, the present invention further provides a robot, comprising:

[0212] A camera is provided on the robot body (such as the robot head) and the task execution device as described in any of the above embodiments, and the camera is used to collect environmental images.

[0213] As described in any of the above embodiments, the task execution device can decompose the task to be executed into multiple simple subtasks using object information and execution instructions. This allows for dynamic decomposition of the task to be executed based on specific environmental information and execution instructions, without relying on a predefined instruction set. This enables the robot to handle more complex and flexible execution instructions and adapt to complex and ever-changing application scenarios. Furthermore, when faced with a long text execution instruction containing multiple execution objects or complex action descriptions, the robot can accurately identify the multiple execution objects or complex actions using object information and execution instructions, and decompose the task to be executed into multiple subtasks containing a single execution object or simple action.

[0214] Figure 4 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communications bus 440. The processor 410 may call logic instructions in the memory 430 to execute a task execution method, which includes: identifying object information in the environment based on an environment image corresponding to the task to be executed; decomposing the task to be executed into multiple subtasks based on an execution instruction describing the task to be executed and the object information, each subtask including an execution object and an execution action of the execution object; identifying the position and posture of each execution object, and executing the corresponding subtask based on the position and posture of each execution object and the execution action.

[0215] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0216] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the task execution method provided by the above methods, which includes: based on the environment image corresponding to the task to be executed, identifying the object information in the environment; based on the execution instructions used to describe the task to be executed and the object information, decomposing the task to be executed into multiple subtasks, each subtask including an execution object and the execution action of the execution object; identifying the posture of each execution object, and executing the corresponding subtask based on the posture and execution action of each execution object.

[0217] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the task execution method provided by the above-mentioned methods, the method comprising: identifying object information in the environment based on an environment image corresponding to the task to be executed; decomposing the task to be executed into multiple subtasks based on an execution instruction for describing the task to be executed and the object information, each subtask including an execution object and an execution action of the execution object; identifying the posture of each execution object, and executing the corresponding subtask based on the posture and execution action of each execution object.

[0218] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0219] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A task execution method, characterized in that: include: Based on the environment image corresponding to the task to be performed, identify the object information in the environment; Converting the execution instruction describing the task to be executed into an execution text; extracting a target text associated with the item information from the execution text based on the item information; Based on the target text, the task to be executed is decomposed into a plurality of subtasks, each subtask including an execution object and an execution action of the execution object; Identify the position and posture of each execution object, and execute the corresponding subtask based on the position and execution action of each execution object; The step of extracting a target text associated with the item information from the execution text and decomposing the task to be executed into a plurality of subtasks based on the target text includes: Constructing a task decomposition prompt text for prompting task decomposition requirements; Based on the task decomposition model, the task decomposition prompt text is applied to extract the target text associated with the item information from the execution text, and based on the target text, the task to be executed is decomposed into multiple subtasks.

2. The task execution method according to claim 1, characterized in that: The step of extracting a target text associated with the item information from the execution text based on the item information includes: Performing entity recognition on the execution text to obtain at least one execution entity; determining, based on the item information, a target entity that matches the object in the environment from the at least one execution entity; extracting an action entity associated with the target entity from the execution text; The target text is constructed based on the action entity and the execution entity.

3. The task execution method according to any one of claims 1 to 2, characterized in that: The identifying of object information in the environment based on the environment image corresponding to the task to be performed includes: Constructing identification prompt text for prompting item information identification requirements; Based on the recognition model, the recognition prompt text is applied to identify and obtain the object information from the environment image.

4. The task execution method according to any one of claims 1 to 2, characterized in that: The identifying the posture of each execution object includes: Identify and obtain detection frames of various execution objects from the environment image; Based on the detection box of each execution object, the pose of each execution object is determined.

5. The task execution method according to any one of claims 1 to 2, characterized in that: Each execution action exists in the preset action library.

6. A task execution device, characterized in that: include: A recognition unit, configured to recognize object information in the environment based on an image of the environment corresponding to the task to be performed; a decomposition unit, configured to convert the execution instruction describing the task to be executed into an execution text; and extract a target text associated with the item information from the execution text based on the item information; Based on the target text, the task to be executed is decomposed into a plurality of subtasks, each subtask including an execution object and an execution action of the execution object; An execution unit, configured to identify the position and posture of each execution object and execute a corresponding subtask based on the position and posture of each execution object and the execution action; The step of extracting a target text associated with the item information from the execution text and decomposing the task to be executed into a plurality of subtasks based on the target text includes: Constructing a task decomposition prompt text for prompting task decomposition requirements; Based on the task decomposition model, the task decomposition prompt text is applied to extract the target text associated with the item information from the execution text, and based on the target text, the task to be executed is decomposed into multiple subtasks.

7. A robot, characterized in that: include: A camera provided on the robot body and the task execution device as claimed in claim 6, wherein the camera is used to collect the environmental image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the task execution method according to any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task execution method according to any one of claims 1 to 5 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the task execution method according to any one of claims 1 to 5 is implemented.

Images