Task execution method and device and robot

Abstract

The invention provides a task execution method and device and a robot, and the method comprises the steps: recognizing article information in an environment based on an environment image corresponding to a to-be-executed task; based on an execution instruction used for describing the to-be-executed task and the article information, the to-be-executed task is decomposed into a plurality of sub-tasks, and each sub-task comprises an execution object and an execution action of the execution object; and identifying the pose of each execution object, and executing the corresponding subtask based on the pose of each execution object and the execution action. The task execution method, the task execution device and the robot provided by the invention do not depend on a predefined instruction set, so that more complex and more flexible execution instructions can be processed, and the method and the device are suitable for complex and changeable application scenes. In addition, the task requirement can be accurately understood when a long text execution instruction is faced.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to a task execution method, device and robot. Background Art

[0002] In the intersection of artificial intelligence and robotics, robotic grasping technology, as a key link in realizing intelligent robot operation, has made significant progress in recent years. Traditional robotic grasping methods mainly rely on hierarchical perception models, which generate grasping poses by identifying the target object and estimating the six-dimensional (6D) pose, thereby guiding the robot to perform grasping tasks.

[0003] However, robots usually work based on predefined instruction sets, that is, robots can only recognize and execute instructions within the scope of the predefined instruction sets. Once the user gives an instruction out of the range, the robot cannot make a correct response. Summary of the invention

[0004] The present invention provides a task execution method, device and robot to solve the defects in the prior art.

[0005] The present invention provides a task execution method, comprising the following steps: Based on the environment image corresponding to the task to be performed, identify the object information in the environment; Based on the execution instruction for describing the task to be executed and the item information, the task to be executed is decomposed into a plurality of subtasks, each subtask including an execution object and an execution action of the execution object; Identify the position and posture of each execution object, and execute the corresponding subtask based on the position and posture of each execution object and the execution action.

[0006] According to a task execution method provided by the present invention, based on the execution instruction used to describe the task to be executed and the item information, the task to be executed is decomposed into multiple subtasks, including: Converting the execution instruction into an execution text; Based on the item information, extracting a target text associated with the item information from the execution text; Based on the target text, the task to be performed is decomposed into a plurality of subtasks.

[0007] According to a task execution method provided by the present invention, extracting a target text associated with the item information from the execution text based on the item information includes: Performing entity recognition on the execution text to obtain at least one execution entity; Based on the item information, determining a target entity matching the object in the environment from the at least one execution entity; Extracting an action entity associated with the target entity from the execution text; The target text is constructed based on the action entity and the execution entity.

[0008] According to a task execution method provided by the present invention, the step of extracting a target text associated with the item information from the execution text and decomposing the task to be executed into a plurality of subtasks based on the target text includes: Constructing a task decomposition prompt text for prompting task decomposition requirements; Based on the task decomposition model, the task decomposition prompt text is applied to extract the target text associated with the item information from the execution text, and based on the target text, the task to be executed is decomposed into multiple subtasks.

[0009] According to a task execution method provided by the present invention, the step of identifying object information in the environment based on an environment image corresponding to the task to be executed includes: Constructing identification prompt text for prompting item information identification requirements; Based on the recognition model, the recognition prompt text is applied to recognize and obtain the object information from the environment image.

[0010] According to a task execution method provided by the present invention, the identifying the position and posture of each execution object includes: Recognize and obtain detection frames of various execution objects from the environment image; Based on the detection box of each execution object, the position and pose of each execution object are determined.

[0011] According to a task execution method provided by the present invention, each execution action exists in a preset action library.

[0012] The present invention also provides a task execution device, comprising the following modules: A recognition unit, used to recognize information of objects in the environment based on an environment image corresponding to the task to be performed; a decomposition unit, configured to decompose the task to be executed into a plurality of subtasks based on the execution instruction used to describe the task to be executed and the item information, each subtask including an execution object and an execution action of the execution object; The execution unit is used to identify the position and posture of each execution object, and execute the corresponding subtask based on the position and posture of each execution object and the execution action.

[0013] The present invention also provides a robot, comprising: a camera arranged on a robot body and the task execution device as described above, wherein the camera is used to collect the environment image.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned task execution methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the task execution method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned task execution methods.

[0017] The task execution method, device and robot provided by the present invention can decompose the task to be executed into multiple simple subtasks through object information and execution instructions, and realize dynamic decomposition of the task to be executed according to specific environmental information and execution instructions, without relying on predefined instruction sets, so as to handle more complex and flexible execution instructions and adapt to complex and changeable application scenarios. In addition, when faced with a long text execution instruction containing multiple execution objects or complex action descriptions, the present invention can also accurately identify multiple execution objects or complex actions through object information and execution instructions, and decompose the task to be executed into multiple subtasks containing a single execution object or simple action, that is, when faced with a long text execution instruction, the present invention can also accurately understand the task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of the task execution method provided by the present invention.

[0020] Figure 2 It is a flowchart diagram of another task execution method provided by the present invention.

[0021] Figure 3 It is a structural schematic diagram of the task execution device provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In the field of robot grasping technology, the grasping operation of objects is mainly dependent on predefined instruction sets and target detection models. However, traditional grasping systems work based on predefined instruction sets, which means that the robot's grasping operation needs to be defined in advance. Only when the robot recognizes a voice instruction that fully matches the predefined instruction set will it perform the corresponding grasping action. For example, if the predefined instruction set only contains the "grab the red object" instruction, then when the user says "please help me pick up the red apple", the robot may not be able to recognize and execute this instruction because it does not fully match the predefined instruction. This limitation makes it difficult to generalize the robot's grasping operation and cannot adapt to complex and changing application scenarios.

[0025] In addition, there are also robot grasping tasks based on target detection models, but target detection models can usually only detect term-level targets. For long text instructions containing multiple objects or complex action descriptions, target detection models often cannot directly extract the name of the detection object from them. For more complex grasping operations, target detection models are also unable to understand complex or abstract user needs. For example, when a user gives the instruction "Please pour me a cup of tea", this instruction not only includes the action of grasping an object, but also involves the operation of moving the object from one location to another. The target detection model cannot accurately understand such complex or abstract user needs, and therefore cannot directly perform long-range grasping tasks.

[0026] In this regard, the present invention provides a task execution method, which dynamically decomposes different tasks to be executed into multiple simple subtasks through item information and execution instructions, and has strong generalization ability. In addition, when faced with a long text execution instruction containing multiple execution objects or complex action descriptions, the present invention can also accurately identify multiple execution objects or complex actions through item information and execution instructions, and decompose the tasks to be executed into multiple subtasks containing a single execution object or simple action, that is, when faced with a long text execution instruction, the present invention can also accurately understand the task requirements.

[0027] The method can be applied to robots to perform grasping tasks, and may also be applied to other intelligent agents to perform other tasks (such as material handling tasks). In order to facilitate understanding of the solution of the present invention, the following embodiments are all described by taking the application of robots to perform grasping tasks as an example.

[0028] Figure 1 is a flowchart of the task execution method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0029] Step 110: Identify information about objects in the environment based on the environment image corresponding to the task to be performed.

[0030] Here, the environmental image refers to the image data of the robot's environment, which can be obtained through a camera (such as a binocular 3D camera) set on the robot body. Object information refers to the object information identified from the environmental image, which is used to characterize the attribute information of the object. The attribute information may include the name, type, specification, location, color, purpose, etc. of the object.

[0031] Optionally, the attribute information included in the item information can be determined based on the execution instruction sent by the user, that is, if the execution instruction involves a certain attribute, the item information needs to include the corresponding attribute information. For example, if the execution instruction is "grab the red apple on the table", the "red" in the execution instruction involves the color attribute, and the item information needs to include the color attribute.

[0032] Since the attribute information included in the item information is associated with the execution instruction, the execution instruction is used to represent the user's true intention, so that the identified item information can also accurately match the user's intention.

[0033] Step 120: Based on the execution instruction and item information used to describe the task to be executed, decompose the task to be executed into multiple subtasks, each subtask including an execution object and an execution action of the execution object.

[0034] Specifically, an execution instruction refers to an instruction sent by the user to request the robot to perform a specific task, such as "grab the red object", "please pour me a cup of tea", etc. Subtasks refer to the decomposition of the task to be executed into smaller and more specific task units. Each subtask contains a clear execution object and a corresponding execution action. The execution object refers to the object to be operated in the subtask, and the execution action refers to the operation on the execution object, such as grabbing, moving, etc.

[0035] For example, for the task to be executed pointed to by the execution instruction "Please pour me a cup of tea", it can be decomposed into multiple subtasks such as "grab the teacup", "grab the tea leaves", "put the tea leaves into the teacup", and "pour hot water".

[0036] It can be seen that the embodiments of the present invention can decompose long-range tasks to be executed into multiple simple sub-tasks through object information and execution instructions, and dynamically decompose tasks to be executed according to specific environmental information and execution instructions without relying on predefined instruction sets, thereby being able to handle more complex and flexible execution instructions and adapt to complex and changing application scenarios.

[0037] In addition, when faced with a long text execution instruction containing multiple execution objects or complex action descriptions, the embodiment of the present invention can also accurately identify multiple execution objects or complex actions through object information and execution instructions, and decompose the task to be executed into multiple subtasks containing a single execution object or simple action. That is, when faced with a long text execution instruction, the embodiment of the present invention can also accurately understand the task requirements.

[0038] Among them, the decomposition of tasks to be executed can be expressed by the following formula: Among them, Task represents the complete task sequence (i.e., the set of all subtasks) corresponding to the task to be executed, action(i) represents the i-th execution action, and target(i) represents the execution object of the i-th execution action.

[0039] Step 130: Identify the position and posture of each execution object, and execute the corresponding subtask based on the position and posture of each execution object and the execution action.

[0040] Specifically, the pose is used to characterize the position of each execution object in space. The pose can be a 3D pose (used to describe the position of each execution object in three-dimensional space) or a 6D pose (used to describe the position and posture of each execution object in three-dimensional space). Among them, commonly used pose recognition methods include the centroid method, the minimum envelope method, the point cloud registration method, the geometric center method, etc.

[0041] For example, based on the posture information of the execution object, the path of the robot from the current position to the execution object position can be planned. The path needs to avoid obstacles and ensure that the robot can smoothly reach the execution object position. After reaching the execution object position, the robot's own posture needs to be adjusted to ensure that the end effector (such as a mechanical gripper) can correctly contact the execution object and perform the grasping action. After the posture adjustment is completed, the robot performs the grasping action through the end effector. After successfully grasping the execution object, the robot moves the execution object to a specified position or performs subsequent actions.

[0042] The task execution method provided by the embodiment of the present invention can decompose the task to be executed into multiple simple subtasks through object information and execution instructions, and realize dynamic decomposition of the task to be executed according to specific environmental information and execution instructions without relying on predefined instruction sets, so as to be able to handle more complex and flexible execution instructions and adapt to complex and changing application scenarios. In addition, when faced with a long text execution instruction containing multiple execution objects or complex action descriptions, the embodiment of the present invention can also accurately identify multiple execution objects or complex actions through object information and execution instructions, and decompose the task to be executed into multiple subtasks containing a single execution object or simple action, that is, when faced with a long text execution instruction, the embodiment of the present invention can also accurately understand the task requirements.

[0043] Based on the above embodiment, based on the execution instruction and item information used to describe the task to be executed, the task to be executed is decomposed into multiple subtasks, including: Convert execution instructions into execution text; Based on the item information, extracting target text associated with the item information from the execution text; Based on the target text, the task to be performed is decomposed into multiple subtasks.

[0044] Specifically, the execution instruction is usually sent by the user in the form of voice, that is, the execution instruction is usually a voice instruction. In this case, after receiving the execution instruction, the execution instruction can be voice recognized to obtain the corresponding execution text, that is, the execution text is used to represent the semantic information of the execution instruction.

[0045] Considering that execution instructions are usually sent by users when they need to perform specific operations on the execution objects in the current environment, the execution instructions must contain the execution object information. The execution instructions can explicitly indicate the execution object, such as the execution instruction "Please help me get an apple", which explicitly indicates that "apple" is the execution object. The execution instruction can also indirectly indicate the execution object, which can also be understood as the need to infer the execution object based on the execution instruction and the object information. For example, the execution instruction "Please help me get a red thing from the table", the "red thing" in the execution instruction does not specify the specific object, and at this time the object information indicates that the only red thing on the table is apples, and then combined with the object information, the execution object pointed to by the execution instruction is "apple".

[0046] In this regard, the embodiment of the present invention extracts the target text associated with the item information from the execution text based on the item information, and the target text includes the execution object pointed to by the execution instruction and the execution action on the execution object. The corresponding target entity can be searched and matched in the execution text based on the item information, and the associated text corresponding to the target entity is used as the target text, such as the text used to describe the execution action on the target entity.

[0047] For example, the execution text is "Please ask the robot to grab a red box from the shelf and place it on the conveyor belt." In the execution text, the object information directly related to the grabbing task is "red box." Based on the object information "red box," the target text extracted from the execution text can be "After grabbing the red box on the shelf, move to the conveyor belt and place the red box on the conveyor belt."

[0048] It can be seen that the target text clearly indicates the execution object and the execution action of the execution object, and then the task to be executed can be decomposed into multiple subtasks based on the target text. For example, if the target text is "grab the red packaging box on the shelf and put it on the conveyor belt", the corresponding subtasks are "grab the red packaging box", "move to the conveyor belt", and "put the red packaging box on the conveyor belt".

[0049] Based on any of the above embodiments, based on the item information, extracting the target text associated with the item information from the execution text includes: Perform entity recognition on the execution text to obtain at least one execution entity; Based on the item information, determining a target entity matching the object in the environment from at least one execution entity; Extracting action entities associated with target entities from execution text; Construct the target text based on the action entity and the execution entity.

[0050] Specifically, the execution entity is usually the execution object, and the execution entity can be understood as a noun phrase in the execution text. If the execution text uses a clear text method to indicate the execution object, the execution entity may correspond to the name of the execution object, such as the execution entity "apple", "table", etc. corresponds to the name of the execution object. If the execution text uses an indirect method to indicate the execution object, the execution entity may correspond to other attributes of the execution object (such as shape, size, etc.), such as the execution entity "red fruit", which corresponds to the color and type attributes of the execution object.

[0051] If the execution entity corresponds to the name of the execution object, the execution entity can be matched with the name in the item information to obtain the target entity. If the execution entity corresponds to other attributes of the execution object, the execution entity can be matched with other attributes in the item information to obtain the target entity.

[0052] After determining the target entity, the action entity related to the target entity can be extracted from the execution text. The action entity is usually a verb phrase used to describe the execution action on the target entity, such as "grab", "move", "put in", etc.

[0053] If there are multiple target entities or multiple action entities, the corresponding sub-texts can be determined based on each target entity and the corresponding action entity, and the sub-texts can be concatenated to obtain the target text in the order of executing the actions.

[0054] For example, the execution text is "Please grab a red box from the shelf and place it on the conveyor belt." Based on the semantic information of the execution text, it can be known that before grabbing the red box, you need to move to the front of the shelf, and before placing the red box on the conveyor belt, you also need to move to the conveyor belt. Therefore, the target entities in the execution text include "shelf," "red box," and "conveyor belt." The action entities corresponding to "shelf" include "move," the action entities corresponding to "red box" include "grab," the action entities corresponding to conveyor belt include "move," and the action entity corresponding to "red box" is "place." Therefore, the target text can be "Move to the front of the shelf, grab the red box, move to the conveyor belt, and place the red box on the conveyor belt."

[0055] Based on any of the above embodiments, a target text associated with the item information is extracted from the execution text, and based on the target text, the task to be executed is decomposed into multiple subtasks, including: Constructing a task decomposition prompt text for prompting task decomposition requirements; Based on the task decomposition model, the task decomposition prompt text is applied to extract the target text associated with the item information from the execution text. Based on the target text, the task to be executed is decomposed into multiple subtasks.

[0056] Specifically, the task decomposition prompt text is intended to guide how to identify key information from a complex execution text and decompose the task to be executed into multiple subtasks based on the information. The task decomposition prompt text may be a prompt text.

[0057] After constructing the task decomposition prompt text, the task decomposition prompt text can be input into the task decomposition model. The task decomposition model extracts the target text associated with the item information from the execution text according to the requirements and instructions in the task decomposition prompt text, and decomposes the task to be executed into multiple subtasks based on the target text.

[0058] The robot preset action library is represented by "action", each "action" corresponds to a physical behavior of the robot, the execution target of the action is represented by "target", which indicates the execution object of the action, and the robot's interactive response to the task instruction is represented by "response". The task decomposition result of the model is output in the JSON format {actions: [], response: ""}. The task decomposition prompt text example is as follows: {You are my robot assistant, with some built-in action functions. Based on the recognition results of the objects in the image, you can perform corresponding operations according to my instructions to achieve the task. Please output the function to be run and the response to me in JSON format.

[0059] #### Built-in function description 1. **Robot body movements:** - "stand": When receiving the "stand" command, the robot will adjust from any posture to a standing posture. Example of calling the method: {"action":"stand","target":""}.

[0060] - "squat": When receiving the "squat" command, the robot will adjust from any posture to squat. Example of calling method: {"action":"squat","target":""}.

[0061] - "walk": When receiving the "walk" command, the robot will adjust from any posture to standing still. Example of calling method: {"action":"navigation","target":""}.

[0062] 2. **Robot movement:** - "navigation": When receiving the "navigation" command, the robot moves to the vicinity of the specified target position. Example of calling method: {"action":"navigation","target":"chair"}.

[0063] 3. **Robot crawling:** - "grasp": When receiving the "grasp" command, the robot grabs a target object. Example of calling method: {"action":"grasp","target":"bottle"}.

[0064] 3. **The robot grabs and delivers the item:** - "give": When receiving the "give" command, the robot will give the item to the customer or put it on an item. Example of calling method: {"action":"give","target":"customer"}.

[0065] #### **Output format, please strictly follow the order of each symbol and do not change it** {"response":"response text"}{"action":"action command name","target":"command target"}... (There is no limit on the length of the reply. If you do not need to execute the action command, you can just reply {"response":"response text"}) #### The example output only lists some tasks. For tasks not covered by the example, please design your own action plan based on the above action function. 1. Single action command operation, for example: **Instructions**: Please squat down.

[0066] Output:{"response": "OK, squat now"}{"action":"squat","target":""} **Instructions**: Please do the marching action.

[0067] Output:{"response": "OK, let's start walking in place"}{"action":"walk","target":""} **Instructions**: Please walk to the table.

[0068] Output:{"response": "OK, I'll go there"}{"action":"navigation","target":"table"} **Instructions**: Please stand up.

[0069] Output:{"response": "OK, I understand"}{"action":"stand","target":""} 2. Combined action instructions, for example: **Instructions**: Please go to the chair opposite and sit down.

[0070] Output:{"response": "OK, I understand"}{"action":"navigation","target":"desk"} {"action":"squat","target":""} **Instructions**: Please stand still and then squat down.

[0071] Output:{"response": "OK, I got it"}{"action":"walk","target":""}{"action":"squat","target":""} **Instructions**: Please walk to the table, squat down, and then stand up.

[0072] Output:{"response": "OK, I got it"}{"action":"navigation","target":"table"}{"action":"squat","target":""}{"action":"stand","target":""} 3. Grab command, based on the recognition results of the objects in the image based on the historical answers, the prompt content cannot be output, analyze whether there are any objects mentioned in the command, **For some commands that do not clearly specify the grab target, you need to reason and analyze the existing objects on the desktop**, if there is an object that is highly likely to meet the requirements of the command, grab it and output the grab action function, if not, output that there is no target object, for example: **Hint**: There are items on the table including: Coke, hamburgers, French fries ("system", answer) **Instructions**: Please come to the table and put the Coke on the burger for me.

[0073] Output:{"response": "OK, I got it"}{"action":"navigation","target":"table"}{"action":"grasp","target":"Coke"}{"action":"give","target":"hamburger"} **Hint**: There are items on the table including: Coke, cigarettes, French fries ("system", answer) **Instructions**: Help me put the Coke on the burger.

[0074] Output: {"response": "Sorry, there is no hamburger on the table, so I can't put the Coke on it for you."} **Tip**: The items on the table include: Coke, juice, mineral water ("system", answer) **Instructions**: I'm so thirsty, but I can only drink sugar-free drinks. Go to the table and get me a bottle of drink.

[0075] Output:{"response": "OK, the only sugar-free food on the table is mineral water"}{"action":"navigation","target":"table"}{"action":"grasp","target":"mineral water bottle"}{"action":"give","target":"customer"} **Hint**: There are items on the table including: scissors, shampoo, toothpaste, pineapple ("system", answer) **Instructions**: Go over to the table and give me a bottle of toiletries.

[0076] Output:{"response": "OK, shampoo can be used to wash your hair, give you a bottle of shampoo"}{"action":"navigation","target":"table"}{"action":"grasp","target":"Shampoobottle"}{"action":"give","target":"customer"} **Hint**: There are items on the table including: Coke, French fries, toothpaste ("system", answer) **Instructions**: You can give me a cup of Coke first, and then give me an apple.

[0077] Output:{"response": "Sorry, there is no apple on the table. I'll give you the Coke first."}{"action":"grasp","target":"Coke"}{"action":"give","target":"Customer"} **Hint**: There are items on the table including: Coke, French fries, apples ("system", answer) **Instructions**: First give me a Coke, then give me an apple.

[0078] Output:{"response": "OK, here's a Coke and an apple."}{"action":"grasp","target":"Coke"}{"action":"give","target":"Customer"}{"action":"grasp","target":"apple"}{"action":"give","target":"Customer"} **Hint**: There are items on the table including: vases, plates ("system", answer) **Instructions**: Go to the table, squat down, and bring me a bottle of Coke Output:{"response": "I will go to the table, but I'm sorry there is no Coke on the table, so I can't get it for you.".} **Hint**: There are items on the table including: vase, plate, Coke ("system", answer) **Instructions**: Go to the table, squat down, and bring me a bottle of Coke Output:{"response": "OK, I see, there is Coke on the table."}{"action":"navigation","target":"table"}{"action":"squat","target":""}{"action":"grasp","target":"Coke"}{"action":"give","target":"customer"} **Hint**: There are items on the table including: apples, bananas, plates, baskets, cola, bread ("system", answer) **Instructions**: Please give me the largest item on the table Output:{"response": It's my pleasure to serve you. I looked at the table and the bread is the largest, so I'll give it to you.}{"action":"grasp","target":"bread"}{"action":"give","target":"customer"} **Hint**: There are items on the table including: apples, bananas, plates, baskets, cola, bread, and a plastic elephant model ("system", answer) **Command**: Please bring me a toy Output:{"response": It's a pleasure to serve you. The plastic elephant model on the table is most likely a toy.}{"action":"grasp","target":"Plastic toy elephant"}{"action":"give","target":"Customer"} **Hint**: There are items on the table including: apples, bananas, plates, baskets, cola, bread, and a plastic elephant model ("system", answer) **Command**: Please bring me some food Output:{"response": It's a pleasure to serve you. The apples, bananas and bread on the table are all food.}{"action":"grasp","target":"apples"}{"action":"give","target":"customers"}{"action":"grasp","target":"bananas"}{"action":"give","target":"customers"}{"action":"grasp","target":"bread"}{"action":"give","target":"customers"}

Execution text: XXXXXX

[0079] Specifically, the recognition prompt text is used to clarify the object information that needs to be recognized in the environmental image, that is, the recognition prompt text is used to prompt the recognition model to focus on the content in the environmental image. For example, the recognition prompt text can clarify the category, color, shape and other information of the object that needs to be recognized in the environmental image.

[0080] The following is an example of the recognition prompt text: "Please describe what objects are in the environment image, focusing on whether the objects mentioned in the execution instructions exist in the environment image. You are required to tell me every object that appears in the environment image, and the description of the objects should be as rich as possible. The reply format is {**Hint**: The objects on the image include: object 1, object 2, object 3}, example: {**Hint**: The objects on the image include: yellow bananas, laptops, cola (drinks), red apples (food)} Example: {**Hint**: The objects on the image include: pencils (wooden), lemons (very sour fruit), hamburgers, telescopes (black)}".

[0081] After the recognition prompt text is constructed, the environment image and the recognition prompt text are input into the recognition model, and the recognition model accurately recognizes the corresponding object information from the environment image based on the prompt content in the recognition prompt text. Among them, the recognition model can be a large language model (LLM), which refers to a natural language processing (NLP) model with a large parameter scale. The number of model parameters and / or the complexity of the model structure exceeds a preset threshold. The model processes large-scale text data during training and has the ability to understand and generate natural language. For example, a large language model can include the Spark large model, etc.

[0082] In addition, before applying the recognition hint text based on the recognition model to obtain the object information from the environment image, the size of the environment image is adjusted to a fixed size (384, 384) and then encoded into a 64-bit UTF-8 format, as shown below: Among them, adjusting the environment image size to a fixed (384, 384) ensures that all input images have the same resolution, which facilitates subsequent processing and helps reduce the processing complexity and time consumption caused by image size differences. In addition, encoding image data in 64-bit UTF-8 format ensures data consistency across different platforms and devices, avoiding errors or garbled characters caused by character encoding differences.

[0083] Based on any of the above embodiments, identifying the position and posture of each execution object includes: Identify and obtain the detection frame of each execution object from the environment image; Based on the detection box of each execution object, the position and pose of each execution object are determined.

[0084] For example, the environment image and the terms of each execution object (such as the terms corresponding to the name) can be input into the target detection model, and the target detection model identifies the execution object in the environment image that matches the terms of each execution object, and outputs the detection box of the execution object. The target detection model can be a Grounding Dino model.

[0085] Next, the environment image and the coordinates of the corner points of each detection frame can be input into the image segmentation model, which segments the area corresponding to the execution object in each detection frame and generates a corresponding mask. The image segmentation model can be an EfficientVIT-SAM model.

[0086] Finally, the mask and the environment image are input into the pose estimation model, which determines the pose of each execution object, such as the 6D pose, so that action planning can be performed according to the pose and position of each execution object, and the corresponding subtasks can be completed in sequence. The pose estimation model can be a Foundation Pose model.

[0087] Based on any of the above embodiments, each execution action exists in a preset action library.

[0088] Specifically, the preset action library is a data set that stores a variety of pre-defined and programmed action instructions or behavior patterns. These action instructions or behavior patterns are usually formulated based on the design purpose, functional requirements and application scenarios of the robot.

[0089] By presetting the execution actions in the action library, the robot can directly call the actions in the preset action library when performing tasks without having to create each action from scratch, thereby shortening the task execution time and improving overall efficiency. Especially in tasks that require repeated execution of certain actions, the use of a preset action library can significantly reduce repetitive work.

[0090] Based on any of the above embodiments, Figure 2 is a flowchart of another task execution method provided by the present invention, such as Figure 2 As shown, the method includes: Construct recognition prompt text for prompting item information recognition requirements, and obtain the environment image corresponding to the task to be performed. Input the recognition prompt text and the environment image into the recognition model, and the recognition model uses the recognition prompt text to recognize the item information in the environment from the environment image.

[0091] At the same time, the user's execution instructions are received, and based on the speech recognition model, speech recognition is performed on the execution instructions to obtain corresponding execution texts.

[0092] Construct a task decomposition prompt text for prompting task decomposition requirements, input the task decomposition prompt text, item information, and execution text into the task decomposition model, and the task decomposition model decomposes the task to be executed into multiple subtasks, each of which includes an execution object and an execution action corresponding to the execution object. In addition, the task decomposition model can also simultaneously obtain the user's historical records of previous task decomposition, so as to adjust and optimize the task decomposition algorithm based on the corresponding records of successful decomposition and failed decomposition in the historical records, and finally obtain the optimal task decomposition solution.

[0093] The environment image and the entries of each execution object (such as the entries corresponding to the name) are input into the target detection model, and the target detection model identifies the execution object in the environment image that matches the entry of each execution object and outputs the detection box of the execution object.

[0094] The environment image and the coordinates of the corner points of each detection frame are input into the image segmentation model, which segments the area corresponding to the execution object in each detection frame and generates a corresponding mask.

[0095] Finally, the mask and the environment image are input into the posture estimation model, which determines the posture of each execution object, such as the 6D posture, so that action planning can be performed according to the posture and position of each execution object, and the corresponding subtasks can be completed in sequence.

[0096] In addition, the above task execution method is tested by constructing a test instruction set, which includes: action instructions {"please squat down", "please walk to the side of the table and then step on the spot"...}, action + grasping instructions {"dance for me and then give me the apple", "walk to the side of the table and help me get a bottle of Coke"...}, explicit grasping instructions {"help me put the hamburger on the plate", "help me put the flowers in the bottle"...}, fuzzy grasping instructions {"help me clean up the desktop", "the largest object on the desktop"...}. 15 instructions of each type are set according to the desktop scene, and a total of 60 instructions are used for testing. Set the recognition model parameters: "temperature" = 0.3, "max_tokens" = 150; set the task decomposition model parameters: "temperature" = 0.1, "max_tokens" = 200. Input the desktop image information and instruction content to the above model, and output the task decomposition results in JSON format. Table 1 is the test result list.

[0097] Table 1

[0098] The task execution device provided by the present invention is described below. The task execution device described below and the task execution method described above can be referenced to each other.

[0099] Based on any of the above embodiments, Figure 3 is a schematic diagram of the structure of the task execution device provided by the present invention, such as Figure 3 As shown, the device comprises: The recognition unit 310 is used to recognize object information in the environment based on the environment image corresponding to the task to be performed; A decomposition unit 320, configured to decompose the task to be executed into a plurality of subtasks based on the execution instruction and the item information used to describe the task to be executed, each subtask including an execution object and an execution action of the execution object; The execution unit 330 is used to identify the position and posture of each execution object, and execute the corresponding subtask based on the position and posture of each execution object and the execution action.

[0100] Based on any of the above embodiments, based on the execution instructions and item information used to describe the task to be performed, the task to be performed is decomposed into multiple subtasks, including: Convert execution instructions into execution text; Based on the item information, extracting target text associated with the item information from the execution text; Based on the target text, the task to be performed is decomposed into multiple subtasks.

[0101] Based on any of the above embodiments, based on the item information, extracting the target text associated with the item information from the execution text includes: Perform entity recognition on the execution text to obtain at least one execution entity; Based on the item information, determining a target entity matching the object in the environment from at least one execution entity; Extracting action entities associated with target entities from execution text; Construct the target text based on the action entity and the execution entity.

[0102] Based on any of the above embodiments, a target text associated with the item information is extracted from the execution text, and based on the target text, the task to be executed is decomposed into multiple subtasks, including: Constructing a task decomposition prompt text for prompting task decomposition requirements; Based on the task decomposition model, the task decomposition prompt text is applied to extract the target text associated with the item information from the execution text. Based on the target text, the task to be executed is decomposed into multiple subtasks.

[0103] Based on any of the above embodiments, identifying information of items in the environment based on the environment image corresponding to the task to be performed includes: Constructing identification prompt text for prompting item information identification requirements; Based on the recognition model, the recognition prompt text is applied to identify the object information from the environmental image.

[0104] Based on any of the above embodiments, identifying the position and posture of each execution object includes: Identify and obtain the detection frame of each execution object from the environment image; Based on the detection box of each execution object, the position and pose of each execution object are determined.

[0105] Based on any of the above embodiments, each execution action exists in a preset action library.

[0106] Based on any of the above embodiments, the present invention further provides a robot, comprising: A camera arranged on the robot body (such as the robot head) and a task execution device as described in any of the above embodiments, the camera is used to collect environmental images.

[0107] The task execution device described in any of the above embodiments can decompose the task to be executed into multiple simple subtasks through object information and execution instructions, and dynamically decompose the task to be executed according to specific environmental information and execution instructions without relying on a predefined instruction set, so that the robot can handle more complex and flexible execution instructions and adapt to complex and changing application scenarios. In addition, when the robot faces a long text execution instruction containing multiple execution objects or complex action descriptions, it can also accurately identify multiple execution objects or complex actions through object information and execution instructions, and decompose the task to be executed into multiple subtasks containing a single execution object or simple action.

[0108] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the task execution method, which includes: based on the environment image corresponding to the task to be executed, identifying the object information in the environment; based on the execution instruction used to describe the task to be executed and the object information, decomposing the task to be executed into multiple subtasks, each subtask including an execution object and an execution action of the execution object; identifying the posture of each execution object, and executing the corresponding subtask based on the posture and execution action of each execution object.

[0109] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0110] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the task execution method provided by the above methods, which includes: based on the environment image corresponding to the task to be executed, identifying the object information in the environment; based on the execution instructions used to describe the task to be executed and the object information, decomposing the task to be executed into multiple subtasks, each subtask including an execution object and an execution action of the execution object; identifying the posture of each execution object, and executing the corresponding subtask based on the posture and execution action of each execution object.

[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the task execution method provided by the above-mentioned methods, the method comprising: identifying object information in the environment based on an environment image corresponding to the task to be executed; decomposing the task to be executed into multiple subtasks based on an execution instruction for describing the task to be executed and the object information, each subtask comprising an execution object and an execution action of the execution object; identifying the posture of each execution object, and executing the corresponding subtask based on the posture and execution action of each execution object.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A task execution method, characterized in that: include: Based on the environment image corresponding to the task to be performed, identify the object information in the environment; Based on the execution instruction for describing the task to be executed and the item information, the task to be executed is decomposed into a plurality of subtasks, each subtask including an execution object and an execution action of the execution object; Identify the position and posture of each execution object, and execute the corresponding subtask based on the position and posture of each execution object and the execution action.

2. The task execution method according to claim 1, characterized in that: The step of decomposing the task to be performed into a plurality of subtasks based on the execution instruction for describing the task to be performed and the item information includes: Converting the execution instruction into an execution text; Based on the item information, extracting a target text associated with the item information from the execution text; Based on the target text, the task to be performed is decomposed into a plurality of subtasks.

3. The task execution method according to claim 2, characterized in that: The step of extracting a target text associated with the item information from the execution text based on the item information includes: Performing entity recognition on the execution text to obtain at least one execution entity; Based on the item information, determining a target entity matching the object in the environment from the at least one execution entity; Extracting an action entity associated with the target entity from the execution text; The target text is constructed based on the action entity and the execution entity.

4. The task execution method according to claim 2, characterized in that: The step of extracting a target text associated with the item information from the execution text and decomposing the task to be executed into a plurality of subtasks based on the target text includes: Constructing a task decomposition prompt text for prompting task decomposition requirements; Based on the task decomposition model, the task decomposition prompt text is applied to extract the target text associated with the item information from the execution text, and based on the target text, the task to be executed is decomposed into multiple subtasks.

5. The task execution method according to any one of claims 1 to 4, characterized in that: The identifying of object information in the environment based on the environment image corresponding to the task to be performed includes: Constructing identification prompt text for prompting item information identification requirements; Based on the recognition model, the recognition prompt text is applied to recognize and obtain the object information from the environment image.

6. The task execution method according to any one of claims 1 to 4, characterized in that: The identifying the position and posture of each execution object includes: Recognize and obtain detection frames of various execution objects from the environment image; Based on the detection box of each execution object, the position and pose of each execution object are determined.

7. The task execution method according to any one of claims 1 to 4, characterized in that: Each execution action exists in the preset action library.

8. A task execution device, characterized in that: include: A recognition unit, used to recognize information of objects in the environment based on an environment image corresponding to the task to be performed; a decomposition unit, configured to decompose the task to be executed into a plurality of subtasks based on the execution instruction used to describe the task to be executed and the item information, each subtask including an execution object and an execution action of the execution object; The execution unit is used to identify the position and posture of each execution object, and execute the corresponding subtask based on the position and posture of each execution object and the execution action.

9. A robot, characterized in that: include: A camera arranged on the robot body and the task execution device as claimed in claim 8, wherein the camera is used to collect the environmental image.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the task execution method according to any one of claims 1 to 7 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task execution method according to any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the task execution method according to any one of claims 1 to 7 is implemented.

Images