Task planning method and system for robot with body, controller and robot with body

Abstract

The invention relates to the technical field of robot intelligence, and discloses a robot task planning method and system, a controller and a robot, and the method comprises the steps: carrying out the task decoupling and planning of a task instruction of the robot based on a hierarchical scene graph and a task knowledge base, and obtaining a preliminary task plan; according to the preliminary task plan, the robot with the body is controlled to execute subtasks, and a preliminary execution result is fed back; under the condition that the preliminary execution result shows that the robot with the body completes the preliminary task planning but does not complete the task instruction, whether a candidate execution object exists in the hierarchical scene graph or not is judged; and under the condition that the candidate execution object exists in the hierarchical scene graph, performing dynamic planning adjustment according to the candidate execution object to obtain an adjustment task plan. The method has the beneficial effects that the perception capability of the robot to the actual environment is improved, so that the task planning can be adjusted according to the actual environment, and the task execution effect of the robot is improved.

Description

Technical Field

[0001] The present application relates to the field of embodied intelligent technology, and in particular to an embodied robot task planning method, system, controller and embodied robot. Background Art

[0002] With the development of science and technology, embodied robots appear in different scenes of people's lives. The embodied robots in different scenes perform corresponding tasks based on the task instructions issued by the user, thereby serving the user. After the user issues the task instruction, the controller of the embodied robot receives the task instruction and uses the big model to reason and analyze the task instruction, thereby formulating the corresponding task plan for the embodied robot. In the related technology, the flexibility of controlling the embodied robot to perform tasks according to the task plan still needs to be improved. Summary of the invention

[0003] The present application provides an embodied robot task planning method, system, controller and embodied robot, which plan the task instructions of the embodied robot according to a hierarchical scene graph, solving the technical problem of the lack of real-time perception of the actual environment when performing task planning in related technologies, thereby being able to adjust the task planning according to the actual environment and improving the effect of the embodied robot in completing the task instructions.

[0004] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0005] In a first aspect, an embodiment of the present application provides a method for embodied robot task planning, the method comprising:

[0006] Based on the pre-constructed hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are task-decoupled and planned to obtain a preliminary task plan including a plurality of subtasks; wherein the subtasks correspond to preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are objects or positions that the embodied robot needs to reach during the execution of the task instructions;

[0007] Controlling the embodied robot to execute the multiple subtasks according to the preliminary task plan, and feeding back preliminary execution results of the multiple subtasks;

[0008] In the case where the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction, determining whether there is a candidate execution object in the hierarchical scene graph; wherein the candidate execution object is an object or position that can replace the preliminary execution object;

[0009] In the case where the candidate execution object exists in the hierarchical scene graph, dynamic planning adjustment is performed according to the candidate execution object to obtain an adjusted task plan to control the embodied robot to complete the task instruction.

[0010] The embodied robot task planning method proposed in the embodiment of the present application utilizes a pre-constructed hierarchical scene graph and a task knowledge base to perform task planning on the embodied robot, obtains a preliminary task plan, and performs reasoning based on the execution results of the subtasks in the preliminary task plan to determine whether the preliminary task plan can be adjusted. This method improves the embodied robot's ability to grasp the actual environment by constructing a hierarchical scene graph for the scene in which the embodied robot is located, thereby improving the embodied robot's execution performance in a complex environment. In addition, compared with the technical solutions that rely on preliminary task planning in the related art, this method also performs reasoning based on the execution results of the subtasks in the preliminary task plan and the hierarchical scene graph, so that the preliminary task plan can be adjusted according to the environmental changes in the actual scene, thereby improving the embodied robot's ability to respond to changes in the actual environment and improving the flexibility of the embodied robot when performing tasks.

[0011] Optionally, based on the pre-built hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are task decoupled and planned to obtain a preliminary task plan containing multiple subtasks, including:

[0012] Using the large model to perform task decoupling and object recognition on the task instruction, to obtain the target object and execution range in the task instruction; wherein the execution range is the spatial range in which the embodied robot executes the task instruction;

[0013] According to the target object, the execution scope and the task knowledge base, object screening is performed in the hierarchical scene graph to obtain the preliminary execution object;

[0014] Task planning is performed according to the target object and the preliminary execution object to obtain the multiple subtasks and formulate the preliminary task plan.

[0015] Optionally, the performing object screening in the hierarchical scene graph according to the target object, the execution scope and the task knowledge base to obtain the preliminary execution object includes:

[0016] Determining a target level at which the target object is located in the hierarchical scene graph;

[0017] Determining, at a level above the target level in the hierarchical scene graph, a plurality of candidate execution objects existing within the execution scope;

[0018] According to the task knowledge base, the multiple candidate execution objects and the target object are analyzed for association one by one, and the preliminary execution object that meets the association condition is determined from the multiple candidate execution objects.

[0019] Optionally, the hierarchical scene graph is obtained according to the following method:

[0020] Capturing images of the scene in which the embodied robot is located to obtain a plurality of scene images;

[0021] Using the large model to perform image description on the multiple scene images to obtain description results;

[0022] Based on the task knowledge base, hierarchically classify the objects or positions in the scene where the embodied robot is located according to the description results, and determine the scene level corresponding to each object or position;

[0023] A scene graph structure is constructed according to the correlation between the objects or positions and the corresponding scene hierarchies to obtain the hierarchical scene graph.

[0024] Optionally, the embodied robot has a plurality of preset task tools; performing task planning according to the target object and the preliminary execution object, obtaining the plurality of subtasks, and formulating the preliminary task plan, comprises:

[0025] According to the target object, the preliminary execution object and the task instruction, a corresponding first task tool is determined from the preset task tools to generate a first subtask;

[0026] Performing task analysis on the first subtask using the large model and the task knowledge base, and determining a second task tool from the preset task tools to generate a second subtask;

[0027] The second subtask is associated with the first subtask in an execution order to obtain the preliminary task plan.

[0028] Optionally, the determining whether there is a candidate execution object in the hierarchical scene graph includes:

[0029] Determining the to-be-determined execution object from the hierarchical scene graph according to the level at which the preliminary execution object is located in the hierarchical scene graph and the execution scope of the task instruction;

[0030] Using the large model, an association analysis is performed on the to-be-determined execution object according to the task knowledge base to determine whether the candidate execution object exists in the hierarchical scene graph; wherein the association between the candidate execution object and the target object in the task instruction is weaker than the association between the preliminary execution object and the target object.

[0031] Optionally, the method further comprises:

[0032] According to the adjustment task plan, controlling the embodied robot to execute the subtasks in the adjustment task plan, and feeding back the adjustment execution result;

[0033] When the adjustment execution result indicates that the embodied robot has completed the adjustment task plan but has not yet completed the task instruction, the above-mentioned dynamic planning adjustment process is executed again to obtain a new adjustment task plan as the adjustment task plan, so as to repeatedly control the embodied robot to execute the corresponding subtasks until the task instruction end condition is met.

[0034] In a second aspect, an embodiment of the present application provides an embodied robot task planning system, characterized in that the system comprises:

[0035] A preliminary task planning module, for performing task decoupling and planning on the task instructions of the embodied robot based on a pre-built hierarchical scene graph and a task knowledge base, and obtaining a preliminary task plan including a plurality of subtasks; wherein the subtasks correspond to preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are objects or positions that the embodied robot needs to reach during the execution of the task instructions;

[0036] A preliminary task execution module, used to control the embodied robot to execute the multiple subtasks according to the preliminary task plan, and to feed back preliminary execution results of the multiple subtasks;

[0037] A dynamic planning adjustment module, configured to determine whether there is a candidate execution object in the hierarchical scene graph when the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction; wherein the candidate execution object is an object or position that can replace the preliminary execution object;

[0038] The task planning adjustment module is used to perform dynamic planning adjustment according to the candidate execution object when the candidate execution object exists in the hierarchical scene graph, so as to obtain an adjusted task plan to control the embodied robot to complete the task instruction.

[0039] In a third aspect, an embodiment of the present application provides a controller, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the method described in any one of the above embodiments by executing the computer instructions.

[0040] In a fourth aspect, an embodiment of the present application provides an embodied robot, comprising a controller according to any one of the above embodiments.

[0041] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute any one of the methods in the above embodiments.

[0042] In a sixth aspect, an embodiment of the present application provides a computer program product, including computer instructions, wherein the computer instructions are used to enable a computer to execute any one of the methods described in the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A step diagram of the embodied robot task planning method provided in an embodiment of the present application;

[0045] Figure 2 A step diagram for formulating a preliminary task plan in an embodiment of the present application is provided;

[0046] Figure 3 A flowchart of initial task planning is provided for an embodiment of the present application;

[0047] Figure 4 A step diagram for determining a preliminary execution object in an embodiment of the present application is provided;

[0048] Figure 5 A diagram of the steps for obtaining a preliminary task plan in an embodiment of the present application is provided;

[0049] Figure 6 A step diagram for determining whether there is a candidate execution object in a hierarchical scene graph in an embodiment of the present application;

[0050] Figure 7 A step diagram for repeatedly performing dynamic programming adjustment in an embodiment of the present application is provided;

[0051] Figure 8 A module diagram of an embodied robot task planning system provided in an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of the structure of a controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0054] With the development of science and technology, embodied robots appear in different scenarios of people's lives, such as smart homes, logistics and transportation, and service fields. In different scenarios, embodied robots respond to task instructions issued by users and perform corresponding tasks according to the task instructions, thereby serving users and achieving the purpose that users need embodied robots to achieve. In related technologies, after receiving the task instructions issued by the user, the controller for controlling the embodied robot usually uses a large model to perform reasoning analysis on the task instructions, so as to decouple complex task instructions into multiple subtasks through the knowledge base and reasoning logic built into the large model, thereby formulating corresponding task plans for the embodied robot.

[0055] In the above-mentioned related technologies, offline pre-training data is usually used to train the large model, but the offline pre-training data cannot reflect the changes in the actual environment of the embodied robot, resulting in the embodied robot being unable to perceive the changes in the actual environment in time when the actual environment changes, resulting in deviations in the task execution results of the embodied robot.

[0056] Secondly, in the process of executing tasks, the embodied robot is highly dependent on task planning, which leads to the lack of necessary autonomous adaptability of the embodied robot and the inability to adjust the task planning according to the task execution results or actual environmental changes, thus limiting the performance of the embodied robot in complex environments.

[0057] Based on the above problems, the present application provides an embodied robot task planning method, system, controller and embodied robot, the method comprising: based on a hierarchical scene graph and a task knowledge base, task decoupling and planning the task instructions of the embodied robot to obtain a preliminary task plan including multiple subtasks; controlling the embodied robot to execute subtasks according to the preliminary task plan, and feeding back the preliminary execution results of the subtasks; when the preliminary execution results indicate that the embodied robot has completed the preliminary task planning but has not completed the task instructions, determining whether there are candidate execution objects in the hierarchical scene graph; when there are candidate execution objects in the hierarchical scene graph, dynamically planning and adjusting according to the candidate execution objects to obtain an adjusted task plan to control the embodied robot to complete the task instructions.

[0058] The embodied robot task planning method proposed in the embodiment of the present application uses a pre-constructed hierarchical scene graph and a task knowledge base to plan tasks for the embodied robot, obtains a preliminary task plan, and performs reasoning based on the execution results of the subtasks in the preliminary task plan to determine whether the preliminary task plan can be adjusted. This method improves the embodied robot's ability to grasp the actual environment by constructing a hierarchical scene graph for the scene in which the embodied robot is located, thereby improving the embodied robot's execution performance in complex environments.

[0059] In addition, compared with the technical solutions in related technologies that rely on preliminary task planning, this method also performs reasoning through the execution results of subtasks in the preliminary task planning and the hierarchical scene graph, so that the preliminary task planning can be adjusted according to environmental changes in the actual scene, thereby improving the responsiveness of the embodied robot to actual environmental changes and improving the flexibility of the embodied robot when performing tasks.

[0060] The embodied robot task planning method provided in this specification can be applied to embodied robots that can generate task plans according to task instructions and execute corresponding subtasks, including embodied robots and smart home devices. It is understandable that this method can also be used to control intelligent devices other than the above-mentioned embodied robots, such as virtual intelligent bodies, industrial embodied robots, and medical embodied robots, after adaptive modification.

[0061] According to an embodiment of the present application, an embodiment of an embodied robot task planning method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0062] In this embodiment, a method for embodied robot task planning is provided, which can be used for the above-mentioned embodied robot capable of generating task planning according to task instructions and executing corresponding subtasks. Figure 1 As shown, the method includes:

[0063] S100. Based on the pre-constructed hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are decoupled and planned to obtain a preliminary task plan containing multiple subtasks; wherein the subtasks correspond to preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are objects or positions that the embodied robot needs to reach during the execution of the task instructions.

[0064] S200. Control the embodied robot to execute multiple subtasks according to the preliminary task plan, and feed back preliminary execution results of the multiple subtasks.

[0065] S300. When the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction, determine whether there is a candidate execution object in the hierarchical scene graph; wherein the candidate execution object is an object or position that can replace the preliminary execution object.

[0066] S400. When there are candidate execution objects in the hierarchical scene graph, dynamic planning adjustment is performed according to the candidate execution objects to obtain an adjusted task plan to control the embodied robot to complete the task instructions.

[0067] Among them, the hierarchical scene graph can be a hierarchical graph structure about the actual environment in which the embodied robot is located, and the hierarchical scene graph contains multiple scene nodes corresponding to objects or positions in the actual environment, each scene node has a corresponding level, and is connected to other scene nodes through edges. Each level in the hierarchical scene graph can represent the type of each scene node, and the edges between each scene node in the hierarchical scene graph represent the spatial position relationship and the subordinate relationship between the corresponding objects or positions. In some embodiments, according to the execution results of the corresponding subtasks in the preliminary task planning and the adjustment task planning, the information of each scene node in the hierarchical scene graph can be updated so that the hierarchical scene graph can correctly feedback the actual environment.

[0068] To illustrate the composition of the hierarchical scene graph by way of example, when the actual environment of the embodied robot is a house, the hierarchical scene graph may include four levels, wherein the first level may be a floor; the second level may be a room, such as a bedroom, a living room, and a kitchen; the third level may be a large object that is not often moved in actual situations, such as a bed, a sofa, and a bookcase; the fourth level may be a small object that can be moved at any time, such as a book and a water cup. Among them, the small object as a scene node on the fourth level is defined as a transient object, and its position information and state information will be temporarily stored in the hierarchical scene graph. When the feedback execution result indicates that the position information or state information of the small object has changed compared to the hierarchical scene graph, the scene node on the fourth level of the hierarchical scene graph is updated to improve the degree of conformity between the hierarchical scene graph and the actual environment.

[0069] The task knowledge base can be a knowledge base containing knowledge related to the task instructions, and the related knowledge can be common sense knowledge and professional knowledge related to the task instructions. When performing task planning, the large model can identify the task instructions, determine the target object in the task instructions, and then decouple and plan the tasks according to the task instructions and the target object based on the related knowledge, thereby obtaining a preliminary task plan containing multiple subtasks. It should be noted that the subtasks correspond to preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are connected to the target objects in the hierarchical scene graph through edges, indicating that there is a corresponding relationship between the preliminary execution objects and the target objects, such as a subordinate relationship and a relative position relationship.

[0070] To illustrate the role of the task knowledge base, if the task instruction is "find a water cup in the bedroom", the large model is used to identify the task instruction, and it can be determined that the target object is a water cup, and the purpose of the task instruction is to find the location of the target object. In the case where the water cup is not in the hierarchical scene graph, according to the common sense knowledge contained in the task knowledge base, it can be determined that the water cup will be placed on a flat and hard surface, and the height of the water cup can be determined according to the human body's habits. Then, according to the hierarchical scene graph, it can be determined that the objects in the bedroom that meet the above requirements are the coffee table and the bedside table. At this time, the coffee table and the bedside table are used as the preliminary execution objects, and task planning is performed based on the target object and the preliminary execution object to obtain a preliminary task plan.

[0071] Specifically, the embodied robot receives the task instructions issued by the user. The task instructions can be in text form or voice form. For the task instructions in voice form, the task instructions are converted into text form through voice recognition. In response to the task instructions in text form, based on the pre-built hierarchical scene graph and task knowledge base, the task instructions are decoupled and identified using a large model, so as to determine the target object in the task instruction, that is, the object with which the embodied robot mainly interacts. According to the target scene and the task instruction, the preliminary execution object corresponding to the target object is determined from the hierarchical scene graph using the task knowledge base, and task planning is performed based on the target object and the preliminary execution object to obtain a preliminary task plan.

[0072] Furthermore, the preliminary task planning includes multiple subtasks, and the embodied robot is controlled to execute the subtasks in sequence based on the preliminary task planning, and the preliminary execution results of the subtasks are fed back in real time. If the preliminary execution result of any subtask indicates that the embodied robot has completed the task instruction, the execution of the subtask is stopped, and the embodied robot is controlled to feed back the execution result of the task instruction to the user. If the preliminary execution results of all subtasks indicate that the embodied robot has not completed the task instruction, that is, the embodied robot has completed all preliminary task planning but has not completed the task instruction, then it is necessary to reason and judge whether to dynamically adjust the preliminary task planning according to the hierarchical scene graph.

[0073] Furthermore, the condition for dynamically adjusting the preliminary task plan is that there are candidate execution objects in the hierarchical scene graph. The candidate execution objects are objects or positions that can replace the preliminary execution objects. It can be understood that, similar to the preliminary execution objects, the candidate execution objects are connected to the target objects through edges, indicating that there is a corresponding relationship between the candidate execution objects and the target objects, such as a subordinate relationship and a relative position relationship. In the case that there are candidate execution objects in the hierarchical scene graph, multiple subtasks are generated according to the candidate execution objects and the target objects for dynamically adjusting the preliminary task plan, thereby obtaining an adjusted task plan to control the embodied robot to complete the task instructions.

[0074] Compared with the related art, the present application first constructs a hierarchical scene graph for the actual environment in which the embodied robot is located, and improves the embodied robot's ability to grasp the complex actual environment through the hierarchical scene graph, thereby improving the accuracy of the resulting task planning and the degree of conformity with the actual environment. Secondly, the hierarchical scene graph in the present application can be updated according to the execution results fed back after the embodied robot executes the subtask, so that the hierarchical scene graph can correctly feedback the actual environment and improve the embodied robot's perception of changes in the actual environment. Furthermore, the present application also makes inferences and judgments on whether the preliminary task planning can be adjusted based on the hierarchical scene graph and the execution results of the subtask, so that the embodied robot can respond in a timely manner when the actual environment changes, thereby improving the embodied robot's dynamic response ability to changes in the actual environment.

[0075] The embodied robot task planning method provided in this embodiment uses a pre-constructed hierarchical scene graph and a task knowledge base to plan tasks for the embodied robot, obtains a preliminary task plan, and performs reasoning based on the execution results of the subtasks in the preliminary task plan to determine whether the preliminary task plan can be adjusted. This method improves the embodied robot's ability to grasp the actual environment by constructing a hierarchical scene graph for the scene in which the embodied robot is located, thereby improving the embodied robot's execution performance in complex environments.

[0076] In addition, compared with the technical solutions in related technologies that rely on preliminary task planning, this method also performs reasoning through the execution results of subtasks in the preliminary task planning and the hierarchical scene graph, so that the preliminary task planning can be adjusted according to environmental changes in the actual scene, thereby improving the responsiveness of the embodied robot to actual environmental changes and improving the flexibility of the embodied robot when performing tasks.

[0077] Reference Figure 2 As shown, as an embodiment of the present application, based on the pre-built hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are task decoupled and planned to obtain a preliminary task plan containing multiple subtasks, including:

[0078] S110. Use the large model to perform task decoupling and object recognition on the task instructions to obtain the target object and execution scope in the task instructions; wherein the execution scope is the spatial range in which the embodied robot executes the task instructions.

[0079] S120. According to the target object, execution scope and task knowledge base, objects are screened in the hierarchical scene graph to obtain preliminary execution objects.

[0080] S130. Perform task planning according to the target object and the preliminary execution object, obtain multiple subtasks, and formulate a preliminary task plan.

[0081] Specifically, using the big model to perform task decoupling and object recognition on task instructions includes: using the big model to decouple and recognize the task instructions in text form, and extracting the corresponding target object and execution scope from them. It can be understood that the spatial range corresponding to the execution scope has a corresponding scene node in the hierarchical scene graph, and there is a subordinate relationship or a relative spatial relationship between the target object and the execution scope.

[0082] Furthermore, the corresponding scene nodes are determined in the hierarchical scene graph according to the obtained target object and execution scope, and multiple scene nodes that may serve as preliminary execution objects are determined in the hierarchical scene graph according to the above scene nodes. The task knowledge base is used to perform object screening on multiple scene nodes that may serve as preliminary execution objects to obtain preliminary execution objects. After obtaining the target object and the preliminary execution object, tasks are combined according to the target object and the preliminary execution object to obtain multiple subtasks, so that preliminary task planning can be formulated according to the above multiple subtasks. Among them, any subtask corresponds to a target object and any preliminary execution object.

[0083] For example, refer to Figure 3 As shown, in this embodiment, the task instruction is "find a water cup in the bedroom". Through the large model, it can be determined that the target object in the task instruction is a water cup and the execution scope is the bedroom. At this time, according to the target object, execution scope and task knowledge base, object screening is performed in the hierarchical scene graph, and it can be determined that the preliminary execution objects include bedside tables and coffee tables. According to the target object and the preliminary execution object, two subtasks can be obtained, namely "find a water cup at the bedside table" and "find a water cup at the coffee table". According to the above subtasks, a preliminary task plan is obtained, and according to the preliminary task plan, the embodied robot is controlled to perform the subtasks.

[0084] Reference Figure 4 As shown, as an embodiment of the present application, according to the target object, the execution scope and the task knowledge base, objects are screened in the hierarchical scene graph to obtain preliminary execution objects, including:

[0085] S122. Determine the target level of the target object in the hierarchical scene graph.

[0086] S124. Determine, at an upper level of the target level in the hierarchical scene graph, a plurality of candidate execution objects existing within the execution scope.

[0087] S126. Perform association analysis on multiple candidate execution objects and target objects one by one according to the task knowledge base, and determine a preliminary execution object that meets the association condition from the multiple candidate execution objects.

[0088] Specifically, there is a subordinate relationship or spatial position relationship between different levels in the hierarchical scene graph. The scene nodes on the next level may be subordinate to the scene nodes on the previous level, or located within the scene nodes on the previous level in the actual environment. After determining the target object, the previous level of the target level is the level where the preliminary execution object may be located. At this time, multiple candidate execution objects are determined on the previous level according to the execution scope. It is understandable that there may be no correlation between the candidate execution objects and the target object.

[0089] Furthermore, the large model is used to perform association analysis on multiple candidate execution objects and target objects one by one based on common sense knowledge or professional knowledge in the task knowledge base. If the association between any candidate execution object and the target object exceeds a threshold, the candidate execution object meets the association condition and is used as a preliminary execution object. In some embodiments, the method of association analysis can be to analyze the weights of the edges in the hierarchical scene graph, or to query the task knowledge base based on the preliminary execution object and the target object to obtain the association between the preliminary execution object and the target object in the hierarchical scene graph.

[0090] As an embodiment of the present application, a hierarchical scene graph is obtained according to the following method:

[0091] S102. Capture images of the scene where the embodied robot is located to obtain multiple scene images.

[0092] S104. Use the large model to perform image description on multiple scene images to obtain description results.

[0093] S106. Based on the task knowledge base, hierarchically classify the objects or positions in the scene where the embodied robot is located according to the description results, and determine the scene level corresponding to each object or position.

[0094] S108. Construct a scene graph structure according to the correlation between the objects or positions and the corresponding scene hierarchies to obtain a hierarchical scene graph.

[0095] Specifically, the actual environment where the embodied robot is located is imaged from multiple angles to obtain multiple visual images of the actual environment as scene images. Corresponding prompt statements are designed and input into the large model to guide the large model to describe the actual environment according to the scene images to obtain description results. It can be understood that the description results may include object types, object attributes, and relative positions between objects in the actual scene.

[0096] Furthermore, the common sense knowledge or professional knowledge in the task knowledge base is used to hierarchically classify the objects or positions in the actual scene according to the description results to determine the scene level corresponding to each object or position. The hierarchical classification step may include: classifying the objects or positions in the actual scene according to the description results, and sorting the different types of objects or positions according to the classification results to determine the scene level corresponding to each object or position.

[0097] Reference Figure 5 As shown, as an embodiment of the present application, the embodied robot has a plurality of preset task tools; performs task planning according to the target object and the preliminary execution object, obtains a plurality of subtasks, and formulates a preliminary task plan, including:

[0098] S132. According to the target object, the preliminary execution object and the task instruction, determine the corresponding first task tool from the preset task tools to generate a first subtask.

[0099] S134. Perform task analysis on the first subtask using the large model and the task knowledge base, and determine the second task tool from the preset task tools to generate the second subtask.

[0100] S136. Associate the second subtask with the first subtask in the execution order to obtain a preliminary task plan.

[0101] Among them, the preset task tool can be a function possessed by the embodied robot, which is implemented by the corresponding tool installed in the embodied robot. Exemplarily, the preset task tools may include moving (Move), finding or observing (Look), and feedback of task instruction results (Say). The call and execution of each preset task tool are through the API interface, and are selected based on the actions in the task instruction. It should be noted that the list of preset task tools is open, and users can add new preset task tools to the above list according to the needs of the actual scenario, thereby expanding the range of tasks that the embodied robot can perform.

[0102] Specifically, a corresponding first task tool is determined from preset task tools according to the action in the task instruction, and a first subtask is generated based on the target object, the preliminary execution object and the first task tool.

[0103] Furthermore, the different initial execution objects in different first subtasks result in different execution positions of the first subtasks, which makes it impossible for the first subtasks to be executed continuously. Therefore, the second subtask is required as a connecting subtask between different first subtasks. The first subtask and the second subtask constitute a preliminary task plan, so that the embodied robot can be controlled to execute tasks continuously. There can be multiple second subtasks. According to the execution order of the first subtasks, the multiple second subtasks are respectively associated with the corresponding first subtasks, so as to obtain a preliminary task plan.

[0104] To illustrate the generation method of the second subtask, if the initial execution objects of the first subtask include the bedside table and the coffee table, and in the actual scene the bedside table and the coffee table are in different positions, then the first subtask is analyzed using the large model and the task knowledge base, and it can be determined that the second task tool is Move. At this time, the second subtask used to connect the first subtask is "Move from the bedside table to the coffee table".

[0105] Reference Figure 6 As shown, as an embodiment of the present application, determining whether there is a candidate execution object in the hierarchical scene graph includes:

[0106] S310. Determine the to-be-determined execution object from the hierarchical scene graph according to the level of the preliminary execution object in the hierarchical scene graph and the execution scope of the task instruction.

[0107] S320. Using the large model, perform association analysis on the to-be-determined execution object according to the task knowledge base to determine whether there is a candidate execution object in the hierarchical scene graph; wherein the association between the candidate execution object and the target object in the task instruction is weaker than the association between the preliminary execution object and the target object.

[0108] Specifically, the execution object to be determined may be an object or position other than the preliminary execution object that is at the same level as the preliminary execution object in the hierarchical scene graph. The execution object to be determined is determined based on the level of the preliminary execution object in the hierarchical scene graph and the execution scope of the task instruction, and whether there is a candidate execution object in the hierarchical scene graph is determined based on the execution object to be determined.

[0109] Furthermore, the large model is used to perform association analysis on the execution objects to be determined one by one according to the task knowledge base, and the association between each execution object to be determined and the target object time is determined to determine whether there is a candidate execution object in the hierarchical scene graph. It can be understood that if the target object is not in the hierarchical scene graph, the preliminary execution object is obtained by performing association analysis based on the task knowledge base and the target object using the large model, and the association between the candidate execution object and the target object in the task instruction is weaker than the association between the preliminary execution object and the target object.

[0110] In some embodiments, if the target object is in the hierarchical scene graph, the preliminary execution object can be determined as the scene node connected to the target object in the hierarchical scene graph, thereby obtaining a preliminary task plan. In this case, if the actual environment changes, and the task instruction is not completed after the embodied robot is controlled to perform the task according to the preliminary task plan, it is necessary to determine the candidate execution object from the hierarchical scene graph through object screening and association analysis to adjust the preliminary task plan. It is understandable that in this embodiment, the correlation between the candidate execution object and the target object is irrelevant to the correlation between the preliminary execution object and the target object.

[0111] Reference Figure 7 As shown, as an embodiment of the present application, the method further includes:

[0112] S410. According to the adjustment task plan, control the embodied robot to execute the subtasks in the adjustment task plan, and feed back the adjustment execution results.

[0113] S420. When the adjustment execution result indicates that the embodied robot has completed the adjustment task plan but has not yet completed the task instruction, the above-mentioned dynamic planning adjustment process is executed again to obtain a new adjustment task plan as the adjustment task plan to repeatedly control the embodied robot to execute the corresponding subtasks until the task instruction end conditions are met.

[0114] Specifically, the dynamic planning adjustment process may include: after determining the candidate execution object, according to the target object, the candidate execution object and the task instruction, the corresponding third task tool is determined from the preset task tool to generate a third subtask for adjusting the preliminary task plan. The third subtask is task parsed using the large model and the task knowledge base, and the fourth task tool is determined from the preset task tool to generate a fourth subtask for adjusting the preliminary task plan. After obtaining the third subtask and the fourth subtask, the fourth subtask is associated with the third subtask according to the execution order of the third subtask to obtain the adjusted task plan. It can be understood that the adjusted task plan is obtained by replacing the first subtask in the preliminary task plan with the third subtask, and replacing the second subtask in the preliminary task plan with the fourth subtask.

[0115] Furthermore, when the above dynamic programming adjustment process is repeatedly executed, the correlation between the candidate execution object and the target object determined in each round is weaker than the correlation between the candidate execution object and the target object in the previous round, until the task instruction end condition is met.

[0116] Furthermore, the task instruction termination condition may include a completion condition and an adjustment number condition, wherein the completion condition is that the embodied robot completes the task instruction, and the adjustment number condition is that the number of times the above-mentioned dynamic planning adjustment process is repeatedly executed reaches a preset threshold. When any of the above-mentioned task instruction termination conditions is met, the currently executed adjustment task planning is terminated, and the embodied robot is controlled to return to feedback the task instruction result to the user. It should be noted that if it is determined in any round that there is no candidate execution object in the hierarchical scene graph, the above-mentioned dynamic planning adjustment process cannot be executed at this time, the currently executed adjustment task planning is terminated, and the embodied robot is controlled to return to feedback the task instruction result to the user.

[0117] Accordingly, please refer to Figure 8 , an embodiment of the present application provides an embodied robot task planning system, the system comprising:

[0118] The preliminary task planning module 100 is used to perform task decoupling and planning on the task instructions of the embodied robot based on the pre-built hierarchical scene graph and task knowledge base, and obtain a preliminary task plan including multiple subtasks; wherein the subtasks correspond to preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are objects or positions that the embodied robot needs to reach during the execution of the task instructions;

[0119] A preliminary task execution module 200, used to control the embodied robot to execute multiple subtasks according to the preliminary task plan, and to feed back preliminary execution results of the multiple subtasks;

[0120] The dynamic planning adjustment module 300 is used to determine whether there is a candidate execution object in the hierarchical scene graph when the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction; wherein the candidate execution object is an object or position that can replace the preliminary execution object;

[0121] The task planning adjustment module 400 is used to perform dynamic planning adjustment according to the candidate execution objects when there are candidate execution objects in the hierarchical scene graph, so as to obtain an adjusted task plan to control the embodied robot to complete the task instructions.

[0122] In some optional implementations, the preliminary mission planning module 100 includes:

[0123] The decoupling and identification unit is used to use the large model to perform task decoupling and object identification on the task instructions, and obtain the target object and execution range in the task instructions; wherein the execution range is the spatial range in which the embodied robot executes the task instructions.

[0124] The object screening unit is used to screen objects in the hierarchical scene graph according to the target object, execution scope and task knowledge base to obtain preliminary execution objects.

[0125] The task planning unit is used to plan the task according to the target object and the preliminary execution object, obtain multiple subtasks, and formulate a preliminary task plan.

[0126] In some optional implementations, the object screening unit includes:

[0127] The level determination subunit is used to determine the target level of the target object in the hierarchical scene graph.

[0128] The candidate determination subunit is used to determine multiple candidate execution objects existing in the execution scope at an upper level of the target level in the hierarchical scene graph.

[0129] The association analysis subunit is used to perform association analysis on multiple candidate execution objects and target objects one by one according to the task knowledge base, and determine the preliminary execution object that meets the association condition from the multiple candidate execution objects.

[0130] In some optional implementations, the preliminary task planning module 100 further includes a scene graph construction unit, including:

[0131] The image acquisition subunit is used to acquire images of the scene in which the embodied robot is located to obtain multiple scene images.

[0132] The image description subunit is used to use the large model to perform image description on multiple scene images to obtain description results.

[0133] The hierarchical grading subunit is used to hierarchically grade the objects or positions in the scene where the embodied robot is located based on the task knowledge base and the description results, and determine the scene level corresponding to each object or position.

[0134] The scene graph construction subunit is used to construct a scene graph structure according to the correlation between objects or positions and the corresponding scene hierarchy to obtain a hierarchical scene graph.

[0135] In some optional embodiments, the embodied robot has a plurality of preset task tools; the task planning unit includes:

[0136] The first subtask generating subunit is used to determine the corresponding first task tool from the preset task tools according to the target object, the preliminary execution object and the task instruction, so as to generate the first subtask.

[0137] The second subtask generating subunit is used to perform task analysis on the first subtask by using the large model and the task knowledge base, and determine the second task tool from the preset task tools to generate the second subtask.

[0138] The subtask associating subunit is used to associate the second subtask with the first subtask according to the execution order to obtain a preliminary task plan.

[0139] In some optional implementations, the dynamic programming adjustment module 300 includes:

[0140] The candidate determination unit is used to determine the to-be-determined execution object from the hierarchical scene graph according to the level of the preliminary execution object in the hierarchical scene graph and the execution scope of the task instruction.

[0141] The association analysis unit is used to use the large model to perform association analysis on the to-be-determined execution object according to the task knowledge base, and determine whether there is a candidate execution object in the hierarchical scene graph; wherein the association between the candidate execution object and the target object in the task instruction is weaker than the association between the preliminary execution object and the target object.

[0142] In some optional embodiments, the system further includes a cycle adjustment module, including:

[0143] The adjustment task execution unit is used to control the embodied robot to execute the subtasks in the adjustment task plan according to the adjustment task plan, and to feedback the adjustment execution results.

[0144] The loop adjustment unit is used to execute the above-mentioned dynamic planning adjustment process again when the adjustment execution result indicates that the embodied robot has completed the adjustment task plan but has not yet completed the task instruction, so as to obtain a new adjustment task plan as the adjustment task plan, so as to repeatedly control the embodied robot to execute the corresponding subtasks until the task instruction end condition is met.

[0145] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0146] The embodied robot task planning system in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0147] See also Fig. 9 , Fig. 9It is a structural diagram of a controller provided by an embodiment of the present application. As shown in the figure, the controller includes: one or more processors 10, a memory 20, and an interface for connecting various components, including a high-speed interface and a low-speed interface. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the controller, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple controllers can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.

[0148] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0149] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0150] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the controller, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the controller via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0151] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0152] The controller further comprises a communication interface 30 for the controller to communicate with other devices or a communication network.

[0153] An embodiment of the present application further provides an embodied robot, comprising the controller described in the above embodiment, and the controller controls the embodied robot through the method shown in the above embodiment.

[0154] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0155] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a controller reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the controller executes the method of any embodiment of the present application.

[0156] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

[0157] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0158] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0159] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0160] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0164] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0166] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for embodied robot task planning, characterized in that: The method comprises: Based on the pre-constructed hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are task-decoupled and planned to obtain a preliminary task plan including a plurality of subtasks; wherein the subtasks correspond to preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are objects or positions that the embodied robot needs to reach during the execution of the task instructions; Controlling the embodied robot to execute the multiple subtasks according to the preliminary task plan, and feeding back preliminary execution results of the multiple subtasks; In the case where the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction, determining whether there is a candidate execution object in the hierarchical scene graph; wherein the candidate execution object is an object or position that can replace the preliminary execution object; In the case where the candidate execution object exists in the hierarchical scene graph, dynamic planning adjustment is performed according to the candidate execution object to obtain an adjusted task plan to control the embodied robot to complete the task instruction.

2. The method according to claim 1, characterized in that: Based on the pre-built hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are decoupled and planned to obtain a preliminary task plan containing multiple subtasks, including: Using the large model to perform task decoupling and object recognition on the task instruction, to obtain the target object and execution range in the task instruction; wherein the execution range is the spatial range in which the embodied robot executes the task instruction; According to the target object, the execution scope and the task knowledge base, object screening is performed in the hierarchical scene graph to obtain the preliminary execution object; Task planning is performed according to the target object and the preliminary execution object to obtain the multiple subtasks and formulate the preliminary task plan.

3. The method according to claim 2, characterized in that The object screening in the hierarchical scene graph according to the target object, the execution scope and the task knowledge base to obtain the preliminary execution object includes: Determining a target level at which the target object is located in the hierarchical scene graph; Determining, at a level above the target level in the hierarchical scene graph, a plurality of candidate execution objects existing within the execution scope; According to the task knowledge base, the multiple candidate execution objects and the target object are analyzed for association one by one, and the preliminary execution object that meets the association condition is determined from the multiple candidate execution objects.

4. The method according to claim 2, characterized in that: The hierarchical scene graph is obtained in the following manner: Capturing images of the scene in which the embodied robot is located to obtain a plurality of scene images; Using the large model to perform image description on the multiple scene images to obtain description results; Based on the task knowledge base, hierarchically classify the objects or positions in the scene where the embodied robot is located according to the description results, and determine the scene level corresponding to each object or position; A scene graph structure is constructed according to the correlation between the objects or positions and the corresponding scene hierarchies to obtain the hierarchical scene graph.

5. The method according to claim 2, characterized in that: The embodied robot is provided with a plurality of preset task tools; the task planning is performed according to the target object and the preliminary execution object to obtain the plurality of subtasks, and the preliminary task planning is formulated, including: According to the target object, the preliminary execution object and the task instruction, a corresponding first task tool is determined from the preset task tools to generate a first subtask; Performing task analysis on the first subtask using the large model and the task knowledge base, and determining a second task tool from the preset task tools to generate a second subtask; The second subtask is associated with the first subtask in an execution order to obtain the preliminary task plan.

6. The method according to claim 1, characterized in that The determining whether there is a candidate execution object in the hierarchical scene graph includes: Determining the to-be-determined execution object from the hierarchical scene graph according to the level at which the preliminary execution object is located in the hierarchical scene graph and the execution scope of the task instruction; Using the large model, an association analysis is performed on the to-be-determined execution object according to the task knowledge base to determine whether the candidate execution object exists in the hierarchical scene graph; wherein the association between the candidate execution object and the target object in the task instruction is weaker than the association between the preliminary execution object and the target object.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: According to the adjustment task plan, controlling the embodied robot to execute the subtasks in the adjustment task plan, and feeding back the adjustment execution result; When the adjustment execution result indicates that the embodied robot has completed the adjustment task plan but has not yet completed the task instruction, the above-mentioned dynamic planning adjustment process is executed again to obtain a new adjustment task plan as the adjustment task plan, so as to repeatedly control the embodied robot to execute the corresponding subtasks until the task instruction end condition is met.

8. An embodied robot task planning system, characterized in that: The system comprises: A preliminary task planning module, for performing task decoupling and planning on the task instructions of the embodied robot based on a pre-built hierarchical scene graph and a task knowledge base, and obtaining a preliminary task plan including a plurality of subtasks; wherein the subtasks correspond to preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are objects or positions that the embodied robot needs to reach during the execution of the task instructions; A preliminary task execution module, used to control the embodied robot to execute the multiple subtasks according to the preliminary task plan, and to feed back preliminary execution results of the multiple subtasks; A dynamic planning adjustment module, configured to determine whether there is a candidate execution object in the hierarchical scene graph when the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction; wherein the candidate execution object is an object or position that can replace the preliminary execution object; The task planning adjustment module is used to perform dynamic planning adjustment according to the candidate execution object when the candidate execution object exists in the hierarchical scene graph, so as to obtain an adjusted task plan to control the embodied robot to complete the task instruction.

9. A controller, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. An embodied robot, characterized in that: Comprising the controller as claimed in claim 9.

Images