A method, system, controller and embodied robot for embodied robot task planning

Abstract

The application relates to the technical field of embodied intelligence, and discloses an embodied robot task planning method, system, controller and embodied robot, the method comprising the following steps: based on a hierarchical scene graph and a task knowledge base, task decoupling and planning are performed on a task instruction of an embodied robot to obtain a preliminary task planning; the embodied robot is controlled to execute a subtask according to the preliminary task planning, and a preliminary execution result is fed back; in the case that the preliminary execution result indicates that the embodied robot completes the preliminary task planning but does not complete the task instruction, it is judged whether there is a candidate execution object in the hierarchical scene graph; in the case that there is a candidate execution object in the hierarchical scene graph, dynamic planning adjustment is performed according to the candidate execution object to obtain an adjusted task planning. The application has the beneficial effect of improving the perception ability of the embodied robot to the actual environment, so that the task planning can be adjusted according to the actual environment, and the task execution effect of the embodied robot is improved.

Description

Technical Field

[0001] This application relates to the field of embodied intelligence technology, and in particular to an embodied robot task planning method, system, controller, and embodied robot. Background Technology

[0002] With the development of technology, avatars are appearing in various aspects of people's lives. In these scenarios, avatars execute corresponding tasks based on user-issued instructions, thus serving the user. After a user issues a task instruction, the avatar's controller receives the instruction and uses a large model to perform reasoning and analysis, thereby formulating a corresponding task plan for the avatar. However, the flexibility in controlling avatars to execute tasks based on this task plan still needs improvement. Summary of the Invention

[0003] This application provides a task planning method, system, controller, and embodied robot. It plans the task instructions of the embodied robot based on a hierarchical scene diagram, which solves the technical problem of lacking real-time perception of the actual environment when planning tasks in related technologies. This allows the task planning to be adjusted according to the actual environment, thereby improving the effectiveness of the embodied robot in completing task instructions.

[0004] To achieve the above objectives, the main technical solutions adopted in this application include:

[0005] In a first aspect, embodiments of this application provide a task planning method for an embodied robot, the method comprising:

[0006] Based on a pre-built hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are decoupled and planned to obtain a preliminary task plan containing multiple sub-tasks; wherein, each sub-task corresponds to a preliminary execution object in the hierarchical scene graph, and the preliminary execution object is the object or position that the embodied robot needs to reach during the execution of the task instructions.

[0007] According to the preliminary task plan, the robot is controlled to execute the multiple sub-tasks, and the preliminary execution results of the multiple sub-tasks are fed back.

[0008] If the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction, it is determined whether there is a candidate execution object in the hierarchical scene graph; wherein, the candidate execution object is an object or position that can replace the preliminary execution object;

[0009] If the candidate execution object exists in the hierarchical scene diagram, dynamic planning and adjustment are performed based on the candidate execution object to obtain an adjusted task plan, so as to control the embodied robot to complete the task instruction.

[0010] The embodied robot task planning method proposed in this application utilizes a pre-constructed hierarchical scene graph and task knowledge base to perform task planning for the embodied robot, obtaining a preliminary task plan. Based on the execution results of sub-tasks in the preliminary task plan, it infers whether the preliminary task plan can be adjusted. This method improves the embodied robot's ability to master the actual environment by constructing a hierarchical scene graph of the scene in which the embodied robot is located, thereby improving its performance in complex environments. Furthermore, compared with related technologies that rely on preliminary task planning, this method also uses the execution results of sub-tasks in the preliminary task plan and the hierarchical scene graph for inference, enabling adjustments to the preliminary task plan based on environmental changes in the actual scene. This improves the embodied robot's responsiveness to changes in the actual environment and enhances its flexibility when performing tasks.

[0011] Optionally, based on a pre-built hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are decoupled and planned to obtain a preliminary task plan containing multiple sub-tasks, including:

[0012] The task instructions are decoupled and object identified using a large model to obtain the target object and execution range in the task instructions; wherein, the execution range is the spatial range in which the embodied robot executes the task instructions;

[0013] Based on the target object, the execution scope, and the task knowledge base, object filtering is performed in the hierarchical scene graph to obtain the preliminary execution object;

[0014] Based on the target object and the initial execution object, task planning is performed to obtain the multiple sub-tasks, and the initial task plan is formulated.

[0015] Optionally, the step of filtering objects in the hierarchical scene graph based on the target object, the execution scope, and the task knowledge base to obtain the preliminary execution object includes:

[0016] Determine the target level of the target object in the hierarchical scene graph;

[0017] In the hierarchical scene diagram, at the level above the target level, determine multiple candidate execution objects existing within the execution range;

[0018] Based on the task knowledge base, the multiple candidate execution objects are analyzed one by one with the target object to determine the preliminary execution object that meets the correlation conditions from the multiple candidate execution objects.

[0019] Optionally, the hierarchical scene diagram is obtained as follows:

[0020] Images of the scene where the embodied robot is located are captured to obtain multiple scene images;

[0021] A large model is used to perform image description on the multiple scene images to obtain the description results;

[0022] Based on the task knowledge base, the objects or positions in the scene where the embodied robot is located are classified hierarchically according to the description results, and the scene level corresponding to each object or position is determined.

[0023] The scene graph structure is constructed based on the correlation between objects or locations and the corresponding scene hierarchy to obtain the hierarchical scene graph.

[0024] Optionally, the embodied robot possesses multiple preset task tools; the step of performing task planning based on the target object and the initial execution object to obtain the multiple sub-tasks, and formulating the initial task plan, includes:

[0025] Based on the target object, the initial execution object, and the task instructions, a corresponding first task tool is determined from the preset task tools to generate a first subtask;

[0026] The first subtask is parsed using the large model and the task knowledge base, and a second task tool is determined from the preset task tools to generate the second subtask.

[0027] The second subtask is associated with the first subtask according to the execution order to obtain the preliminary task plan.

[0028] Optionally, determining whether there is a candidate execution object in the hierarchical scene graph includes:

[0029] Based on the level of the preliminary execution object in the hierarchical scene diagram and the execution scope of the task instruction, determine the execution object to be determined from the hierarchical scene diagram;

[0030] Using a large model, an association analysis is performed on the object to be executed based on the task knowledge base to determine whether the candidate object exists in the hierarchical scene graph; wherein, the association between the candidate object and the target object in the task instruction is weaker than the association between the initial object and the target object.

[0031] Optionally, the method further includes:

[0032] According to the adjusted task plan, the robot is controlled to execute the sub-tasks in the adjusted task plan and the results of the adjusted execution are fed back.

[0033] If the adjustment execution result indicates that the embodied robot has completed the adjustment task planning but still has not completed the task instruction, the above dynamic planning adjustment process is executed again to obtain a new adjustment task plan. This new adjustment task plan is then used to repeatedly control the embodied robot to execute the corresponding sub-tasks until the task instruction termination condition is met.

[0034] Secondly, embodiments of this application provide a task planning system for an embodied robot, characterized in that the system includes:

[0035] The preliminary task planning module is used to decouple and plan the task instructions of the embodied robot based on a pre-built hierarchical scene graph and task knowledge base, so as to obtain a preliminary task plan containing multiple sub-tasks; wherein, the sub-tasks correspond to preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are the objects or positions that the embodied robot needs to reach during the execution of the task instructions.

[0036] The preliminary task execution module is used to control the embodied robot to execute the multiple sub-tasks according to the preliminary task plan, and to provide feedback on the preliminary execution results of the multiple sub-tasks;

[0037] The dynamic planning adjustment module is used to determine whether there is a candidate execution object in the hierarchical scene graph when the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction; wherein, the candidate execution object is an object or position that can replace the preliminary execution object;

[0038] The task planning and adjustment module is used to dynamically plan and adjust the task plan based on the candidate execution object when the candidate execution object exists in the hierarchical scene diagram, so as to control the embodied robot to complete the task instruction.

[0039] Thirdly, embodiments of this application provide a controller, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the method described in any of the above embodiments.

[0040] Fourthly, embodiments of this application provide a body-worn robot, including the controller described in any of the above embodiments.

[0041] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which are used to cause a computer to perform the method described in any one of the above embodiments.

[0042] Sixthly, embodiments of this application provide a computer program product, including computer instructions, which are used to cause a computer to perform the method described in any of the above embodiments. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating the steps of the embodied robot task planning method provided in this application embodiment;

[0045] Figure 2 This application provides a flowchart illustrating the steps involved in developing an initial task plan in its embodiments.

[0046] Figure 3 A flowchart of the initial task planning is provided for embodiments of this application;

[0047] Figure 4 This application provides a flowchart illustrating the steps for determining the initial execution target in an embodiment of the application.

[0048] Figure 5 This application provides a flowchart illustrating the steps involved in obtaining preliminary task planning in its embodiments.

[0049] Figure 6 This application provides a flowchart illustrating the steps for determining whether a candidate execution object exists in a hierarchical scene graph.

[0050] Figure 7 A flowchart illustrating the steps of repeatedly performing dynamic programming adjustments in the embodiments of this application is provided;

[0051] Figure 8 A block diagram of the embodied robot task planning system provided in the embodiments of this application;

[0052] Figure 9 This is a schematic diagram of the structure of a controller provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] With the development of technology, embodied robots are appearing in various scenarios of people's lives, such as smart homes, logistics and transportation, and service industries. In different scenarios, embodied robots respond to task instructions issued by users and execute corresponding tasks according to the instructions, thereby serving users and achieving the goals that users require from the embodied robot. In related technologies, after receiving task instructions from users, the controller used to control the embodied robot typically uses a large model to perform reasoning analysis on the task instructions. By leveraging the knowledge base and reasoning logic built into the large model, complex task instructions are decoupled into multiple sub-tasks, thereby formulating corresponding task plans for the embodied robot.

[0055] In the aforementioned technologies, large models are typically trained using offline pre-training data. However, offline pre-training data cannot reflect changes in the actual environment in which the embodied robot is located. This results in the embodied robot being unable to perceive changes in the actual environment in a timely manner, leading to deviations in the robot's task execution results.

[0056] Secondly, during task execution, embodied robots are highly dependent on task planning, which leads to a lack of necessary autonomous adaptability. They are unable to adjust task planning based on task execution results or changes in the actual environment, thus limiting their performance in complex environments.

[0057] To address the aforementioned issues, this application provides a task planning method, system, controller, and embodied robot for embodied robots. The method includes: decoupling and planning the task instructions of the embodied robot based on a hierarchical scene graph and a task knowledge base to obtain a preliminary task plan containing multiple sub-tasks; controlling the embodied robot to execute the sub-tasks according to the preliminary task plan and providing feedback on the preliminary execution results of the sub-tasks; if the preliminary execution results indicate that the embodied robot has completed the preliminary task plan but has not completed the task instructions, determining whether there are candidate execution objects in the hierarchical scene graph; if there are candidate execution objects in the hierarchical scene graph, dynamically adjusting the planning based on the candidate execution objects to obtain an adjusted task plan, thereby controlling the embodied robot to complete the task instructions.

[0058] The embodied robot task planning method proposed in this application utilizes a pre-constructed hierarchical scene graph and task knowledge base to perform task planning for the embodied robot, obtaining a preliminary task plan. Based on the execution results of sub-tasks in the preliminary task plan, it infers whether the preliminary task plan can be adjusted. This method improves the embodied robot's ability to master the actual environment by constructing a hierarchical scene graph of the scene in which the embodied robot is located, thereby improving the embodied robot's performance in complex environments.

[0059] Furthermore, compared with related technologies that rely on preliminary task planning, this method also uses the execution results of sub-tasks and hierarchical scene diagrams in the preliminary task planning to perform reasoning. This allows the preliminary task planning to be adjusted according to environmental changes in the actual scene, improving the embodied robot's responsiveness to changes in the actual environment and enhancing the flexibility of the embodied robot when performing tasks.

[0060] The avatar robot task planning method provided in this specification can be applied to avatar robots capable of generating task plans and executing corresponding sub-tasks based on task instructions, including avatar robots and smart home devices. It is understood that, after adaptive modifications, this method can also be used to control intelligent devices other than the aforementioned avatar robots, such as virtual intelligent agents, industrial avatar robots, and medical avatar robots.

[0061] According to an embodiment of this application, a method for planning tasks for an embodied robot is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0062] This embodiment provides a task planning method for embodied robots, which can be used in the aforementioned embodied robots capable of generating task plans and executing corresponding sub-tasks based on task instructions. (Refer to...) Figure 1 As shown, the method includes:

[0063] S100. Based on the pre-built hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are decoupled and planned to obtain a preliminary task plan containing multiple sub-tasks; wherein, each sub-task corresponds to a preliminary execution object in the hierarchical scene graph, and the preliminary execution object is the object or position that the embodied robot needs to reach during the execution of the task instructions.

[0064] S200. Control the embodied robot to execute multiple sub-tasks according to the preliminary task plan, and provide feedback on the preliminary execution results of the multiple sub-tasks.

[0065] S300. If the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instructions, determine whether there is a candidate execution object in the hierarchical scene graph; wherein, the candidate execution object is an object or position that can replace the preliminary execution object.

[0066] S400. When there are candidate execution objects in the hierarchical scene diagram, dynamic planning and adjustment are performed based on the candidate execution objects to obtain the adjusted task plan, so as to control the embodied robot to complete the task instructions.

[0067] The hierarchical scene graph can be a hierarchical graph structure representing the actual environment in which the embodied robot is located. The hierarchical scene graph contains multiple scene nodes corresponding to objects or locations in the actual environment. Each scene node has a corresponding level and is connected to other scene nodes via edges. Each level in the hierarchical scene graph represents the type of each scene node, and the edges between scene nodes represent the spatial positional relationships and subordinate relationships between the corresponding objects or locations. In some embodiments, based on the execution results of the corresponding subtasks in the preliminary task planning and adjustment task planning, the information of each scene node in the hierarchical scene graph can be updated, enabling the hierarchical scene graph to accurately reflect the actual environment.

[0068] The hierarchical scene graph is illustrated exemplarily when the embodied robot's actual environment is inside a house. The hierarchical scene graph can include four levels: the first level can be a floor; the second level can be rooms, such as bedrooms, living rooms, and kitchens; the third level can be large objects that don't move frequently in reality, such as beds, sofas, and bookshelves; and the fourth level can be small objects that can be moved at any time, such as books and water glasses. The small objects that serve as scene nodes in the fourth level are defined as transient objects. Their position and state information are temporarily stored in the hierarchical scene graph. When the feedback execution result indicates that the position or state information of the small object has changed compared to the hierarchical scene graph, the scene nodes in the fourth level of the hierarchical scene graph are updated to improve the consistency between the hierarchical scene graph and the actual environment.

[0069] The task knowledge base can be a repository containing knowledge related to task instructions. This knowledge can include common sense and specialized knowledge related to the task instructions. During task planning, the large model can identify task instructions, determine the target object within those instructions, and then, based on relevant knowledge, decouple and plan the task according to the instructions and target object, thus obtaining a preliminary task plan containing multiple subtasks. It should be noted that each subtask corresponds to a preliminary execution object in the hierarchical scene graph. The preliminary execution object is connected to the target object in the hierarchical scene graph via edges, indicating a relationship between them, such as a subordinate relationship or a relative positional relationship.

[0070] To illustrate the role of the task knowledge base, if the task instruction is "find the water glass in the bedroom," using a large model to identify the task instruction, we can determine that the target object is the water glass, and the purpose of the task instruction is to find the location of the target object. If the water glass is not in the hierarchical scene diagram, analysis based on common sense knowledge contained in the task knowledge base can determine that the water glass would be placed on a flat and hard surface, and based on human habits, the height at which the water glass is placed can be determined. Therefore, according to the hierarchical scene diagram, the objects in the bedroom that meet these requirements are the coffee table and the bedside table. At this point, the coffee table and the bedside table are taken as the initial execution objects. Based on the target object and the initial execution objects, task planning is performed to obtain the preliminary task plan.

[0071] Specifically, the embodied robot receives task instructions from the user, which can be in text or voice format. For voice instructions, they are converted into text using speech recognition. In response to text instructions, based on a pre-built hierarchical scene graph and task knowledge base, a large model is used to decouple and identify the task instructions, thereby determining the target object in the task instructions—the object the embodied robot primarily interacts with. Based on the target scene and task instructions, the task knowledge base is used to determine the initial execution object corresponding to the target object from the hierarchical scene graph. Task planning is then performed based on the target object and the initial execution object to obtain the preliminary task plan.

[0072] Furthermore, the initial task planning includes multiple sub-tasks. Based on the initial task planning, the embodied robot is controlled to execute these sub-tasks sequentially, and the initial execution results of these sub-tasks are fed back in real time. If the initial execution result of any sub-task indicates that the embodied robot has completed the task instruction, the execution of the sub-task stops, and the embodied robot is controlled to provide feedback to the user on the execution result of the task instruction. If the initial execution results of all sub-tasks indicate that the embodied robot has not completed the task instruction, that is, the embodied robot has completed all the initial task planning but has not completed the task instruction, then it is necessary to reason and judge whether to dynamically adjust the initial task planning based on the hierarchical scene diagram.

[0073] Furthermore, the condition for dynamically adjusting the initial task plan is the existence of candidate execution objects in the hierarchical scene graph. These candidate execution objects are objects or positions that can replace the initial execution object. Essentially, similar to the initial execution object, the candidate execution object is connected to the target object via edges, indicating a relationship between them, such as a subordinate relationship or a relative positional relationship. When candidate execution objects exist in the hierarchical scene graph, multiple sub-tasks are generated based on the candidate execution objects and the target object to dynamically adjust the initial task plan, thereby obtaining an adjusted task plan to control the embodied robot to complete the task instructions.

[0074] Compared with related technologies, this application first constructs a hierarchical scene map of the actual environment in which the embodied robot is located. This hierarchical scene map enhances the embodied robot's ability to master complex real-world environments, thereby improving the accuracy of the resulting task planning and its consistency with the actual environment. Secondly, the hierarchical scene map in this application can be updated based on the execution results of the embodied robot after performing sub-tasks, ensuring that the hierarchical scene map accurately reflects the actual environment and improving the embodied robot's ability to perceive changes in the actual environment. Furthermore, this application also uses the hierarchical scene map and the execution results of sub-tasks to reason and judge whether the initial task plan can be adjusted, enabling the embodied robot to respond promptly to changes in the actual environment and improving its dynamic response capability to these changes.

[0075] The embodied robot task planning method provided in this embodiment utilizes a pre-constructed hierarchical scene graph and task knowledge base to perform task planning for the embodied robot, obtaining a preliminary task plan. Based on the execution results of sub-tasks in the preliminary task plan, it infers whether the preliminary task plan can be adjusted. This method improves the embodied robot's ability to master the actual environment by constructing a hierarchical scene graph of the scene in which the embodied robot is located, thereby improving the embodied robot's performance in complex environments.

[0076] Furthermore, compared with related technologies that rely on preliminary task planning, this method also uses the execution results of sub-tasks and hierarchical scene diagrams in the preliminary task planning to perform reasoning. This allows the preliminary task planning to be adjusted according to environmental changes in the actual scene, improving the embodied robot's responsiveness to changes in the actual environment and enhancing the flexibility of the embodied robot when performing tasks.

[0077] Reference Figure 2 As shown, in one embodiment of this application, based on a pre-built hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are decoupled and planned to obtain a preliminary task plan containing multiple sub-tasks, including:

[0078] S110. Use a large model to decouple the task instructions and identify the objects to obtain the target objects and execution ranges in the task instructions; where the execution range is the spatial range within which the embodied robot executes the task instructions.

[0079] S120. Based on the target object, execution scope, and task knowledge base, filter objects in the hierarchical scene diagram to obtain preliminary execution objects.

[0080] S130. Based on the target object and the initial execution object, perform task planning to obtain multiple sub-tasks and formulate a preliminary task plan.

[0081] Specifically, using a large model to decouple and identify task instructions includes: using a large model to decouple and identify text-based task instructions, extracting the corresponding target objects and execution scopes. It can be understood that the spatial range corresponding to the execution scope has corresponding scene nodes in the hierarchical scene graph, and there is a subordinate or relative spatial relationship between the target object and the execution scope.

[0082] Furthermore, based on the obtained target object and execution scope, corresponding scene nodes are determined in the hierarchical scene graph, and multiple scene nodes that may serve as initial execution objects are identified in the hierarchical scene graph based on these scene nodes. The task knowledge base is used to filter the multiple scene nodes that may serve as initial execution objects to obtain the initial execution objects. After obtaining the target object and the initial execution objects, tasks are combined based on the target object and the initial execution objects to obtain multiple subtasks, thereby enabling the formulation of preliminary task planning based on these multiple subtasks. Each subtask corresponds to a target object and a specific initial execution object.

[0083] For example, refer to Figure 3 As shown, in this embodiment, the task instruction is "find a water glass in the bedroom." The large model determines that the target object in this task instruction is a water glass, and the execution scope is the bedroom. Based on the target object, execution scope, and task knowledge base, object filtering is performed in the hierarchical scene graph, identifying the bedside table and coffee table as the initial execution objects. Two sub-tasks are derived from the target object and the initial execution objects: "find a water glass on the bedside table" and "find a water glass on the coffee table." A preliminary task plan is obtained based on these sub-tasks, and the embodied robot is controlled to execute these sub-tasks according to this preliminary task plan.

[0084] Reference Figure 4 As shown in one embodiment of this application, based on the target object, execution scope, and task knowledge base, object filtering is performed in the hierarchical scene graph to obtain preliminary execution objects, including:

[0085] S122. Determine the target level of the target object in the hierarchical scene diagram.

[0086] S124. In the hierarchical scene diagram, identify multiple candidate execution objects existing within the execution scope at the level above the target level.

[0087] S126. Based on the task knowledge base, perform a one-to-one association analysis between multiple candidate execution objects and the target object, and determine the preliminary execution object that meets the association conditions from multiple candidate execution objects.

[0088] Specifically, in a hierarchical scene diagram, different levels have subordinate or spatial relationships. A scene node at a lower level may be subordinate to a scene node at a higher level, or it may be located within a scene node at a higher level in the actual environment. After determining the target object, the level above the target level is the level where the initial execution object may reside. At this point, multiple candidate execution objects are determined at this higher level based on the execution scope. It is understandable that there may be no correlation between the candidate execution objects and the target object.

[0089] Furthermore, using a large model, based on common sense or professional knowledge in the task knowledge base, a step-by-step association analysis is performed on multiple candidate execution objects and the target object. If the association between any candidate execution object and the target object exceeds a threshold, then the candidate execution object meets the association condition and is used as the initial execution object. In some embodiments, the association analysis method can be to analyze the weights of edges in the hierarchical scene graph, or it can be to query the task knowledge base based on the initial execution object and the target object to obtain the association between the initial execution object and the target object in the hierarchical scene graph.

[0090] As one embodiment of this application, the hierarchical scene diagram is obtained according to the following method:

[0091] S102. Collect images of the scene where the embodied robot is located to obtain multiple scene images.

[0092] S104. Use a large model to perform image description on multiple scene images and obtain the description results.

[0093] S106. Based on the task knowledge base, classify the objects or positions in the scene where the embodied robot is located according to the description results, and determine the scene level corresponding to each object or position.

[0094] S108. Construct a scene graph structure based on the correlation between each object or position and the corresponding scene hierarchy to obtain a hierarchical scene graph.

[0095] Specifically, multi-angle image acquisition is performed on the actual environment in which the embodied robot is located, resulting in multiple visual images of the actual environment, which serve as scene images. Corresponding Prompt statements are designed and input into the large model to guide it in describing the actual environment based on the scene images, yielding a description result. It is understood that the description result can include the types of objects in the actual scene, object attributes, and the relative positions between objects.

[0096] Furthermore, by utilizing common sense or professional knowledge from the task knowledge base, objects or locations in the actual scene are hierarchically classified based on the description results to determine the scene level corresponding to each object or location. The hierarchical classification steps may include: classifying objects or locations in the actual scene according to the description results, and sorting different categories of objects or locations according to the classification results, thereby determining the scene level corresponding to each object or location.

[0097] Reference Figure 5 As shown, in one embodiment of this application, the embodied robot possesses multiple preset task tools; it performs task planning based on the target object and the initial execution object, obtaining multiple sub-tasks, and formulates an initial task plan, including:

[0098] S132. Based on the target object, the initial execution object, and the task instructions, determine the corresponding first task tool from the preset task tools to generate the first subtask.

[0099] S134. Utilize the large model and task knowledge base to perform task analysis on the first subtask, determine the second task tool from the preset task tools, and generate the second subtask.

[0100] S136. Associate the second subtask with the first subtask according to the execution order to obtain the preliminary task plan.

[0101] The preset task tools can be functions inherent to the embodied robot, implemented by corresponding tools installed within it. For example, preset task tools may include Move, Look, and Say (responding to task command results). Each preset task tool is invoked and executed through an API interface, selected based on the action in the task command. It should be noted that the list of preset task tools is open; users can add new preset task tools to the list according to the needs of the actual scenario, thereby expanding the range of tasks the embodied robot can perform.

[0102] Specifically, the first task tool is determined from the preset task tools according to the action in the task instruction, and the first subtask is generated based on the target object, the initial execution object and the first task tool.

[0103] Furthermore, the different initial execution objects within different first subtasks lead to different execution positions for the first subtasks, preventing them from being executed consecutively. Therefore, second subtasks are needed as connecting subtasks between different first subtasks. The initial task plan is composed of the first and second subtasks, enabling the embodied robot to execute tasks continuously. There can be multiple second subtasks. Following the execution order of the first subtasks, multiple second subtasks are associated with their corresponding first subtasks, thus obtaining the initial task plan.

[0104] To illustrate how the second subtask is generated, if the initial execution objects of the first subtask include a bedside table and a coffee table, and the bedside table and coffee table are in different positions in the actual scenario, then by using the large model and task knowledge base to perform task parsing on the first subtask, it can be determined that the tool for the second task is "Move". In this case, the second subtask used to connect the first subtask is "Move from the bedside table to the coffee table".

[0105] Reference Figure 6 As shown in one embodiment of this application, determining whether a candidate execution object exists in the hierarchical scene graph includes:

[0106] S310. Based on the level of the preliminary execution object in the hierarchical scene diagram and the execution scope of the task instructions, determine the execution object to be determined from the hierarchical scene diagram.

[0107] S320. Using a large model, perform correlation analysis on the target execution object based on the task knowledge base to determine whether there is a candidate execution object in the hierarchical scene graph; among them, the correlation between the candidate execution object and the target object in the task instruction is weaker than the correlation between the initial execution object and the target object.

[0108] Specifically, the object to be determined can be an object or location other than the initial object in the hierarchical scene diagram that is at the same level as the initial object. The object to be determined is determined based on the level of the initial object in the hierarchical scene diagram and the execution range of the task instruction. Then, based on the object to be determined, it is determined whether there is a candidate object in the hierarchical scene diagram.

[0109] Furthermore, using a large model, a correlation analysis is performed on each of the objects to be executed based on the task knowledge base to determine the temporal correlation between each object and the target object, in order to determine whether there are candidate objects for execution in the hierarchical scene graph. It is understandable that if the target object is not in the hierarchical scene graph, the initial execution object is obtained through correlation analysis using the large model based on the task knowledge base and the target object, while the correlation between the candidate object and the target object in the task instruction is weaker than the correlation between the initial execution object and the target object.

[0110] In some embodiments, if the target object is in the hierarchical scene graph, the initial execution object can be determined as the scene node in the hierarchical scene graph connected to the target object, thus obtaining the initial task plan. However, if the actual environment changes and the embodied robot fails to complete the task instruction after executing the task according to the initial task plan, it is necessary to determine candidate execution objects from the hierarchical scene graph through object filtering and association analysis to adjust the initial task plan. It is understood that in this embodiment, the association between the candidate execution object and the target object is unrelated to the association between the initial execution object and the target object.

[0111] Reference Figure 7 As shown, in one embodiment of this application, the method further includes:

[0112] S410. Based on the adjusted task plan, control the embodied robot to execute the sub-tasks in the adjusted task plan and provide feedback on the adjusted execution results.

[0113] S420. If the result of the adjustment indicates that the embodied robot has completed the adjustment task planning but still has not completed the task instruction, the above dynamic planning adjustment process is executed again to obtain a new adjustment task plan, which is used as the adjustment task plan to repeatedly control the embodied robot to execute the corresponding sub-tasks until the task instruction termination condition is met.

[0114] Specifically, the dynamic programming adjustment process may include: after identifying candidate execution objects, determining a corresponding third task tool from the preset task tools based on the target object, candidate execution objects, and task instructions to generate a third subtask for adjusting the initial task plan. Utilizing a large model and task knowledge base, the third subtask is parsed to determine a fourth task tool from the preset task tools, generating a fourth subtask for adjusting the initial task plan. After obtaining the third and fourth subtasks, the fourth subtask is associated with the third subtasks according to their execution order, resulting in the adjusted task plan. It can be understood that the adjusted task plan is obtained by replacing the first subtask in the initial task plan with the third subtask and replacing the second subtask in the initial task plan with the fourth subtask.

[0115] Furthermore, when repeating the above dynamic programming adjustment process, the correlation between the candidate execution object and the target object determined in each round is weaker than the correlation between the candidate execution object and the target object in the previous round, until the task instruction termination condition is met.

[0116] Furthermore, the task instruction termination condition may include a completion condition and an adjustment count condition. The completion condition is that the embodied robot completes the task instruction, and the adjustment count condition is that the number of times the dynamic programming adjustment process described above is repeated reaches a preset threshold. When either of the above task instruction termination conditions is met, the currently executing adjustment task planning ends, and the embodied robot returns to provide feedback on the task instruction result to the user. It should be noted that if no candidate execution object is found in the decision-making hierarchy scene graph in any round, the above dynamic programming adjustment process cannot be executed, the currently executing adjustment task planning ends, and the embodied robot returns to provide feedback on the task instruction result to the user.

[0117] Accordingly, please refer to Figure 8 This application provides a task planning system for an embodied robot, the system comprising:

[0118] The preliminary task planning module 100 is used to decouple and plan the task instructions of the embodied robot based on the pre-built hierarchical scene graph and task knowledge base, and obtain a preliminary task plan containing multiple sub-tasks. Among them, the sub-tasks correspond to the preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are the objects or positions that the embodied robot needs to reach during the execution of the task instructions.

[0119] The preliminary task execution module 200 is used to control the embodied robot to execute multiple sub-tasks according to the preliminary task plan, and to provide feedback on the preliminary execution results of the multiple sub-tasks.

[0120] The dynamic planning adjustment module 300 is used to determine whether there is a candidate execution object in the hierarchical scene graph when the initial execution result indicates that the embodied robot has completed the initial task planning but has not completed the task instruction; whereby the candidate execution object is an object or position that can replace the initial execution object.

[0121] The task planning and adjustment module 400 is used to dynamically plan and adjust the task plan based on the candidate execution objects when there are candidate execution objects in the hierarchical scene diagram, so as to control the embodied robot to complete the task instructions.

[0122] In some alternative implementations, the preliminary task planning module 100 includes:

[0123] The decoupling and identification unit is used to decouple the task instructions and identify the object using a large model, so as to obtain the target object and execution range in the task instructions; wherein, the execution range is the spatial range in which the embodied robot executes the task instructions.

[0124] The object filtering unit is used to filter objects in the hierarchical scene graph based on the target object, execution scope, and task knowledge base to obtain preliminary execution objects.

[0125] The task planning unit is used to plan tasks based on the target object and the initial execution object, resulting in multiple sub-tasks, and to formulate an initial task plan.

[0126] In some alternative implementations, the object filtering unit includes:

[0127] The hierarchy determination subunit is used to determine the target hierarchy of the target object in the hierarchy scene graph.

[0128] The candidate determination subunit is used to determine multiple candidate execution objects existing within the execution range at the level above the target level in the hierarchical scene graph.

[0129] The association analysis subunit is used to perform one-by-one association analysis between multiple candidate execution objects and the target object based on the task knowledge base, and to determine the preliminary execution object that meets the association conditions from multiple candidate execution objects.

[0130] In some optional implementations, the preliminary task planning module 100 further includes a scene graph construction unit, comprising:

[0131] The image acquisition subunit is used to acquire images of the scene where the embodied robot is located, and obtain multiple scene images.

[0132] The image description subunit is used to perform image description on multiple scene images using a large model to obtain the description results.

[0133] The hierarchical classification subunit is used to classify objects or positions in the scene where the embodied robot is located based on the task knowledge base and the description results, and to determine the scene level corresponding to each object or position.

[0134] The scene graph construction subunit is used to construct the scene graph structure based on the correlation between objects or positions and the corresponding scene hierarchy, thus obtaining a hierarchical scene graph.

[0135] In some alternative implementations, the embodied robot possesses a variety of preset task tools; the task planning unit includes:

[0136] The first subtask generation subunit is used to determine the corresponding first task tool from the preset task tools based on the target object, the initial execution object and the task instructions, so as to generate the first subtask.

[0137] The second subtask generation subunit is used to perform task parsing on the first subtask using the large model and task knowledge base, and to determine the second task tool from the preset task tools in order to generate the second subtask.

[0138] The subtask association subunit is used to associate the second subtask with the first subtask according to the execution order, so as to obtain the preliminary task plan.

[0139] In some alternative implementations, the dynamic programming adjustment module 300 includes:

[0140] The candidate determination unit is used to determine the execution object to be determined from the hierarchical scene diagram based on the level of the preliminary execution object in the hierarchical scene diagram and the execution scope of the task instruction.

[0141] The association analysis unit is used to perform association analysis on the target execution object based on the task knowledge base using the large model to determine whether there is a candidate execution object in the hierarchical scene graph; among them, the association between the candidate execution object and the target object in the task instruction is weaker than the association between the initial execution object and the target object.

[0142] In some alternative implementations, the system further includes a cyclic adjustment module, comprising:

[0143] The task execution unit is used to control the embodied robot to execute sub-tasks in the task plan and provide feedback on the execution results.

[0144] The cyclic adjustment unit is used to execute the above dynamic planning adjustment process again when the adjustment execution result indicates that the embodied robot has completed the adjustment task planning but still has not completed the task instruction, so as to obtain a new adjustment task plan, and repeatedly control the embodied robot to execute the corresponding sub-tasks until the task instruction termination condition is met.

[0145] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0146] In this embodiment, the embodied robot task planning system is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0147] Please see Figure 9 , Figure 9This is a schematic diagram of a controller provided in an embodiment of this application. As shown in the figure, the controller includes one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the controller, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple controllers can be connected, each device providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 9 Take a processor 10 as an example.

[0148] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0149] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0150] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the controller. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the controller via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0151] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0152] The controller also includes a communication interface 30 for communicating with other devices or communication networks.

[0153] This application also provides a android, including the controller described in the above embodiments, which controls the android using the method shown in the above embodiments.

[0154] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0155] This application provides a computer program product including computer instructions stored in a computer-readable storage medium. A controller's processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the controller to perform the method of any embodiment of this application.

[0156] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

[0157] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0158] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0159] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0164] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0165] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0166] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A task planning method for an embodied robot, characterized in that, The method includes: Based on a pre-built hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are decoupled and planned to obtain a preliminary task plan containing multiple sub-tasks; wherein, each sub-task corresponds to a preliminary execution object in the hierarchical scene graph, and the preliminary execution object is the object or position that the embodied robot needs to reach during the execution of the task instructions. According to the preliminary task plan, the robot is controlled to execute the multiple sub-tasks, and the preliminary execution results of the multiple sub-tasks are fed back. If the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction, it is determined whether there is a candidate execution object in the hierarchical scene graph; wherein, the candidate execution object is an object or position that can replace the preliminary execution object; If the candidate execution object exists in the hierarchical scene diagram, dynamic planning and adjustment are performed based on the candidate execution object to obtain an adjusted task plan, so as to control the embodied robot to complete the task instruction.

2. The method according to claim 1, characterized in that, Based on a pre-built hierarchical scene graph and task knowledge base, the task instructions of the embodied robot are decoupled and planned to obtain a preliminary task plan containing multiple sub-tasks, including: The task instructions are decoupled and object identified using a large model to obtain the target object and execution range in the task instructions; wherein, the execution range is the spatial range in which the embodied robot executes the task instructions; Based on the target object, the execution scope, and the task knowledge base, object filtering is performed in the hierarchical scene graph to obtain the preliminary execution object; Based on the target object and the initial execution object, task planning is performed to obtain the multiple sub-tasks, and the initial task plan is formulated.

3. The method according to claim 2, characterized in that, The step of filtering objects in the hierarchical scene graph based on the target object, the execution scope, and the task knowledge base to obtain the preliminary execution object includes: Determine the target level of the target object in the hierarchical scene graph; In the hierarchical scene diagram, at the level above the target level, determine multiple candidate execution objects existing within the execution range; Based on the task knowledge base, the multiple candidate execution objects are analyzed one by one with the target object to determine the preliminary execution object that meets the correlation conditions from the multiple candidate execution objects.

4. The method according to claim 2, characterized in that, The hierarchical scene diagram is obtained using the following method: Images of the scene where the embodied robot is located are captured to obtain multiple scene images; A large model is used to perform image description on the multiple scene images to obtain the description results; Based on the task knowledge base, the objects or positions in the scene where the embodied robot is located are classified hierarchically according to the description results, and the scene level corresponding to each object or position is determined. The scene graph structure is constructed based on the correlation between objects or locations and the corresponding scene hierarchy to obtain the hierarchical scene graph.

5. The method according to claim 2, characterized in that, The embodied robot possesses multiple preset task tools; the process of task planning based on the target object and the initial execution object to obtain the multiple sub-tasks and formulate the initial task plan includes: Based on the target object, the initial execution object, and the task instructions, a corresponding first task tool is determined from the preset task tools to generate a first subtask; The first subtask is parsed using the large model and the task knowledge base, and a second task tool is determined from the preset task tools to generate the second subtask. The second subtask is associated with the first subtask according to the execution order to obtain the preliminary task plan.

6. The method according to claim 1, characterized in that, The determination of whether there is a candidate execution object in the hierarchical scene graph includes: Based on the level of the preliminary execution object in the hierarchical scene diagram and the execution scope of the task instruction, determine the execution object to be determined from the hierarchical scene diagram; Using a large model, an association analysis is performed on the object to be executed based on the task knowledge base to determine whether the candidate object exists in the hierarchical scene graph; wherein, the association between the candidate object and the target object in the task instruction is weaker than the association between the initial object and the target object.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: According to the adjusted task plan, the robot is controlled to execute the sub-tasks in the adjusted task plan and the results of the adjusted execution are fed back. If the adjustment execution result indicates that the embodied robot has completed the adjustment task planning but still has not completed the task instruction, the above dynamic planning adjustment process is executed again to obtain a new adjustment task plan. This new adjustment task plan is then used to repeatedly control the embodied robot to execute the corresponding sub-tasks until the task instruction termination condition is met.

8. A task planning system for an embodied robot, characterized in that, The system includes: The preliminary task planning module is used to decouple and plan the task instructions of the embodied robot based on a pre-built hierarchical scene graph and task knowledge base, so as to obtain a preliminary task plan containing multiple sub-tasks; wherein, the sub-tasks correspond to preliminary execution objects in the hierarchical scene graph, and the preliminary execution objects are the objects or positions that the embodied robot needs to reach during the execution of the task instructions. The preliminary task execution module is used to control the embodied robot to execute the multiple sub-tasks according to the preliminary task plan, and to provide feedback on the preliminary execution results of the multiple sub-tasks; The dynamic planning adjustment module is used to determine whether there is a candidate execution object in the hierarchical scene graph when the preliminary execution result indicates that the embodied robot has completed the preliminary task planning but has not completed the task instruction; wherein, the candidate execution object is an object or position that can replace the preliminary execution object; The task planning and adjustment module is used to dynamically plan and adjust the task plan based on the candidate execution object when the candidate execution object exists in the hierarchical scene diagram, so as to control the embodied robot to complete the task instruction.

9. A controller, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A embodied robot, characterized in that, Includes the controller described in claim 9.

Images