Method, apparatus, device and computer program product for task planning

Abstract

The embodiment of the invention relates to a method, a device, equipment and a computer program product for task planning. The method includes generating a heterogeneous graph representation of a plurality of objects related to the task based on a current state of the task, where the heterogeneous graph representation includes object nodes indicating the plurality of objects and a first edge indicating a relationship between the plurality of objects in the current state. Further, the method includes generating task actions through a graph neural network based on the heterogeneous graph representation. Therefore, according to the embodiment of the invention, the task planning can be modeled through the heterogeneous graph data, and the task action of the task planning is generated by using the graph neural network, so that the manual design of a learning strategy is avoided, the complexity of the task planning is reduced, and the effect of the task planning can be improved.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computers, and more specifically, to methods, apparatus, devices, and computer program products for task planning. Background Technology

[0002] Task planning is the process of automatically generating a series of actions or decision sequences based on environmental information. It involves multiple steps, including environmental perception, goal setting, path planning, and task execution, aiming to achieve efficient and accurate automated operation. This technology is widely used in robotics, industrial automation, and autonomous driving.

[0003] With technological advancements, the importance of task planning is becoming increasingly apparent. It not only improves production efficiency and reduces labor costs, but also adapts to complex and ever-changing environments, enabling the execution of more refined and diverse tasks. Task planning is becoming a key technology driving progress across multiple industries, and as its level of intelligence and autonomy increases, it will play an even more crucial role in future society. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, device, computer program product, and medium for task planning.

[0005] According to a first aspect of this disclosure, a method for task planning is provided. The method includes generating a heterogeneous graph representation of multiple objects related to the task based on the current state of the task, wherein the heterogeneous graph representation includes object nodes indicating the multiple objects and first edges indicating relationships between the multiple objects in the current state. Furthermore, the method includes generating task actions based on the heterogeneous graph representation using a graph neural network.

[0006] According to a second aspect of this disclosure, an apparatus for task planning is provided. The apparatus includes a heterogeneous representation generation module configured to generate a heterogeneous graph representation of multiple objects related to the task, based on the current state of the task. The heterogeneous graph representation includes object nodes indicating the multiple objects and first edges indicating relationships between the multiple objects in the current state. Furthermore, the apparatus includes a task action generation module configured to generate task actions using a graph neural network based on the heterogeneous graph representation.

[0007] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the device to perform the steps of the method of the first aspect of this disclosure.

[0008] According to a fourth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions that, when executed, cause a computer to perform the steps of the method of the first aspect of this disclosure.

[0009] According to a fifth aspect of this disclosure, a machine-readable storage medium is provided. The machine-readable storage medium stores machine-executable instructions, wherein the machine-executable instructions are executed by a processor to implement the steps of the method of the first aspect of this disclosure.

[0010] The summary section is intended to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0011] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which the same reference numerals generally represent the same parts.

[0012] Figure 1 A schematic diagram of an example environment in which the apparatus and / or methods according to embodiments of the present disclosure may be implemented is shown;

[0013] Figure 2 A flowchart of a method for task planning according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A flowchart illustrating a task planning process according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A schematic diagram illustrating a process of generating scene graph data according to an embodiment of the present disclosure is shown;

[0016] Figure 5 A schematic diagram illustrating a process for converting scene graph data into heterogeneous graph data according to an embodiment of the present disclosure is shown.

[0017] Figure 6 A schematic diagram of the interface of a relational data annotation tool according to an embodiment of the present disclosure is shown;

[0018] Figure 7A A schematic diagram illustrating the process of performing edge prediction according to an embodiment of the present disclosure is shown;

[0019] Figure 7BA schematic diagram is shown illustrating updating a task heterogeneous graph after performing edge prediction according to an embodiment of the present disclosure;

[0020] Figure 8A A schematic diagram of a state diagram of a task according to an embodiment of the present disclosure is shown;

[0021] Figure 8B A schematic diagram illustrating a process for updating a task's state graph using a large language model according to an embodiment of the present disclosure is shown.

[0022] Figure 9 A schematic diagram illustrating the process of performing task planning according to an embodiment of the present disclosure is shown;

[0023] Figure 10 A schematic diagram of an apparatus for task planning according to embodiments of the present disclosure is shown; and

[0024] Figure 11 A schematic block diagram of an example device suitable for implementing embodiments of the present disclosure is shown.

[0025] In the various figures, the same or corresponding labels indicate the same or corresponding parts. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0028] As mentioned earlier, task planning technology is developing rapidly and playing an important role in multiple fields. The task planning process involves generating a series of task decisions or actions to guide the task from its initial state to the expected state. Currently, related technologies typically involve manually designing learning strategies based on task constraints. This not only results in a large workload and high complexity in task decision-making, but also leads to unsatisfactory task planning results.

[0029] To address this, embodiments of this disclosure propose a scheme for task planning that uses heterogeneous graph data to model task planning. First, a heterogeneous graph representation corresponding to the current state of the task is generated. Then, a graph neural network is used to process the heterogeneous graph representation to generate task decisions, thereby completing task planning.

[0030] Therefore, according to the embodiments of this disclosure, task planning can be modeled using heterogeneous graph data, and task actions for task planning can be generated using graph neural networks. This avoids the need for manually designing learning strategies, reduces the complexity of task planning, and improves the effectiveness of task planning.

[0031] The following is for reference Figures 1 to 11 The present disclosure is provided to illustrate the basic principles and several exemplary implementations. It should be understood that these exemplary embodiments are given only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the disclosure in any way.

[0032] Figure 1 An example environment 100 is shown, in which the devices and / or methods of embodiments of this disclosure may be implemented. (e.g.) Figure 1 As shown, the example environment 100 may include a computing device 110, which may be a user terminal, mobile device, computer, etc., or it may be a computing system, a single server, a distributed server, or a cloud-based server. The computing device 110 can generate a corresponding heterogeneous graph representation 130 based on the current state 120 of the task. The heterogeneous graph representation 130 may include multiple objects related to the task and the relationships between these objects in the current state. For example, task planning can be performed for a robot tableware recycling task to instruct the robot to place the tableware into the dishwasher sequentially. The task constraint is that the tableware on top must be placed first. The task state is updated each time tableware is placed in the dishwasher. It should be understood that this task is described as an example only, and the embodiments of this disclosure are not limited to this task. The current state 120 of the task may be text describing the positional relationships between the tableware, or an image of the current task scene, etc. Furthermore, the heterogeneous graph representation 130 may include multiple object nodes, each object node corresponding to a piece of tableware. The connections (also called edges) between object nodes represent the relationships between the tableware. It should be understood that the connections between object nodes and the edges between object nodes represent the same meaning.

[0033] In the description of some embodiments of this disclosure, for a robotic cutlery collection task, a node "P" represents a plate, a node "B" represents a bowl, a node "C" represents a cup, a node "D" represents a table, a node "DW" represents a dishwasher, and the connection between nodes "on" means "on top of," "in" means "among," and "nt" means "beside." For example, heterogeneous diagram representation 130 can indicate that in the current state of the robotic cutlery collection task, plate 131 is in dishwasher 133 (via connection 132), bowl 134 is on top of bowl 135, and so on.

[0034] Based on the heterogeneous graph representation 130, a graph neural network 140 can generate connections between object nodes, where each connection indicates the probability that the objects associated with that connection have a relationship in the current state. It should be understood that the generated connections described herein are connections that do not exist in the heterogeneous graph representation 130. For example, the generated connections may include dashed edges 151-155 in the heterogeneous graph representation 150, where dashed edge 151 may indicate the probability that a relationship exists between its associated plate 134 and dishwasher 133. Task actions (also known as task decisions) can then be generated based on these connections to complete task planning. For example, in a robotic dish recycling task, a task action 160, such as placing plate 134 into dishwasher 133, can be generated based on connection 151, and the task state can be updated by performing this task action.

[0035] Therefore, according to the embodiments of this disclosure, task planning can be modeled using heterogeneous graph data, and task actions for task planning can be generated using graph neural networks. This avoids the need for manually designing learning strategies, reduces the complexity of task planning, and improves the effectiveness of task planning.

[0036] It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to other environments with different structures and / or functionalities.

[0037] The following will combine Figures 2 to 11 The process according to embodiments of this disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0038] Figure 2A flowchart of a method 200 for task planning according to an embodiment of the present disclosure is shown. This method can be executed by a computing device, which may be a user terminal, mobile device, computer, etc., or it may be a computing system, a single server, a distributed server, or a cloud-based server. At block 202, a heterogeneous graph representation including multiple object nodes can be generated based on the current state of the task, wherein the heterogeneous graph representation indicates multiple objects related to the task and the relationships between the multiple objects in the current state. For example, in conjunction with... Figure 1 As described, computing device 110 can generate a heterogeneous graph representation 130 including multiple object nodes based on the current state 120 of a task, wherein the heterogeneous graph representation indicates multiple objects related to the task (e.g., plate 131, dishwasher 133, or bowl 134) and the relationships of the multiple objects in the current state (e.g., connection 132 indicates that plate 131 is in dishwasher 133).

[0039] At box 204, task actions can be generated using a graph neural network based on the heterogeneous graph representation. For example, combining... Figure 1 As described, computing device 110 can generate task actions 160 based on heterogeneous graph representation 130 via graph neural network 140. Thus, according to the method 200 of the embodiments of this disclosure, task planning can be modeled using heterogeneous graph data, and task actions for task planning can be generated using graph neural networks, avoiding the need for manually designing learning strategies, reducing the complexity of task planning, and improving the effectiveness of task planning.

[0040] Figure 3 A flowchart of a task planning process 300 according to an embodiment of the present disclosure is shown. This process can be performed by a computing device, which may be a user terminal, mobile device, computer, etc., or it may be a computing system, a single server, a distributed server, or a cloud-based server. The process 300 according to an embodiment of the present disclosure bridges the gap between scene graph representation and task planning, enabling efficient and automated task execution in complex environments. By utilizing NetworkX, PyTorch Geometric's Heterogeneous Graph Transformers (HGTs), and supervised learning methods, effective task planning is achieved based on heterogeneous graph data obtained from scene graphs, improving the adaptability and automation level of task planning in real-world scenarios.

[0041] like Figure 3As shown in box 302, scene graph data for the task is generated by graph modeling. This scene graph data is of the isomorphic graph type. A scene graph can describe objects in a scene and their spatial relationships (such as positional relationships, containment relationships, etc.). For example, in computer graphics, scene graphs can be used to construct and render graphical scenes, and can contain information such as object position, rotation, scaling, materials, and lighting. A scene graph is represented as a graph data structure, where nodes represent objects, and edges (i.e., connections between nodes) represent relationships between objects, such as spatial relationships and containment relationships. For example, task planning can be performed for a robot dish collection task to instruct the robot to place the dishes in a dishwasher. The task constraint is that the dishes placed on top must be processed first, and the task state is updated each time a dish is placed in the dishwasher. It should be understood that this task is described as an example only, and the embodiments of this disclosure are not limited to this task.

[0042] For example, in a robotic cutlery recycling task, graph data modeling can be performed on the cutlery and their relationships in the current state of the task to generate scene graph data for that task. In some embodiments, scene graph data can be generated based on node triples, where each triple includes two objects and a relationship. For example, graph processing tools (e.g., NetworkX) can be used to preprocess the triple data to generate scene graph data. In some embodiments, a large language model can be used to generate ordered triples of nodes, and then scene graph data can be generated based on these ordered triples. The following will combine... Figure 4 This describes the use of large language models to generate scene graph data.

[0043] Figure 4 A schematic diagram of a process 400 for generating scene graph data according to an embodiment of the present disclosure is shown. Figure 4 As shown, text 402 can be a descriptive text of the task scenario, and scene graph data 406 can be generated through the large language model 404. In some embodiments, the large language model parameters can be as follows: max_tokens = 1000, top_p = 1, temperature = 0, frequency_penalty = 0.0, presence_penalty = 0.0, stop = ["\n"]. In some embodiments, these parameters can be adjusted according to the specific task. For example, in some use cases, the parameter "max_tokens" can be adjusted to 2000.

[0044] In the process of generating scene graph data using a large model, prompts can be set. In some embodiments, the prompt can be set to: "Given a prompt, deduce as many relations as possible and provide a list of updates. If an update represents a relation, the format is [entity1, relation, entity2]. Relations are directional, so the order is important." With this prompt, the large language model can split the input text into subject-verb-object triples and store the results in a list. It should be understood that because relations are directional, the order of elements in the triples is important.

[0045] For example, if the prompt is "Place A into B", then the ordered triplet data generated by the large model is "[["A","in","B"]]", where "A" can be called the first object node and "B" can be called the second object node; if the prompt is "Place C next to D", then the ordered triplet data generated by the large model is "[["C","nt","D"]]"; if the prompt is "Put E on the F", then the ordered triplet data generated by the large model is "[["E","on","F"]]"; if the prompt is "A is on the B, B is on the C, D in the E", then the ordered triplet data generated by the large model is "[["A","on","B"],["B","on","C"],["D","in","E"]]".

[0046] Combination Figure 4 The text 402 can be described as "Plate 1 is on Plate 2, Bowl 2 is on Plate 1, Bowl 1 is on Bowl 2, Plate 3 is in the dishwasher, Cup 1 is next to Plate 1, and Cup 1 is also next to Plate 2." The generated ordered triplet data would then be "[["P1","on","P2"],["B2","on","P1"],["B1","on","B2"],["P3","in","DW"],["C1","nt","P1"],["C1","nt","P2"]]". This triplet data can then be preprocessed using graph processing tools (e.g., NetworkX) to generate scene graph data 406. Based on this process, the task scene can be described using natural language, and then a large language model can be used to generate triplet data, thus generating the task's scene graph data. This automates task scene modeling, avoids manual processing, and improves the efficiency and flexibility of scene modeling.

[0047] Return to reference Figure 3At box 304, scene graph data can be converted into heterogeneous graph data. Graph processing tools are flexible in handling graph-based data, but the generated scene graph data (i.e., homogeneous graph data) is not directly applicable to training graph neural network models. To achieve effective task planning, scene graph data needs to be converted into heterogeneous graph data. Heterogeneous graph data has multiple types of object nodes and multiple types of edges, which can better represent complex scenes and relationships, enhance the expressive power and applicability of the model, improve the training efficiency and prediction accuracy of graph neural network models, and help achieve more accurate and efficient task decisions. For example, in the task scenario of a robot collecting tableware, there are usually multiple types of objects, such as "plates," "bowls," "cups," "tables," and "dishwashers," and there are also multiple types of relationships, such as "above," "in," and "next to," etc. In contrast, homogeneous graphs usually have nodes and edges of the same type. The following will combine... Figure 5 Describe the process of transforming graph data.

[0048] Figure 5 A schematic diagram of a process 500 for converting scene graph data into heterogeneous graph data according to an embodiment of the present disclosure is shown. Figure 5 As shown, scene graph data 502 can be converted into heterogeneous graph data 504. During graph data conversion, the scene graph data can be traversed, saving the node type and initial node embedding of each object node, as well as the edge type and initial edge embedding of each edge. Then, heterogeneous graph data is generated based on the node type and initial node embedding of each object node and the edge type and initial edge embedding of each edge. For example, a node type mapping and a type index mapping can be created. The node type mapping maps nodes to node types, and the type index mapping tracks the index for each node type. Then, the object nodes of the scene graph data can be traversed, assigning a unique index to each node based on its node type (e.g., "bowl," "cup," "plate"). These indexes are stored in the node type mapping and categorized by node type in the "type index mapping." Next, a heterogeneous graph instance is created to traverse the scene graph data to obtain and store information on node type, edge type, and edge confidence. Then, a node type quantity mapping and an edge type quantity mapping can be created to track the number of nodes and edges for each node type, respectively.

[0049] Next, the nodes in the scene graph data can be traversed, generating feature tensors (initialized to zero tensors) for each node type. These feature tensors are stored in the heterogeneous graph instance. Then, an edge index map can be created to track the edge indices for each edge type. The edges in the scene graph data can then be traversed, generating triples for each edge, including the source node, target node, and edge type. These triples can be added to the edge index map. For each edge type, the corresponding list of edge indices is converted into tensors and stored in the heterogeneous graph instance. Furthermore, zero tensors can be initialized for each edge type as edge attributes. In this way, the scene graph data can be transformed into heterogeneous graph data, which includes information such as node type and initial node embeddings, edge type and initial edge embeddings, laying the foundation for training the graph neural network.

[0050] Return to reference Figure 3 At box 306, heterogeneous graph data can be used to train a graph neural network. In some embodiments, the graph neural network can be a heterogeneous graph transformer (HGT) model. For example, the heterogeneous graph transformer model can include four main parameters: "hidden_channels" (the dimension of the hidden layers), "out_channels" (the dimension of the output layers), "num_heads" (the number of multi-head attention), and "num_layers" (the number of transformer layers), and these parameters can be adjusted according to the size of the dataset. The node embeddings of the object nodes can be continuously adjusted and updated during the training of the graph neural network model, and the heterogeneous graph transformer model can output the adjusted node embeddings. In some embodiments, the heterogeneous graph transformer model can be implemented using the PyTorch Geometric library. PyTorchGeometric is a PyTorch-based library that supports writing and training graph neural networks and is suitable for processing irregularly structured data such as graphics, 3D meshes, and point clouds.

[0051] In some embodiments, supervised learning methods can be used to train the graph neural network. For example, the graph neural network can be trained by comparing the predicted output of the graph neural network with the ground truth in the heterogeneous graph training data using a loss function, i.e., a supervised learning method. In some embodiments, object nodes corresponding to baseline task actions can be assigned ground truth labels with high scores, where a high score can be above a threshold score. For example, in a robotic dish recycling task, for objects that can be placed in a dishwasher (i.e., objects corresponding to baseline task actions), the baseline ground truth label for that object can be set to a high value (e.g., 0.9) in the heterogeneous graph training data, while for objects that cannot be placed in a dishwasher, their baseline ground truth label can be set to a low value. In training the graph neural network, it is necessary to label the heterogeneous graph training data. Figure 6 A relational data annotation tool for labeling training data of heterogeneous graphs is demonstrated, which can be used to train graph neural networks using the training data.

[0052] Figure 6 A schematic diagram of an interface 600 for a relational data annotation tool according to an embodiment of the present disclosure is shown. Figure 6 As shown, the relationship preset option 602 lists common relationship types, including "In," "Left," "Next," "Occludes," "On," "Partof," "Right," and "Under." The relationship list 604 displays the currently labeled relationship; for example, "In (9->25)" indicates that object number 9 is inside object number 25. The image panel 606 displays the image to be labeled, such as the arrangement of dishes inside a dishwasher, with multiple lines of different colors, each representing a relationship from the relationship preset option 602. Furthermore, the brightness setting 608 and contrast setting 610 further enhance the clarity of the objects and their relationships in the image to be labeled. Thus, the relationship data labeling tool allows for efficient labeling of object relationships in complex scenes, generating a large amount of accurate scene graph data to train graph neural network models, thereby improving the training effect of the graph neural network models.

[0053] Return to reference Figure 3 At box 308, a graph neural network is used to predict edges in the heterogeneous graph. For example, the graph neural network can be used to obtain the node embedding of each object node. In some embodiments, the vector dot product can be computed on the node embeddings of two object nodes, and the vector dot product can be used as the probability that a second edge exists between the two object nodes to determine the second edge with the highest probability, where the second edge is the edge between each source node (e.g., tableware) and the target node (e.g., dishwasher). Figure 1 To describe, the action of calculating the vector dot product can be... Figure 1 The calculation will be performed using the computing device 110. The following will be combined with... Figure 7A This describes the process of edge prediction using graph neural networks. After each edge prediction operation, the task state changes, which will be discussed below. Figure 7B This describes the process of updating the heterogeneous graph representation of a task after performing edge prediction. In some embodiments of this disclosure, a task state graph can be used to represent the change process of the task state, where each node in the task state graph is the scene graph corresponding to that state. The following will combine... Figure 8AThe task's state graph is described. Furthermore, after each edge prediction, the task's state changes, and the task's context graph also changes. Large language models can be used to update the task's context and state graphs using natural language. The following will combine... Figure 8B Describe the process of updating the scene graph and state graph using a large language model.

[0054] Figure 7A A schematic diagram of a process 700A for performing edge prediction according to an embodiment of the present disclosure is shown. Figure 7A As shown, in the heterogeneous graph representation of the current state of the task, dashed lines represent edges to be predicted. For example, bowl 704 is on top of bowl 706, and bowl 704 has not yet been placed in dishwasher 708, therefore there is no connection between plate 704 and dishwasher 708. However, there is an edge 710 between plate 702 and dishwasher 708, indicating that plate is in dishwasher 708. When predicting edges, the node embedding of each object node can be obtained, and then the vector dot product of the node embeddings of two object nodes can be calculated. This vector dot product is then used as the probability that an edge exists between the two object nodes.

[0055] For example, the vector dot product of the embeddings of bowl 704 and dishwasher 708 can be calculated as the predicted probability (i.e., existence probability) of edge 712. Similarly, the predicted probabilities of edges 714, 717, 718, and 720 can be calculated. In some embodiments, the task decision (i.e., the task action to be performed) in the current state of the task can be determined based on the predicted probabilities of the edges. For example, the edge with the highest predicted probability can be selected from edges 712, 714, 717, 718, and 720, and the object node with the edge with the highest probability can be placed in dishwasher 708. For example, if edge 712 has the highest predicted probability, then the task decision can be "put bowl 704 in dishwasher 708". After this task decision is executed, the state of the task changes, and its heterogeneous graph is updated accordingly. Figure 7B A schematic diagram of a process 700B for updating a task heterogeneous graph after performing edge prediction, according to an embodiment of the present disclosure, is shown. Figure 7B As shown, when the command "Place bowl 704 into dishwasher 708" is executed, there is an edge 712 between bowl 704 and dishwasher 708 (from...). Figure 7A The dashed line becomes Figure 7B (The solid line indicates that bowl 704 is in dishwasher 708.)

[0056] Figure 8A A schematic diagram of a state diagram 800A of a task according to an embodiment of the present disclosure is shown. Figure 8AAs shown, the task scenario diagram corresponding to state node 802 (state 0) is 804. After performing the action of putting the bowl into the dishwasher, the task transitions from state node 802 to state node 806 (state 1), and the task scenario diagram corresponding to state node 806 is 808. It should be understood that as the task state changes, the task's state diagram will continue to be updated, from state 1 to state 2 to state N.

[0057] Figure 8B A schematic diagram of process 800B for updating the state graph of a task using a large language model according to an embodiment of the present disclosure is shown. Figure 8B As shown, text 810 (i.e., the natural language instruction) can be "Put the bowl in the dishwasher". First, text 810 can be converted into triples ["B", "in", "DW"] using a large language model. Then, a breadth-first search (BFS) can be performed starting from the object node corresponding to "bowl". In this way, all objects above "bowl" can be identified and stored in a list, and then a pop operation is performed on the list to remove these objects one by one. When all objects in the list have been removed, "bowl" can be moved directly to "dishwasher", thereby generating a scene graph to update the task's state graph.

[0058] The process of performing a breadth-first search can be described using the following example. For instance, suppose the initial scene graph is "(['A','on','B'],['B','on','C'],['C','on','D'],['E','in','F'],['D','on','desk'])", and the natural language instruction is "move C to F". Then, the breadth-first search is performed starting from C. First, the list of objects on C is identified as ['B','A']. Then, a pop operation is performed, moving A to the table (desk). The new scene graph is "(['A','on','desk'],['B','on','C'],['C','on','D'],['E','in','F'],['D','on','desk'])". Next, the pop-up operation continues, moving B to the table. The new scene graph is "(['A','on','desk'],['B','on','desk'],['C','on','D'],['E','in','F'],['D','on','desk'])", making the list on object C empty. Then, object C can be directly moved to object F, and the new scene graph is "(['A','on','desk'],['B','on','desk'],['C','in','F'],['E','in','F'],['D','on','desk'])". In this way, the task's scene graph and state graph can be easily updated using a large language model and breadth-first search.

[0059] Return to reference Figure 3 At box 310, task planning can be performed based on edge prediction. For example, as mentioned earlier, the task decision can be determined by selecting the edge with the highest probability based on edge prediction, and the task's scenario graph, heterogeneous graph, and state graph can be updated. Then, edge prediction can continue to be iteratively executed until the task ends. The following section combines... Figure 9 This describes the process of executing task planning.

[0060] Figure 9A schematic diagram of a task planning process 900 according to an embodiment of the present disclosure is shown. For example, the initial state of the task is state 902 (state 0), which corresponds to task scenario graph 904. As previously described, a graph neural network can be used to perform edge prediction based on task scenario graph 904 to generate a task decision (e.g., putting bowl 1 into the dishwasher). By executing the task decision, the task can be transitioned from state 902 to state 912 (state 1), which corresponds to task scenario graph 914. Similarly, a graph neural network can be used to perform edge prediction based on task scenario graph 914 to generate a task decision, transitioning the task state to state 922 (state 2), which corresponds to task scenario graph 924. Task planning ends when the task state reaches the final state 952. In some embodiments, a large model can be used to generate a reference scenario graph of the expected state, and then, after the task undergoes state transitions, the task scenario graph and the reference scenario graph are compared. Figure 1 When the task reaches its expected state, it can be determined that the task has been completed. For example, a user can specify that the final state of the task is "all objects except the table are put in the dishwasher." The large model can then generate a reference scene diagram of the expected state of the task using this instruction. Then, when the task scene diagram matches the reference scene diagram—that is, when the task state corresponds to the reference scene diagram—it can be determined that the task planning has reached its expected state.

[0061] Figure 10 A schematic diagram of an apparatus 1000 for task planning according to an embodiment of the present disclosure is shown. The apparatus 1000 includes a heterogeneous representation generation module 1002 configured to generate a heterogeneous graph representation of multiple objects related to the task based on the current state of the task, wherein the heterogeneous graph representation includes object nodes indicating the multiple objects and first edges indicating relationships between the multiple objects in the current state. Furthermore, the apparatus 1000 includes a task action generation module 1004 configured to generate task actions using a graph neural network based on the heterogeneous graph representation.

[0062] In some embodiments, the task action generation module 1004 includes: a second edge prediction module configured to predict a second edge between at least two object nodes in the heterogeneous graph representation using a graph neural network based on the heterogeneous graph representation, wherein the second edge indicates the possibility that there is a task action relationship between the objects represented by the at least two object nodes; and a task action determination module configured to determine the task action in the current state based on the prediction result.

[0063] In some embodiments, the apparatus 1000 further includes: a current state update module configured to update the current state of the task based on the generated task action; and a task state update module configured to repeatedly execute the steps of generating a task action based on the current state of the task, and updating the current state of the task based on the generated task action, until the task reaches the expected state.

[0064] In some embodiments, the heterogeneous representation generation module 1002 includes: a homogeneous representation generation module, which generates a homogeneous graph representation corresponding to the current state of the task, the homogeneous graph representation including object information of the plurality of objects and connection information between the plurality of objects; and a heterogeneous representation second generation module, configured to generate a heterogeneous graph representation including the plurality of object nodes based on the homogeneous graph representation, wherein the plurality of object nodes have multiple object node types.

[0065] In some embodiments, the heterogeneous representation second generation module includes: a node information storage module configured to store the node type and initial node embedding of each object node and the connection type and initial connection embedding of each connection by traversing the homogeneous graph representation; and a heterogeneous representation third generation module configured to generate the heterogeneous graph representation based on the node type and initial node embedding of each object node and the connection type and initial connection embedding of each connection.

[0066] In some embodiments, the isomorphic representation generation module includes: a state text acquisition module configured to acquire text corresponding to the current state of the task; a triple generation module configured to generate ordered triples based on the text using a language model, the ordered triples including a first object node, a connection of object nodes, and a second object node; and an isomorphic representation second generation module configured to generate the isomorphic graph representation corresponding to the current state based on the ordered triples.

[0067] In some embodiments, the apparatus 1000 further includes: a task state transition module configured to transition the task from the current state to the next state corresponding to the execution of the task action; and a state graph update module configured to update the state graph of the task based on the next state, wherein the state graph includes multiple state nodes and a task scenario graph corresponding to each state node, wherein each state node corresponds to a task action of the generated task, and the task scenario graph includes an isomorphic graph representation of the task at the corresponding state node.

[0068] In some embodiments, the state graph update module includes: an action triple generation module configured to generate ordered triples of the task action based on the action representation text corresponding to the task action using the language model; a next isomorphic graph generation module configured to generate an isomorphic graph representation of the next state based on the ordered triples of the task action and the isomorphic graph representation of the current state through breadth-first search; and a second state graph update module configured to update the state graph of the task using the isomorphic graph representation of the next state.

[0069] In some embodiments, the apparatus 1000 further includes: an expected state generation module configured to generate an expected isomorphic graph representation corresponding to the expected state based on text corresponding to the expected state of the task; and a task state determination module configured to determine that the task has reached the expected state in response to the latest state node in the state graph of the task corresponding to the expected isomorphic graph representation.

[0070] In some embodiments, the apparatus 1000 further includes a graph network training module configured to train the graph neural network using heterogeneous graph training data, wherein object nodes in the heterogeneous graph training data corresponding to baseline task actions are assigned ground truth labels with high scores.

[0071] In some embodiments, the apparatus 1000 further includes a training data annotation module configured to annotate connections between object nodes in the heterogeneous graph training data using an annotation tool, wherein the annotation tool is configured to annotate in a visual manner.

[0072] In some embodiments, the task action determination module includes: a node embedding generation module configured to generate node embeddings of object nodes in the heterogeneous graph representation using the graph neural network; a high-resolution connection determination module configured to determine the second edge with the highest probability by using the inner product of the node embeddings as the probability that there is a second edge between two object nodes; and a task action second determination module configured to generate the task action based on the second edge with the highest probability and the corresponding object node.

[0073] Figure 11A schematic block diagram of an example device 1100 suitable for implementing embodiments of the present disclosure is illustrated. As shown, device 1100 includes a processor 1101 that can perform various appropriate actions and processes according to computer program instructions loaded into random access memory (RAM) 1103 based on computer program instructions stored in read-only memory (ROM) 1102. Various programs and data required for operation of device 1100 may also be stored in RAM 1103. The processor 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0074] The various methods and processes described above can be executed by processor 1101. For example, in some embodiments, the various methods and processes described above can be implemented as computer software programs tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102. When the computer program is loaded into RAM 1103 and executed by processor 1101, one or more actions of the methods and processes described above can be performed.

[0075] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0076] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), and any suitable combination thereof. The computer-readable storage medium as used herein is not to be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0077] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0078] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0079] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0080] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0081] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0083] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method (200) for task planning, comprising: Based on the current state of the task, generate (202) a heterogeneous graph representation of multiple objects related to the task, wherein the heterogeneous graph representation includes object nodes indicating the multiple objects and a first edge indicating the relationship between the multiple objects in the current state; Based on the heterogeneous graph representation, task actions (204) are generated through a graph neural network.

2. The method (200) according to claim 1, wherein, Based on the heterogeneous graph representation, generating the task action through the graph neural network includes: Based on the heterogeneous graph representation, a second edge between at least two object nodes in the heterogeneous graph representation is predicted using a graph neural network, wherein the second edge indicates the possibility of a task action relationship between the objects represented by the at least two object nodes; and Based on the prediction results, the task action in the current state is determined.

3. The method (200) according to claim 1, further comprising: The current state of the task is updated based on the generated task action; Repeat the steps of generating task actions based on the current state of the task, and updating the current state of the task based on the generated task actions, until the task reaches the expected state.

4. The method (200) of claim 1, wherein generating the heterogeneous graph representation of the plurality of objects related to the task comprises: Based on the current state of the task, generate an isomorphic graph representation corresponding to the current state, wherein the isomorphic graph representation includes object information of the plurality of objects and connection information between the plurality of objects; as well as Based on the homogeneous graph representation, a heterogeneous graph representation including the plurality of object nodes is generated, wherein the plurality of object nodes have multiple object node types.

5. The method (200) according to claim 4, wherein generating the heterogeneous graph representation including the plurality of object nodes comprises: By traversing the isomorphic graph representation, the node type and initial node embedding of each object node, as well as the connection type and initial connection embedding of each connection, are saved. The heterogeneous graph representation is generated based on the node type and initial node embedding of each object node, and the connection type and initial connection embedding of each connection.

6. The method (200) of claim 4, wherein generating the isomorphic graph representation comprises: Obtain the text corresponding to the current state of the task; Based on the text, an ordered triplet is generated using a language model. The ordered triplet includes a first object node, a connection of object nodes, and a second object node. as well as Based on the ordered triples, the isomorphic graph representation corresponding to the current state is generated.

7. The method (200) according to claim 6, further comprising: Transition the task from its current state to the next state corresponding to the execution of the task action; Based on the next state, the state graph of the task is updated. The state graph includes multiple state nodes and a task scenario graph corresponding to each state node. Each state node corresponds to a task action of the generated task. The task scenario graph includes an isomorphic graph representation of the task in the corresponding state node.

8. The method (200) of claim 7, wherein updating the state graph of the task comprises: Based on the action representation text corresponding to the task action, the language model is used to generate ordered triples of the task action; Based on the ordered triples of the task actions and the isomorphic graph representation of the current state, the isomorphic graph representation of the next state is generated by breadth-first search. as well as The state graph of the task is updated using the isomorphic graph representation of the next state.

9. The method (200) according to claim 7, further comprising: Based on the text corresponding to the expected state of the task, generate an expected isomorphic graph representation corresponding to the expected state; as well as In response to the latest state node in the state graph of the task corresponding to the expected isomorphic graph representation, it is determined that the task has reached the expected state.

10. The method (200) according to claim 1, further comprising: The graph neural network is trained using heterogeneous graph training data, wherein object nodes in the heterogeneous graph training data corresponding to baseline task actions are assigned ground truth labels with high scores.

11. The method (200) according to claim 10, further comprising: The connection between object nodes in the heterogeneous graph training data is labeled using a labeling tool configured to perform the labeling in a visual manner.

12. The method (200) according to claim 2, wherein generating the task action comprises: The graph neural network is used to generate node embeddings of object nodes in the heterogeneous graph representation; The second edge with the highest probability is determined by using the inner product of the vectors embedded in the nodes as the probability that a second edge exists between two object nodes; as well as The task action is generated based on the second edge with the highest probability and the corresponding object node.

13. An apparatus for task planning, comprising: The heterogeneous representation generation module is configured to generate a heterogeneous graph representation of multiple objects related to the task based on the current state of the task, wherein the heterogeneous graph representation includes object nodes indicating the multiple objects and a first edge indicating the relationship between the multiple objects in the current state; as well as The task action generation module is configured to generate task actions based on the heterogeneous graph representation using a graph neural network.

14. An electronic device, comprising: At least one processor; as well as A memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the device to perform the method according to any one of claims 1 to 12.

15. A computer program product tangibly stored on a non-transient computer-readable medium and comprising machine-executable instructions for performing the method according to any one of claims 1 to 12.

Images