Method and system for generating a robot task plan

Abstract

A computer-implemented method and system is disclosed comprising receiving, by a large language model (LLM), a free-form (FF) natural language (NL) instruction in order to generate a task plan for a robot operating in a hierarchical multi-region environment. The method and system further comprises receiving, by the LLM, a hierarchical graph data structure characterising the hierarchical multi-region environment and associated items and then processing, by the LLM, the FF NL instruction and the hierarchical graph data structure to generate a subgraph of the hierarchical graph data structure relevant to the task. In one aspect, an actionable task plan for the robot is generated based on the subgraph.

Description

METHOD AND SYSTEM FOR GENERATING A ROBOT TASK PLANPRIORITY DOCUMENTS

[0001] The present application claims priority from Australian Provisional Patent Application No. 2023904229 titled “METHOD AND SYSTEM FOR GENERATING A ROBOT TASK PLAN” and filed on 22 December 2023, the content of which is incorporated by reference in its entirety.INCORPORATION BY REFERENCE

[0002] The following publication is referred to in the present application and its contents are incorporated by reference in their entirety:• “SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning”, Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, Niko Suenderhauf, 7th Annual Conference on Robot Learning, 2023, Atlanta, USA (see also arXiv:2307.06135 [cs.RO])TECHNICAL FIELD

[0003] The present disclosure relates to the generating of a task plan for a robot. In a particular form, the present disclosure relates to generating a task plan for a robot operating in a multi-region environment.BACKGROUND

[0004] While robots may be preprogrammed to carry out a predefined task in accordance with a task plan that defines a sequence of robot actions, in circumstances where the original task is expressed as a free form (FF) natural language (NL) instruction, a task plan comprising a sequence of robot operating commands or instructions will need to be generated based on the NL instruction that is implementable by the robot.

[0005] For some simple instructions involving a single region (eg, a room), the generation of a task plan may in principle be feasible based on pre-encoded information describing exhaustively the items that may be present in the region. However, the generation of a task plan becomes increasingly more challenging when the robot is tasked to operate in a multi-region environment (eg, a multi-room environment or even a multi-room / multi-level environment) where in such environments the number of items present may grow exponentially and describing the scene exhaustively can be both memory and / or computationally intensive.SUMMARY

[0006] In one aspect, the present disclosure provides a method, implemented by one or more processors, comprising: receiving, by a large language model (LLM), a free-form (FF) natural language (NL) instruction for a robot to carry out a task in a hierarchical multi -region environment; receiving, by the LLM, a hierarchical graph data structure characterising the hierarchical multiregion environment and associated items; processing, by the LLM, the FF NL instruction and the hierarchical graph data structure to generate a subgraph of the hierarchical graph data structure relevant to the task.

[0007] In another form, processing, by the LLM, the FF NL instruction and the hierarchical graph data structure to generate a subgraph of the hierarchical graph data structure relevant to the task comprises: generating, by the LLM, from the FF NL instruction and the hierarchical graph data structure one or more graph manipulation commands to identify at least one region in the hierarchical graph data structure relevant to the task; processing, by the LLM, the hierarchical graph data structure based on the one or more graph manipulation commands and the FF NL instruction to determine the subgraph comprising the selection of regions relevant to the task.

[0008] In another form, processing, by the LLM, the hierarchical graph data structure based on the one or more graph manipulation commands and the FF NL instruction to determine the subgraph comprises: providing a collapsed or contracted hierarchical graph data structure; and determining the subgraph by manipulating the contracted hierarchical graph data structure in accordance with the FF NL instruction and the one or more graph manipulation commands to determine relevant nodes of the hierarchical graph data structure for the task.

[0009] In another form, processing, by the LLM, the hierarchical graph data structure based on the one or more graph manipulation commands and the FF NL instruction to determine the subgraph comprises: providing to the LLM first level nodes from the hierarchical graph data structure corresponding to regions of the first level of the hierarchical multi-region environment; identifying, by the LLM, relevant first level nodes from the first level of nodes based on the FF NL instruction and the one or more graph manipulation commands; identifying, by the LLM, relevant successive level nodes from any successive levels of nodes of the hierarchical graph data structure corresponding to regions of the successive levels of the hierarchical multi-region environment based on the FF NL instruction and the one or more graph manipulation commands, wherein the relevant successive level nodes depend initially from the identified relevant first level nodes and in turn from identified relevant successive level nodes identified from a node level above;determining the subgraph to be the identified relevant first level nodes and relevant successive level nodes.

[0010] In another form, identifying, by the LLM, relevant first level nodes based on the FF NL instruction and the one or more graph manipulation commands comprises: identifying, by the LLM, one or more candidate first level nodes based on the FF NL instruction; determining, by the LLM, the one or more graph manipulation commands to comprise one or more commands to expand the one or more candidate first level nodes to reveal a next level of nodes; identifying, by the LLM, for each of the one or more expanded candidate first level nodes whether any of the next level of nodes includes one or more relevant next level nodes based on the FF NL instruction; assigning, by the LLM, a candidate first level node to be a relevant first level node on identification of one or more relevant next level nodes for the candidate first level node.

[0011] In another form the method further comprises: determining, by the LLM, the one or more graph manipulation commands comprise one or more commands to contract any candidate first level nodes on a failure to identify relevant next level nodes.

[0012] In another form, identifying, by the LLM, relevant successive level nodes based on the FF NL instruction and the one or more graph manipulation commands, comprises: determining, by the LLM, the one or more graph manipulation commands comprise one or more commands to expand the one or more candidate successive level nodes to reveal a next level of nodes; identifying, by the LLM, for each of the one or more expanded candidate successive level nodes whether any of the next level of nodes includes one or more relevant next level nodes based on the FF NL instruction; assigning, by the LLM, a candidate successive level node to be a relevant successive level node on identification of one or more relevant next level nodes for the candidate successive level node.

[0013] In another form, the method further comprises: determining, by the LLM, the one or more graph manipulation commands comprise one or more commands to contract any candidate successive level node on failure to identify relevant next level nodes.

[0014] In another form, one or more successive level nodes comprises items associated with regions of the hierarchical multi-region environment.

[0015] In another form, processing, by the LLM, the FF NL instruction and the hierarchical graph data structure to generate a subgraph of the hierarchical graph data structure relevant to the task comprises initially prompting the LLM to divide the processing into intermediate steps.

[0016] In another form, the one or more graph manipulation commands are processable by the LLM.

[0017] In another form, the one or more graph manipulation commands comprises a command for the robot to explore a region of the hierarchical multi-region environment corresponding to an unpopulated portion of the hierarchical graph data structure to identify items located in the region.

[0018] In another form, the hierarchical graph data structure is further populated based on items identified by the robot from exploring the region.

[0019] In another form, the method further comprises: processing, by the LLM, the FF NL instruction and the subgraph to generate a task plan comprising a sequence of actions for the robot to carry out the task.

[0020] In another form, processing, by the LLM, the FF NL instruction and the subgraph to determine a task plan comprising a sequence of actions for the robot to carry out the task comprises: generating, by the LLM, an initial sequence of actions comprising robot locations and manipulation of any items identified in the subgraph to carry out the task; processing the initial sequence of actions by a path planner to supplement the initial sequence of actions with robot navigational instructions to generate the sequence of actions for the robot to carry out the task.

[0021] In another form, the method further comprises: simulating the sequence of actions to determine if the sequence of actions is actionable by the robot and provide feedback indicating conflicting actions; on identification of any conflicting actions in the sequence of actions generating, by the LLM, based on the conflicting actions a modified initial sequence of actions comprising robot locations and manipulation of items identified in the subgraph to carry out the task; and processing the modified initial sequence of actions by the path planner to supplement the modified initial sequence of actions with robot navigational instructions to update the sequence of actions for the robot to carry out the task.

[0022] In another form, the hierarchical multi-region environment comprises regions defined to be in at least two levels of a hierarchy characterising the hierarchical multi-region environment.

[0023] In another form, the hierarchical graph data structure is a scene graph.

[0024] In a second aspect, the present disclosure provides a system, comprising memory storing instructions and one or more processors configmed or operable to execute the instructions to perform the method of the first aspect.

[0025] In a third aspect, the present disclosure provides a robot, comprising: one or more actuators; one or more end effectors; one or more processors operable or configured to carry out the task plan of the first aspect.BRIEF DESCRIPTION OF DRAWINGS

[0026] Embodiments of the present disclosure will be discussed with reference to the accompanying drawings wherein:

[0027] FIG. 1 is a flow diagram of an example method for generating a subgraph in accordance with some embodiments;

[0028] FIGS. 2A-2C are overview diagrams of example hierarchical multi-region environments in accordance with some embodiments;

[0029] FIG. 3 is an overview diagram of a hierarchical graph tree structure and the corresponding hierarchical multi-region environment and associated items in accordance with some embodiments;

[0030] FIG. 4 is a flow diagram of an example method for generating a subgraph of the hierarchical graph data structure relevant to the robot task in accordance with some embodiments;

[0031] FIG. 5A is a flow diagram of an example method of a large language model (LLM) processing a hierarchical graph data structure based on one or more graph manipulation commands and the FF NL instruction to determine a subgraph in accordance with some embodiments;

[0032] FIG. 5B is a flow diagram of another example method of a large language model (LLM) processing a hierarchical graph data structure based on one or more graph manipulation commands and the FF NL instruction to determine a subgraph in accordance with some embodiments;

[0033] FIG. 6 is a flow diagram of an example method for an LLM to identify relevant first level nodes based on the FF NL instruction and the one or more graph manipulation commands in accordance with some embodiments;

[0034] FIG. 7 is a flow diagram of an example method for generating a task plan for a robot in accordance with some embodiments;

[0035] FIG. 8 is a flow diagram of another example method for generating a task plan for a robot in accordance with some embodiments;

[0036] FIG. 9 is a flow diagram of an example method for verifying and updating a task plan for a robot in accordance with some embodiments;

[0037] FIG. 10 is figurative overview process flow diagram of methods for generating a subgraph and task plan for a robot in accordance with some embodiments;

[0038] FIG. 11 is an overview diagram of a hierarchical graph tree structure and the corresponding hierarchical multi-region environment and associated items in accordance with some embodiments;

[0039] FIG. 12 shows the hierarchical multi-region environment and associated items shown in FIG. 11 and a collapsed version of the hierarchical graph tree structure in accordance with some embodiments;

[0040] FIG. 13 is an architecture overview diagram of a robot; and

[0041] FIG. 14 is an architecture overview diagram of a computer system.

[0042] In the following description, like reference characters designate like or corresponding parts throughout the figures.DESCRIPTION OF EMBODIMENTS

[0043] Techniques are disclosed for generating a task plan for a robot to carry out a task in a hierarchical multi-region environment based on a free form (FF) natural language (NL) instruction to carry out the task. The disclosed techniques are particularly beneficial, as they allow for a memory and computationally efficient strategy for the grounding of the robot in multi -region environments where the task plan is based on an understanding of the location of relevant items of interest relevant to the task and the topological arrangement of the different regions in the hierarchical multi-region environment.

[0044] Referring now to FIG. 1, there is shown a flow diagram of a computer-implemented method 100 (ie, a method implemented by one or more processors) for generating a subgraph according to some embodiments of the present disclosure.

[0045] By way of overview, method 100 comprises (at block 110) receiving, by a large language model (LLM), a FF NL instruction 105 for a robot to carry out a task in a hierarchical multilevel multi-region environment and further receiving (at block 120), by the LLM, a hierarchical graph data structure 125 characterising the hierarchical multi-region environment and any associated items present in the environment. Method 100 concludes with the LLM (at block 130) processing the FF NL instruction 105 and the hierarchical graph data structure 125 to determine or generate a subgraph 135 of the hierarchical graph data structure 125 relevant to the task.

[0046] By way of background, language models (LMs) have been developed that can be used to process natural language (NL) content and / or other input(s), to generate LM output that that reflects NL content and / or other content that is responsive to the input(s). For example, large language models (LLMs) have been developed that are trained on massive amounts of data and are able to be utilised to robustly process a wide range of NL inputs and generate corresponding LM output that reflects corresponding NL content that is accurate and responsive to the NL input. An LLM can include at least hundreds of millions of parameters and can often include at least billions of parameters, such as one hundred billion or more parameters. An LLM can, for example, be a sequence-to-sequence model, transformer-based, and / or include an encoder and / or a decoder. One non-limiting example of an LLM is OpenAFs GPT-4. Another non-limiting example of an LLM is GOOGLE'S Language Model for Dialogue Applications (LaMDA).

[0047] Throughout this specification, the term “hierarchical multi-region environment” when describing an environment is taken to mean that the environment comprises multiple regions that are organised in a hierarchy comprising one or more levels.

[0048] Referring now to FIGS. 2A-2C, there are shown overview diagrams of example hierarchical multi-region environments in accordance with some embodiments.

[0049] In one non-limiting example, FIG. 2A shows a hierarchical multi-region environment 210 comprising a “first” level (not necessarily a physical level) comprising regions defined as different “rooms” (eg, bedroom, bathroom, laundry, etc) In this example, the hierarchical multi-region environment comprises a single level of regions.

[0050] In another non-limiting example, FIG. 2B shows a hierarchical multi-region environment 230 comprising a “first” level comprising regions defined as different “floors” of a building (eg, first floor, second floor, etc) and a second level comprising regions defined as rooms for each of the floors of the first level (eg, first floor - reception, first floor - closet, etc, second floor - first office, second floor - restroom, etc).

[0051] In another non-limiting example, FIG. 2C shows a hierarchical multi-region environment 250 that comprises a first level comprising regions that are geographically distinct locations (eg, Queensland University of Technology, University of Queensland, etc), a second level comprising regions defined as buildings of the geographically distinct locations of the first level (eg, QUT - Computer Science Building, QUT - Gym, etc, UQ - Library, UQ - Physics Building, etc) a third level comprising rooms for each of the buildings of the second level (eg, QUT - Computer Science Building - Reception, QUT - Computer Science Building - Computer Lab, etc, QUT - Gym - Weight Room, QUT - Gym - Pool, etc, UQ - Library - Quiet Reading Area, UQ - Library - Book Return, etc, UQ - Physics Building - Workshop, UQ - Physics Building - Student Area, etc).

[0052] As would be appreciated, a hierarchical multi-region environment in accordance with the present disclosure may be adopted to describe any multi-region environment which may be organised in a hierarchy of one or more levels. Another non-limiting example may include Level 1- Countries, Level 2 - States, Level 3 - Counties, etc, for a robot tasked to carry out a task between different countries (eg, an airborne robot). A further non-limiting example may include a first level comprising a university (eg, QUT, UQ) a second level comprising university campuses at a university (eg, Kelvin Grove, Gardens Point, etc.) a third level comprising buildings on a campus (eg, A block, B block) a fourth level comprising floors within a building (eg, first floor, second floor), a fifth level comprising rooms on a floor (eg, kitchen, office 1, office 2 etc) for a robot tasked to carry out a task between different universities and university campuses.

[0053] Referring back to FIG. 1, at block 110, a FF NL instruction 105 is received by the LLM for processing, where the instruction 105 is for a robot to carry out a task in a hierarchical multi-region environment.

[0054] As would be appreciated, the FF NL instruction 105 may be in the form of any suitable input processable by the LLM including, but not limited to, direct text input (eg, an operator typing an instruction into a suitable interface), uploading or being sent a file comprising the FF NL instruction, a voice instruction processed by a suitable speech recognition processor to generate equivalent text input, and / or a hand written instruction again processed by a suitable handwriting or optical character recognition processor.

[0055] At block 120, a hierarchical graph data structure 125 characterising the hierarchical multi-region environment and associated items is received by the LLM. The hierarchical graph data structure 125 characterises the hierarchical multi-region environment and associated items with the nodes and edges of the graph data structure defining the semantic data and semantic relationships of the regions and items of the hierarchical multi-region environment. Additionally, the levels of the hierarchical graph data structure also have semantic meaning in the context of the hierarchical multi-region environment (eg, levels of thehierarchical graph data structure relating to buildings, floors in a building, rooms on a floor, items in a room, etc of the hierarchical graph data structure).

[0056] In one example, the hierarchical graph data structure is defined to be a hierarchical graph Q = (F, E) in which the set of vertices V comprises 1^ U V2U ■■■uVK> with each Vksignifying the set of vertices at a particular level k of the hierarchy. In this formalism, edges stemming from a vertex v Vkmay only terminate in Vk-U VkU Vk+1, i.e. edges connect vertices (referred to as nodes throughout the specification) within the same level, or one level higher or lower. As referred to above, the hierarchical graph data structure not only characterises the hierarchical multi-region environment but also the items associated with the various multi-region environment. In various example, these items may be represented in a “level” or “levels” of nodes of the multi-region environment.

[0057] Referring now to FIG. 3, there is shown an overview diagram of a hierarchical graph data structure 300 and the corresponding hierarchical multi-region environment and associated items 350 in accordance with some embodiments. In this example, hierarchical graph tree structure 300 (equivalent to hierarchical multi-region environment 350) comprises two levels of nodes corresponding to physical levels and comprising a “floor” level (level 1) and a “room” level (level 2) and further the associated items form two additional levels of nodes being an “asset” level (level 3) directed to immovable items and “object” level (level 4) directed to movable items located in a room. In various examples, nodes representing the associated items (eg, assets and objects) encode attributes associated with the items including, but not limited to, state, affordances, colour, weight and 3D pose. In various examples, nodes representing the various regions in the various levels may encode attributes of the particular region (eg, size, GPS coordinates, environment, etc).

[0058] In various examples, the links between the regions are characterised in the hierarchical graph data structure by pose nodes to represent the hierarchical multi-region environment’s topological structure. In other examples, the links between regions may include semantic information relating the regions such as the regions being accessible with respect to each other by stairs, or a ramp or an elevator to further assist the task planning process.

[0059] In one example, the hierarchical graph tree structure may be represented by a scene graph. As would be appreciated, a scene graph is a general data structure commonly used by vector-based graphics editing applications and modern computer games which is configmed to arrange the logical and often spatial representation of a graphical scene in the form of a collection of nodes in a graph or tree structure. In one example, the scene graph incorporates action and state information for the regions and any associated items which provides useful information for task planning.

[0060] In one example, the scene graph may be represented as 3D scene graph (3DSG) (eg, see Armeni et al., “3D scene graph: A structure for unified semantics, 3D space, and camera.”, in Proceedings of the IEEE / CVF international conference on computer vision, pages 5664-5673, 2019). In one example, the 3DSG may be represented as a NetworkX Graph object that may be text-serialised into a JSON data format that may parsed directly by a pre-trained LLM.

[0061] Referring back to FIG. 3, an example of a single asset node from the hierarchical graph data structure 300 based on a 3DSG representation or characterisation of the hierarchical multi-region environment 350 may be represented as {name: coffee ma chine, type: asset, location: kitchen, affordances: [turn on, turn off, release], state: off, attributes: [red, automatic], position: [2.34, 0.45, 2.23]} with an example edge between nodes captured as {kitchen<->coffee_machine}.

[0062] Referring back to FIG. 1, at block 130, the FF NL instruction 105 and the hierarchical graph data structure 125 are processed by the LLM to generate a subgraph 135 of the scene graph relevant to the task. In one example, the subgraph 135 identifies the regions in the hierarchical multi-region environment that are relevant to the task.

[0063] As would be appreciated, the generation of this subgraph will greatly simplify any task planning for a robot as the original multi-region environment, which likely will comprise numerous potential regions, will have been collapsed or compressed or compacted to a subgraph comprising those regions relevant to the task. This will reduce any memory and / or computational burden associated with the task planning process as any task planning process can focus on the subgraph of interest to the task. Additionally, in the case where an LLM may be involved in the task planning process, basing this planning around the subgraph instead of the full scene graph corresponding to the full hierarchical data structure will assist in preventing the LLM from hallucinating and producing infeasible action sequences.

[0064] Referring now to FIG. 4, there is shown a flow diagram of an example method 400 for generating a subgraph relevant to the robot task in accordance with some embodiments. In various examples, block 130 of FIG. 1 may be implemented in accordance with method 400 of FIG. 4

[0065] At block 410, one or more graph manipulation commands are generated by the LLM from the FF NL instruction 105 and the hierarchical graph data structure 125 to identify at least one region in the hierarchical graph data structure 125 relevant to the task. As would be appreciated, an LLM may be configured to process the hierarchical graph data structure 125 to identify potential regions from the regions available in the hierarchical multi-region environment and based on the LLM’s understanding of the FF NL instruction 105 then generate a graph manipulation command to manipulate the hierarchical graph data structure 125 to determine a subgraph 135 comprising the selection of regions relevant to thetask. In one example, the graph manipulation command is in turn processable by the LLM (eg, by calling an external application programming interface (API) to carry out the graph manipulation command)

[0066] At block 420, the hierarchical graph data structure 125 is then processed based on the one or more graph manipulation commands to generate the selection of regions or subgraph 135 relevant to the task.

[0067] Referring now to FIG. 5A there is shown a flow diagram of an example method 500 of an LLM processing a hierarchical graph data structure based on one or more graph manipulation commands and the FF NL instruction to determine a subgraph in accordance with some embodiments. In various examples, block 420 of FIG. 4 may be implemented in accordance with method 500 of FIG. 5 A.

[0068] At block 510, method 500 comprises providing a collapsed or contracted hierarchical graph data structure. In one example, the collapsed hierarchical graph data structure may be collapsed to the top or first level. In another example, the collapsed hierarchical graph data structure may be collapsed to the first two levels. In another example, the collapsed hierarchical graph data structure may only be partially collapsed with respect to nodes that have been determined to not be relevant by an initial filtering process.

[0069] At block 520, the subgraph is determined by manipulating the contracted hierarchical graph data structure in accordance with the FF NL instruction and the one or more graph manipulation commands to determine relevant nodes for the task.

[0070] Referring now to FIG. 5B, there is shown a flow diagram of another example method 550 of an LLM processing a hierarchical graph data structure based on one or more graph manipulation commands and the FF NL instruction to determine a subgraph in accordance with some embodiments. Shown on the left side of the flow diagram is a representation of the nodes being processed in accordance with each block of method 550. In various examples, block 420 of FIG. 4 may be implemented in accordance with method 550 of FIG. 5B.

[0071] At block 560 of method 550, first level nodes are provided to the LLM from the hierarchical graph data structure where these first level nodes correspond to regions of the first level of the hierarchical multi-region environment in which the robot is tasked to operate. As is apparent, the first level nodes correspond to a collapsed or contracted form of the hierarchical graph data structure.

[0072] At block 570 of method 550, relevant first level nodes are identified by the LLM from the first level of nodes based on the FF NL instruction and the one or more graph manipulation commands. In this context “relevant” means that the identified nodes are related to regions that are relevant to the robot task.As shown in FIG. 5B, in this example two first level nodes have been identified as relevant first level nodes (ie, the shaded nodes in level “1”).

[0073] At block 580, relevant successive level nodes from any successive levels of nodes of the hierarchical graph data structure corresponding to regions of the successive levels of the hierarchical multi-region environment (if any) are identified again based on the FF NL instruction and the one or more graph manipulation commands. The relevant successive level nodes are defined to depend initially from the identified relevant first level nodes and in turn from identified relevant successive level nodes identified from a node level above. As shown in FIG. 5B, two first level nodes are identified as relevant nodes and in the successive level (ie, level “2”) three second level nodes are identified as relevant successive level nodes for the second level and in the next successive level (ie, level “3”) three third level nodes are identified as relevant successive level nodes for the third level. In each case, a relevant successive level node depends initially from a relevant first level node and in turn from identified relevant successive level nodes identified from a node level above.

[0074] As would be appreciated, the process at block 580 may be applied to any hierarchical multiregion environment where the regions are arranged over a finite number of levels.

[0075] Finally, at block 590 of method 550, the subgraph is determined to be the identified relevant first level nodes and relevant successive level nodes. As can be seen from FIG. 5B, the subgraph will be a reduced selection of nodes corresponding to a reduced selection of regions of the original full hierarchical multi-region environment.

[0076] Referring now to FIG. 6, there is shown a flow diagram of an example method 600 for an LLM to identify relevant first level nodes based on the FF NL instruction and the one or more graph manipulation commands in accordance with some embodiments. Shown on the left side of the flow diagram is a representation of the nodes being processed in accordance with each block of method 600. In various examples, block 570 of FIG. 5B may be in implemented in accordance with method 600 of FIG. 6.

[0077] At block 610, method 600 comprises identifying one or more candidate first level nodes 611, 612based on the FF NL instruction. In various examples, the LLM will identify the candidate first level nodes 611, 612 based on an attribute of the node (eg, region name, location, etc) and its relevance to the original FF NL instruction.

[0078] At block 620, and noting that only the first level nodes from the hierarchical graph data structure have been provided, the one or more graph manipulation commands are determined by the LLM to comprise one or more commands to expand the one or more candidate first level nodes 611, 612 to reveal a next level of nodes. As can be seen in FIG. 6, two nodes from the first level have been identified ascandidate first level nodes which have been expanded to reveal the next level nodes (ie, nodes in level “2”).

[0079] At block 630, for each of the one or more expanded candidate first level nodes the LLM identifies whether any of the next level of nodes includes one or more relevant next level nodes again based on the FF NL instruction. In various examples, the LLM will identify whether a next level node is relevant to the robot task based on an attribute of the node (eg, region name, location) and its relevance to the original FF NL instruction. As shown in FIG. 6, only candidate first level node 611 is identified to have a relevant next level node 621 whereas the next level nodes of the other candidate first level node 612 have been determined to not be relevant.

[0080] At block 640, a candidate first level node 611 is then assigned by the LLM to be a relevant first level node 641 on identification of one or more relevant next level nodes 621 for the candidate first level node 611. As shown in FIG. 6, in this case a single relevant first level node 641 has been identified.

[0081] In this example, for any candidate first level nodes where no relevant next level nodes have been identified, then the one or more graph manipulation commands are determined by the LLM to comprise one or more commands to contract these candidate first level nodes further reducing the size of any subgraph that is determined.

[0082] In various examples, this process may be repeated to identify relevant successive level nodes by expanding one or more candidate successive level nodes to reveal a next level of nodes as described above and then identifying for each of the one or more expanded candidate successive level nodes whether the next level of nodes includes relevant next level nodes based on the FF NL instructions. Upon determination that an expanded candidate successive level node has one or more relevant next level nodes then this candidate successive level node can be assigned or deemed to be a relevant successive level node. Again, any candidate successive level node where no relevant next level nodes have been identified may be contracted to reduce the size of the resulting subgraph.

[0083] It will be appreciated that the process of expanding a candidate node to reveal a next level of nodes is not limited to operations on the collapsed hierarchical graph data structure but may include any operation or process that allows information about the candidate node to be retrieved including, but not limited to, a database lookup of the subgraph associated with a candidate node selected by the LLM or in another example to the use of a retrieval-augmented generation (RAG) approach to search for preencoded subgraphs associated with the candidate node.

[0084] As would be appreciated, once all the relevant regions and items required to complete the task have been identified, any remainder of the hierarchical graph data structure will not require processing or exploring.

[0085] Consider the example of the following FF NL instruction for a robot to carry out a task in hierarchical multi-region environment:FF NL instruction: make a coffee for Tom and place it in his room

[0086] In this example, the hierarchical multi-region environment comprises a single level of regions corresponding to “room” and multiple regions within this level defined as “bobs room”, “toms room”, “jacks room”, “kitchen”, “livingroom”. In one example, the hierarchical multi-region environment and any asociated items may be characterised by the following hierarchical graph data structure in the form of a scene graph:Scene Graph {nodes: {room: [{id: bobs room}, {id: toms room}, {id: jacks room}, {id: kitchen}, {id: livingroom}], pose: [{id: posel}, {id: pose2}, {id: pose3}, {id: pose4}, {id: pose5}], agent: [{location: bobs room, id: agent}], asset: [{room: toms room, state: free, affordances: [release], id: bed2}, {room: toms room, state: closed, affordances: [open, close, release], id: wardrobe2}, {room: kitchen, state: closed, affordances: [open, close, release], id: fridge}, {room: kitchen, affordances: [turn on, turn off], state: off, id: coffee machine}, {room: bobs room, state: free, affordances: [release], id: bedl}, {room: bobs room, state: closed, affordances: [open, close, release], id: wardrobel}], object: [{affordances: [pickup], state: inside of(wardrobel), attributes: "blue", id: coffee mug}]}, links: [bobs_room<->posel, bobs_room<->agent, bobs_room<->bedl, bobs_room<->wardrobel, toms_room<-> posel, toms_room<->pose2, toms room pose5, toms_room<->bed2, toms_room<->wardrobe2, jacks_room<->pose2, jacks_room<->pose3, kitchen<->pose3, kitchen<->pose4, kitchen<-> poseS, kitchen<->fridge, kitchen<->coffee_machine, Iivingroom<->pose4, wardrobe l<->coffee_mug]}

[0087] In this example, the associated items are defined as further levels “asset” and successive level “object” in the scene graph similar to the hierarchical multi-region environment shown in FIG. 3 except that there is no “floor” level. Also identified is the “agent” which in this case corresponds to the robot and its original region is indicated. The scene graph further comprises “links” which in this example identify the various robot poses and the locations of the various items.

[0088] In accordance with some embodiments of the present disclosure, a collapsed or contracted hierarchical graph data structure comprising the first level of nodes corresponding to regions of the firstlevel (in this case only the region level) is provided. In one example, this may be provided as a reduced scene graph as follows:Scene Graph: {nodes: {room: [{id: bobs room}, {id: toms room}, {id: jacks room}, {id: kitchen}, {id: livingroom}], pose: [{id: posel}, {id: pose2}, {id: pose3}, {id: pose4}, {id: pose5}], agent: [{location: bobs room, id: agent}]}, links: [bobs_room<->posel, bobs_room<->agent, toms room posel, toms_room<->pose2, toms_room<->pose5, jacks_room<->pose2, jacks_room<->pose3, kitchen<->pose3, kitchen<->pose4, kitchen<-> poseS, Iivingroom<->pose4]}

[0089] In accordance with some embodiments, one or more graph manipulation commands are generated by the LLM to identify those regions that are relevant to the robot task.

[0090] In this example, the LLM identifies relevant first level nodes based on the FF NL instructions and the one or more graph manipulation commands. In the example below, the LLM has been prompted to divide the processing of the FF NL instruction into intermediate reasoning steps as follows: chain-of-thought: explore graph for coffee mug, coffee machine and somewhere to place the coffee -> perform common sense exploration first -> explore toms room > explore kitchen -> if items not found then explore other nodes

[0091] resulting in the LLM identifiying a candidate first level node as follows: reasoning: i will expand toms room

[0092] Following identification of the candidate first level node by the LLM, the LLM determines that that the graph manipulation command comprises a command to expand this candidate node as follows: command: expand(toms room)

[0093] In this example, the LLM calls an external application programming interface (API) that functions to expand the first level node corresponding to “toms room” to reveal the next level of nodes which in this example corresponds to items associated with “toms room”.

[0094] The LLM then proceeds to search for other candidate first level nodes as follows:chain-of-thought: i have found a wardrobe in tom's room -> leave this node expanded -> the coffee mug is not in his room -> still have not found the coffee machine -> kitchen might have coffee machine and coffee mug -> explore this node next

[0095] and another candidate first level node is identified: reasoning: i will expand the kitchen next

[0096] with the LLM then determining the graph manipulation command to expand the next candidate first level node as follows: command: expand(kitchen)

[0097] to in this case reveal the items associated with the kitchen.

[0098] This process may then be repeated to search for other candidate first level nodes based on the FF NL instruction as follows:: chain-of-thought: i've found the coffee machine in the kitchen -> the coffee mug however is not in the kitchen -> i will explore other unexpanded nodes now for the coffee mug -> jacks room, bobs room and the living room are unexplored yet -> i will try jack's room next

[0099] and another candidate first level node is identified: reasoning: i will expand jacks room next

[0100] with the LLM then expanding this next candidate first level node as follows: command: expand(jacks room)

[0101] Processing of “jacks room” node is then carried out to determine its relevance based on the FF NL instruction as follows: chain-of-thought: jacks room does not have anything relevant towards solving this task -> I should contract this node to save token memory -> i will explore other unexplored nodes after doing this

[0102] which in this example determines that the node corresponding to “jacks room” is not a relevant node: reasoning: contract jack's room to save memory

[0103] with the LLM determining that the one or more graph manipulation commands comprises a command to contracting this node to save memory as follows: command: contract(jacks room)

[0104] This process continues as follows: chain-of-thought: i have found a wardrobe in tom's room and coffee machine in the kitchen -> the coffee mug is still required for the task -> explore a new room next

[0105] and another candidate first level node is identified: reasoning: i will expand bob's room as it is unexplored

[0106] with the LLM then expanding this next candidate first level node as follows: command: expand(bobs room)

[0107] Processing of the “bobs room” node is then carried out to determine its relevance based on the FF NL instruction as follows: chain-of-thought: i have found the coffee mug, the coffee machine and tom's wardrobe on the graph -> search complete -> I will switch to planning mode-> collect coffee mug -> generate plan for making coffee -> place coffee mug on Tom's wardrobe

[0108] In this manner, a subgraph of the original hierarchical graph data structure has been generated in accordance with this example that is relevant to the robot task of “make a coffee for Tom and place it in his room”. In this example, the subgraph comprises those nodes corresponding to regions of the hierarchical multi-region environment (ie, toms room, kitchen, bobs room) that contain the items (eg, coffee mug, coffee machine, etc) necessary to complete the task.

[0109] As would be apparent, the above example may be generalised to any hierarchical multiregion environment involving successive levels of nodes which may be identified and then expanded orcontracted depending on whether they involve relevant next level nodes, where successive levels of nodes could be either regions or items defined within the node structure that are relevant to the robot task.

[0110] As can be seen, in accordance with the present disclosure given the top or first level of a hierarchical graph data structure representing the hierarchical multi-region environment and the FF NL robot task instruction, an LLM configured in accordance with the present disclosure generates a set of graph manipulation commands to identify the task relevant subgraph.

[0111] In one example, where the full hierarchical graph data structure is already preconstructed and collapsed to the top or first level, the LLM generates graph manipulation commands in the form of expanding or contracting nodes to manipulate the hierarchical graph data structure in order to identify the desired task relevant subgraph.

[0112] In another embodiment, where only a partial hierarchical graph data structure is available, the LLM may generate a graph manipulation command in the form of an “explore” command for a node which may then form part of the task plan (see below) to direct the physical robot to a desired region corresponding to the node where the robot (equipped with a sensor modality) then detects necessary items in the region using its sensors. Based on the detected items, a level of the hierarchical graph data structure which was previously empty may now be populated and the relevance of the node then reassessed to determine whether it should be part of the subgraph. In one example, this population of the hierarchical graph data structure and redetermination of the subgraph or determination whether other regions may need to be explored may occur substantially in real time. In one example, this may occur as the robot explores the multi-region environment.

[0113] In the example above, if only a partial hierarchical graph data structure was provided, where the kitchen node contained no items, a graph manipulation command may be generated for the robot to “explore” the kitchen, given the relevance of the “kitchen” to the coffee making task.

[0114] Referring now to FIG. 7, there is shown a flow diagram of an example method 700 for generating a task plan 715 for a robot in accordance with some embodiments. In this example, method 700 comprises at block 710 processing the FF NL instruction 105 by the LLM and the subgraph 135 to generate a task plan 715 comprising a sequence of actions for the robot to carry out the task. In various examples, the subgraph 135 may be determined in accordance with the present disclosure but in other examples it may be generated by other methods.

[0115] As discussed previously, by confining the LLM to the subgraph of a full hierarchical data structure this will assist in preventing the LLM from hallucinating and producing infeasible action sequences greatly improving the efficiency and practical utility of this process.

[0116] Referring now to FIG. 8, there is shown a flow diagram of an example method 800 for generating a task plan for a robot in accordance with some embodiments. In various example, block 710 of FIG. 7 may be implemented in accordance with method 800 of FIG. 8.

[0117] At block 810, an initial sequence of actions comprising robot locations and manipulations of any items identified in the subgraph to carry out the task is generated by the LLM.

[0118] At block 820, this initial sequence of actions is processed by a path planner to supplement the initial sequence of actions with robot navigational instructions to generate the sequence of actions 825 for the robot to carry out the task.

[0119] In this example, and by contrast to the more general method 700 depicted in FIG. 7, the LLM is confined to generating only the respective regions where the robot will need to be present and the required item manipulations, and not a complete plan involving also the navigational instructions for the robot to move between the various regions of the subgraph that are identified in the initial sequence of actions. The task of generating the actions may then be carried out by a path planner such as Dijkstra which functions to find the optimal route between high-level locations. This functions to shorten the LLM’s planning horizon and allows the LLM to focus on the essential item manipulation components of the robot task.

[0120] As would be appreciated, any suitable path planning or graph search algorithm may be adopted including, but not limited to, A* Search or Breadth First Search. In other examples, the path planner may be configured to incorprate constraints based on the specific task. In one example, the path planning algorithm may be configmed to determine a type of path including, but not limited to, the shortest path, the path involving the least change in elevation, a level path, a path navigatable by a wheeled vehicle, and / or a path that avoids hazardous regions in the multi-region environment, etc.

[0121] Referring now to FIG. 9, there is shown a flow diagram of an example method 900 for verifying and updating a task plan for a robot in accordance with some embodiments. In one example, method 900 may operate on a sequence of actions 825 generated in accordance with method 800 of FIG. 8.

[0122] At block 910 of method 900, the sequence of actions is simulated orverified to determine if the sequence of actions would be actionable by the robot, with feedback provided indicating conflicting actions where this occurs. In one example, the simulation determines whether the generated task plan complies with the predicates, state, and affordances of the subgraph.

[0123] If there are no conflicting actions, the input sequence of actions 825 will form the final or verified sequence of actions comprising the task plan for the robot.

[0124] At block 920, on identification of any conflicting actions in the sequence of actions, the LLM based on the conflicting actions then generates a modified initial sequence of actions comprising robot locations and manipulation of items identified in the subgraph to carry out the task.

[0125] At block 930, this modified initial sequence of actions is processed by the path planner to supplement this modified initial sequence of actions with robot navigational instructions to update the sequence of actions for the robot to carry out the task (see also block 820 of FIG. 8). This verification process can then be iterated or repeated until a final verified sequence of actions 915 is produced.

[0126] In accordance with some embodiments of the present disclosure, the following is an example of the generation of a task plan based on the subgraph generated in the example referred to above based around the FF NL instruction of “make a coffee for Tom and place it in his room”. In this example, the subgraph contained the nodes [toms room, kitchen, bobs room] and an initial sequence of actions comprising robot locations and manipulation of any items identified in the subgraph is generated by the LLM based on the subgraph and the original NL instruction. In this example, the LLM is configured to generate a sequence of reasoned steps that will form the basis of the initial sequence of actions as follows: chain-of-thought: "i have found the coffee mug, the coffee machine and tom's wardrobe on the graph -> search complete -> I will switch to planning mode-> collect coffee mug -> generate plan for making coffee -> place coffee mug on Tom's wardrobe"

[0127] with the resulting initial sequence of actions directed to robot locations and manipulations generated as follows:[goto(bobs room) > access(wardrobel) > pickup(coffee mug) > goto(kitchen) > access(coffee machine) > release(coffee mug) > turn on(coffee machine) > turn off(coffee machine) > pickup(coffee mug) > goto(toms room) > access(wardrobe2) > release(coffee mug) > done]

[0128] This initial sequence of actions is then processed by a path planner to supplement the initial sequence of instruction with robot navigational instructions to generate the sequence of actions for the robot to carry out the task.[goto(pose3) > goto(pose4) > goto(posel) > goto(bobs room) > access(wardrobel) > pickup(coffee mug) > goto(pose2) > goto(pose5) > goto(kitchen) > access(coffee machine) > release(coffee_mug)> turn on(coffee machine) > turn off(coffee machine) > pickup(coffee mug) > goto(pose5) > goto(pose2) > goto(toms room) > access(wardrobe2) > release(coffee mug) > done]

[0129] In accordance with one example, this sequence of actions may be verified by simulating the sequence of actions to determine if they are actionable and provide feedback indicating any conflicting actions.

[0130] In this case, the simulation process indicates that:"coffee mug is not accessible"

[0131] Based on this feedback, a modified initial sequence of actions is generated by the robot based on the feedback from the simulation process involving the identification of conflicting actions. In this example, the LLM is configured to generate a sequence of reasoned steps that will form the basis of the modified initial sequence of actions as follows: chain-of-thought: "the coffee mug was not accessible -> the wardrobe it is in is currently closed - > I update the plan to open the wardrobe first -> generate plan for making coffee -> place coffee mug on Tom's wardrobe"

[0132] with the resulting modified initial sequence of actions directed to robot locations and manipulations generated as follows:[goto(bobs room) > access(wardrobel) > open(wardrobel) > pickup(coffee mug) > goto(kitchen) > access(coffee machine) > release(coffee mug) > turn on(coffee machine) > turn off(coffee machine) > pickup(coffee mug) > goto(toms room) > access(wardrobe2) > release(coffee mug) > done]

[0133] This modified initial sequence of actions is then again processed by the path planner to supplement the initial sequence of instruction with robot navigational instructions to generate the sequence of actions for the robot to carry out the task.[goto(pose3) > goto(pose4) > goto(posel) > goto(bobs room) > access(wardrobel) > open(wardrobel) > pickup(coffee mug) > goto(pose2) > goto(pose5) > goto(kitchen) > access(coffee machine) > release(coffee mug) > turn on(coffee machine) > turn off(coffee machine) > pickup(coffee mug) > goto(pose5) > goto(pose2) > goto(toms room) > access(wardrobe2) > release(coffee mug) > done]

[0134] In this case, the simulation process indicates that the sequence of actions would be actionable by the robot.

[0135] In one example, an input prompt is passed to the LLM to configure the LLM for the subgraph and / or task plan generation process in the form of an agent carrying out methods in accordance with the present disclosure.

[0136] In one example, the input prompt may comprise the agent’s role, details pertaining to the hierarchical multi-region environment, the desired output structure and a set of input-output examples which together form the static prompt used for in-context learning. In this example, the static input prompt is both task- and environment-agnostic and takes up -3900 tokens of the LLM’s input. In one example, during the generation of the subgraph, both the hierarchical graph data structure (in the form of a scene graph) and a memory component of the input prompt may be updated at each step. In the iterative generating of a task plan, a feedback component may be updated based on the simulator to determine whether a sequence of actions is actionable by a robot. In various examples, the LLM is prompted to output a JSON object containing arguments to call provided API functions (eg, “contract” a node, “expand” a node, etc).

[0137] In one example, the input prompt may have the following structure:Agent Role: You are an excellent graph planning agent. Given a graph representation of an environment, you can explore the graph by expanding nodes to find the items of interest. You can then use this graph to generate a step-by-step task plan that the agent can follow to solve a given instruction.Environment Functions: goto(<pose>): Move the agent to any room node or pose node. access(<asset>): Provide access to the set of affordances associated with an asset node and its connected objects. pickup(<object>): Pick up an accessible object from the accessed node. release(<object>): Release grasped object at an asset node. turn_on / off(<object>): Toggle object at agent's node, if accessible and has affordance.open / close(<asset>): Open / close asset at agent's node, affecting object accessibility. done(): Call when the task is completed.Environment State: ontop_of(<asset>): Object is located on <asset> inside_of(<asset>): Object is located inside <asset> inside hand: Object is currently being grasped by the robot / agent closed: Asset can be opened open: Asset can be closed or kept open on: Asset is currently on off: Asset is currently off accessible: The object is not accessible if it is inside an asset and the asset state is "closed".Environment API: expand_node(<node>): Reveal assets / objects connected to a room / floor node. contract_node(<node>): Hide assets / objects, reducing graph size for memory constraints. verify planf): Verify generated plan in the scene graph environment.Output Response Format:{chain of thought: break your problem down into a series of intermediate reasoning steps to help you determine your next command, reasoning: justify why the next action is important mode: "exploring" OR "planning" command: {"command name": Environment API call"node id": node to perform an operation on"plan": task plan if in planning mode}}Example: Provided prompt examples for scene graph manipulation and task plan generation to prime LLM behaviour.Instruction: Natural language description of the task3D Scene Graph: Text-serialised JSON description of a 3D scene graphMemory: History of previously expanded nodesFeedback: External textual feedback from scene graph simulator

[0138] In this example, the bold components represent static components of the prompt that remain fixed throughout both the generation of the subgraph and the task plan based on the subgraph.

[0139] In one example, GPT-4 is employed as the underlying LLM agent. As would be appreciated other LLMs including, but not limited to BERT, XLNet, Gemini, Phi-2, Llama, Palm, BLOOM, Ernie 3.0 Titan, Anthropic's Claude 2 and PanGu-E, Lambda, Vicuna, Falcon, or any fined tuned or custom LLM, may be adopted in accordance with the present disclosure unless otherwise stated.

[0140] Referring now to FIG. 10, there is shown a figurative process flow diagram 1000 of methods for generating a subgraph and task plan for a robot in accordance with some embodiments.

[0141] On the left hand of process flow diagram at block 1, a FF NL instruction 1010 is received by LLM 1050 for a robot 1095 to carry out a task in a hierarchical multi-region environment. In this example, the FF NL instructions 1010 is to “Make Peter a coffee”. A hierarchical graph data structure 1020 is received by LLM 1050 and in this example the hierarchical graph data structure is provided in a collapsed form as a 3D scene graph comprising only the first level nodes where the first level nodes correspond to regions of the first level of the hierarchical multi-region environment.

[0142] Moving now to block 2 of process flow diagram 1000, a graph manipulation command 1030 is generated by the LLM based on the FF NL instruction and the hierarchical graph data structure 1020 (in collapsed form in this example). In this example, graph manipulation command corresponds to an API command to expand the candidate relevant node identified as “kitchen”.

[0143] In this example, the expand command is processed by scene graph simulator 1035 at block 3 which is configmed on receiving an “expand” command for a particular node to return to LLM 1050 an updated hierarchical graph data structure 1021 (in this case an updated 3D scene graph) at block 4 that reveals all the nodes connected to the particular node in the level below. Furthermore, scene graph simulator 1035 is configured on receiving an “expand” command for a particular node to return an updated hierarchical graph data structure (in this case an updated 3D scene graph) that hides all the nodes connected to the particular node in the level below. This process is then repeated through the first and successive level nodes to identify those first level nodes and successive level nodes that are relevant to the robot task and hence will comprise the subgraph 1025. In this example, an updated hierarchical graph data structure 1021 where the “kitchen” node has been expanded is shown at block 4.

[0144] This process is shown figuratively at the bottom of the left hand side of FIG. 10 in the block titled “Full Search Sequence” moving from left to right starting from the original collapsed hierarchical graph data structure 1020 and showing where one of the first level nodes (ie, the centre first level node, ie “kitchen” node) has been expanded to generate updated hierarchical graph data structure1021 comprising a further three nodes in the second level that depend from the kitchen node. In accordance with the present disclosure, the left most node of the further three nodes is expanded in updated hierarchical graph data structure 1022 and then contracted in further updated hierarchical graph data structure 1023. Eventually, this process is completed with the updated hierarchical graph data structure corresponding to subgraph 1025 being identified at the end of the sequence.

[0145] Once the subgraph 1025 has been generated the next stage as shown in process flow diagram 1000 is the generation of the task plan. As shown in block 5, the FF NL instmction 1010 and the subgraph 1025 are processed by the LLM 1050 to generate the task plan (eg, see block 8) for the robot to carry out the task. In one example, LLM 1050 generates an initial sequence of actions 1060 comprising robot locations and manipulation of any items identified in the subgraph 1025 at block 6 (also referred to as a high level plan). The initial sequence of actions 1060 is then processed by path planner 1063 at block 7 as has been previously described to supplement the initial sequence of actions with robot navigational instructions to generate the sequence of actions 1065 for the robot to carry out the task at block 8.

[0146] The sequence of actions 1065 then goes through a verification process where the sequence of actions is simulated by in this example scene graph simulator 1035 which is also operable at block 9 to verify any sequence of actions 1065 to determine if they are actionable and to provide feedback 1038 indicating conflicting action in the sequence of actions 1065, As depicted, the feedback indicates “Cannot release coffee mug here” and the process then continues with the LLM 1050 generating a modified initial sequence of actions at block 6 which are then supplemented at block 7 by path planner 1063 to include robot navigational instructions to update the sequence of actions 1065 at block 8. This process is then iterated until the sequence of action is determined to be actionable by the robot by in this example scene graph simulator 1035 at block 9.

[0147] In this example, the verified sequence of actions in then passed to a low level motion planner 1037 at block 10 to generate low level executable instructions for the particular robot 1095.

[0148] Referring now to FIG. 11, there is shown an overview diagram of a hierarchical graph data structure 1100 and the corresponding hierarchical multi-region environment and associated items 1150 in accordance with some embodiments. In this example, the hierarchical multi-region environment 1150 corresponds to a large-scale office floor spanning 36 rooms (or regions) and 150 associated items comprising assets and objects which a robot can potentially interact with to carry out a task. In this example, the initial starting position 1105 of the robot (or Agent) in the hierarchical graph tree structure 1100 is indicated.

[0149] Referring now to FIG. 12, there is shown the hierarchical multi-region environment and associated items 1150 shown in FIG. 11 and a collapsed or contracted hierarchical graph tree structure1200 that corresponds to hierarchical graph data structure 1100 shown in FIG. 11. In this example, collapsed or contracted hierarchical graph tree structure, contains only the highest level nodes (ie, first level) of the original hierarchical graph tree structure 1100 representing a reduction of approximately 80 % in the number of nodes that will be initially assessed in accordance with the present disclosure to determine the subgraph.

[0150] In this manner, a collapsed or contracted hierarchical graph structure may be provided and the subgraph then determined by manipulating the contracted hierarchical graph data structure in accordance with the FF NL instruction and the one or more graph manipulation commands to determine relevant nodes relevant to the task.

[0151] In various aspects, the FF NL instruction may be classified in accordance with the following categories.

[0152] Simple Search: In this example, the FF NL instruction may directly reference information in the hierarchical graph data structure as well as the basic graph based reasoning capabilities of the LLM.

[0153] Complex Search: In this example, the FF NL instruction may involve an abstract semantic search query which requires complex reasoning and the information required to generate a task plan may not be readily available in the hierarchical graph data structure and has to be inferred by the underlying LLM.

[0154] Simple Planning: In this example, the FF NL instruction may involve a task planning query which requires a search of the hierarchical graph data structure, causal reasoning and environment interaction by the underlying LLM in order to generate the task plan. In various examples, this will typically require shorter horizon plans over single regions.

[0155] Long Horizon Planning: In this example, the FF NL instruction may involve a task planning query which requires multiple interactive steps and reasoning over temporally extended instructions by the underlying LLM. In various examples, this will require longer horizon plans spanning multiple regions.

[0156] Referring now to FIG. 13, there is shown an architecture overview diagram of a robot 1300. Throughout the present disclosure, the term robot is taken to mean a mobile robot capable of moving in a hierarchical multi-region environment. Depending on the task, a robot may include, but not be limited to, a mobile ground based robot, an unmanned aerial vehicle ("UAV") based robot, a humanoid robot, a robot capable of moving over or under water or any combination of these capabilities.

[0157] In this example, robot 1300 includes a robot control system 1310, one or more operational components 1320A-Z, and one or more sensors 1330A-Z. Sensors 1330A-Z may include, but not be limited to, vision sensors, light sensors, pressure sensors, pressure wave sensors, acoustic sensors, proximity sensors, accelerometers, gyroscopes, thermometers and barometers. While sensors 1330A-Z are depicted as being integral to robot 1300, in various examples, one or more of sensors 1330A-Z may be located external to robot 1300 as standalone units.

[0158] Operational components 1320A-Z may include, for example, one or more end effectors and / or one or more servo motors or other actuators to effectuate movement of one or more components of the robot. For example, the robot 1300 may have multiple degrees of freedom and each of the actuators may control the actuation of the robot 1300 within one or more of the degrees of freedom responsive to the control commands. The term actuator encompasses a mechanical or electrical device that creates motion (eg, a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion including motion of the robot.

[0159] The robot control system 1310 may be implemented in one or more processors, such as a CPU, GPU, and / or other controllers) of the robot 1300. In some implementations, the robot 1300 may comprise a "brain box" that may include all or aspects of the control system 1310. For example, the brain box may provide real time bursts of data to the operational components 1320A-Z, with each of the real time bursts comprising a set of one or more control commands that dictate, inter alia, the parameters of motion (if any) for each of one or more of the operational components 1320 A-Z. In some implementations, the robot control system 1310 may be configured to carry out a task plan comprising a sequence of actions in accordance with the present disclosure.

[0160] Although control system 1310 as an integral part of the robot 1300, in some implementations, all or aspects of the control system 1310 may be implemented in a component that is separate from, but in communication with, robot 1300. For example, all or aspects of control system 1310 may be implemented on one or more computing devices that are in wired and / or wireless communication with the robot 1300 such as a separate computing device or system (eg, see FIG. 14).

[0161] Referring now to FIG. 14, there is shown an architecture overview diagram of an example computer system or computing device 1400 that may optionally be utilised in accordance with the present disclosure. Computing device 1400 typically includes at least one processor 1410 which communicates with a number of peripheral devices via bus subsystem 1450. These peripheral devices may include a storage subsystem 1440, including, for example, a memory subsystem 1441 and a filestorage subsystem 1445, user interface input / output devices 1430, and a network interface subsystem 1420. The input and output devices 1430 allow user interaction with computing device 1400.

[0162] Network interface subsystem 1420 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices. User interface input devices 1430 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computing device 1400 or onto a communication network.

[0163] User interface output devices 1430 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term "output device" is intended to include all possible types of devices and ways to output information from computing device 1400 to the user or to another machine or computing device.

[0164] Storage subsystem 1440 stores programming and data constructs that provide the functionality of some or all implementations in accordance with the present disclosure. For example, the storage subsystem 1440 may include the logic to perform selected aspects of method 100 of FIG. 1, and / or method 400 of FIG. 4, and / or methods 500 or 550 of FIGS. 5 A and 5B, and / or method 600 of FIG. 6, and / or method 700 of FIG. 7, and / or method 800 of FIG. 8 and / or method 900 of FIG. 9.

[0165] These software modules are generally executed by processor 1410 alone or in combination with other processors. Memory 1441 used in the storage subsystem 1440 can include a number of memories including a main random access memory (RAM) 1443 for storage of instructions and data during program execution and a read only memory (ROM) 1442 in which fixed instructions are stored. A fde storage subsystem 1445 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystem 1445 in the storage subsystem 1440, or in other machines accessible by the processor(s) 1410.

[0166] Bus subsystem 1450 provides a mechanism for letting the various components and subsystems of computing device 1400 communicate with each other as intended. Although bus subsystem1450 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0167] Computing device 1400 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 1400 depicted in FIG. 14 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 1400 are possible having more or fewer components than the computing device depicted in FIG. 14.

[0168] The computing device 1400 may run any suitable operating system including, but not limited to, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, any operating systems for mobile computing devices, or any other operating system capable of running on the computing device 1400 and performing the operations described in this disclosure. In an embodiment, the operating system may be run on one or more cloud machine instances.

[0169] As will be appreciated in light of this disclosure, the various computer-implemented methods of the present disclosure, are implemented in software, such as a set of instructions (eg, HTML, XML, C, C++, object oriented C, BASIC, Python, etc.) encoded on any computer readable medium or computer program product (eg, hard drive, server, disc, or other suitable non-transitory memory or set of memories), that when executed by one or more processors, cause the various methodologies provided in this disclosure to be carried out.

[0170] Implementations or aspects in accordance with the present disclosure are directed to where a large-scale environment can rapidly grow in complexity making it impractical to process exhaustively and / or pass any representation of the environment to a LLM due to token limits.

[0171] Implementations or aspects in accordance with the present disclosure also recognise that some path planning tasks, such as determining an optimal navigational path, may not be suitable for an LLM and may carried out by other path planning optimisation processors based on an initially generated task plan from an LLM. Furthermore, a task plan verification process can provide feedback to an LLM to allow it to iterate to an actionable task plan.

[0172] Other aspects of a computer-implemented to generate a subgraph and task plan for a robot based on a FF NL instruction may be found in “SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning” , Rana et al, the disclosure of which is incorporated by reference in its entirety.

[0173] The reference to any prior art in this specification is not, and should not be taken as, an acknowledgement or any form of suggestion that such prior art forms part of the common general knowledge.

[0174] It will be understood that the terms “comprise” and “include” and any of their derivatives (e.g. comprises, comprising, includes, including) as used in this specification, and the claims that follow, is to be taken to be inclusive of features to which the term refers, and is not meant to exclude the presence of any additional features unless otherwise stated or implied.

[0175] In some cases, a single embodiment may, for succinctness and / or to assist in understanding the scope of the disclosure, combine multiple features. It is to be understood that in such a case, these multiple features may be provided separately (in separate embodiments), or in any other suitable combination. Alternatively, where separate features are described in separate embodiments, these separate features may be combined into a single embodiment unless otherwise stated or implied. This also applies to the claims which can be recombined in any combination. That is a claim may be amended to include a feature defined in any other claim. Further a phrase referring to “at least one of’ a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover: a, b, c, a-b, a-c, b-c, and a-b-c.

[0176] It will be appreciated by those skilled in the art that the disclosure is not restricted in its use to the particular application or applications described. Neither is the present disclosure restricted in its preferred embodiment with regard to the particular elements and / or features described or depicted herein. It will be appreciated that the disclosure is not limited to the embodiment or embodiments disclosed, but is capable of numerous rearrangements, modifications and substitutions without departing from the scope as set forth and defined by the following claims.

Claims

CLAIMS1. A method, implemented by one or more processors, comprising: receiving, by a large language model (LLM), a free-form (FF) natural language (NL) instruction for a robot to carry out a task in a hierarchical multi -region environment; receiving, by the LLM, a hierarchical graph data structure characterising the hierarchical multiregion environment and associated items; processing, by the LLM, the FF NL instruction and the hierarchical graph data structure to generate a subgraph of the hierarchical graph data structure relevant to the task.

2. The method of claim 1, wherein processing, by the LLM, the FF NL instruction and the hierarchical graph data structure to generate a subgraph of the hierarchical graph data structure relevant to the task comprises: generating, by the LLM, from the FF NL instruction and the hierarchical graph data structure one or more graph manipulation commands to identify at least one region in the hierarchical graph data structure relevant to the task; processing, by the LLM, the hierarchical graph data structure based on the one or more graph manipulation commands and the FF NL instruction to determine the subgraph comprising the selection of regions relevant to the task.

3. The method of claim 2, wherein processing, by the LLM, the hierarchical graph data structure based on the one or more graph manipulation commands and the FF NL instruction to determine the subgraph comprises: providing a collapsed or contracted hierarchical graph data structure; and determining the subgraph by manipulating the contracted hierarchical graph data structure in accordance with the FF NL instruction and the one or more graph manipulation commands to determine relevant nodes of the hierarchical graph data structure for the task.

4. The method of claim 2, wherein processing, by the LLM, the hierarchical graph data structure based on the one or more graph manipulation commands and the FF NL instruction to determine the subgraph comprises: providing to the LLM first level nodes from the hierarchical graph data structure corresponding to regions of the first level of the hierarchical multi-region environment; identifying, by the LLM, relevant first level nodes from the first level of nodes based on the FF NL instruction and the one or more graph manipulation commands; identifying, by the LLM, relevant successive level nodes from any successive levels of nodes of the hierarchical graph data structure corresponding to regions of the successive levels of the hierarchicalmulti-region environment based on the FF NL instruction and the one or more graph manipulation commands, wherein the relevant successive level nodes depend initially from the identified relevant first level nodes and in turn from identified relevant successive level nodes identified from a node level above; determining the subgraph to be the identified relevant first level nodes and relevant successive level nodes.

5. The method of claim 4, wherein identifying, by the LLM, relevant first level nodes based on the FF NL instruction and the one or more graph manipulation commands comprises: identifying, by the LLM, one or more candidate first level nodes based on the FF NL instruction; determining, by the LLM, the one or more graph manipulation commands to comprise one or more commands to expand the one or more candidate first level nodes to reveal a next level of nodes; identifying, by the LLM, for each of the one or more expanded candidate first level nodes whether any of the next level of nodes includes one or more relevant next level nodes based on the FF NL instruction; assigning, by the LLM, a candidate first level node to be a relevant first level node on identification of one or more relevant next level nodes for the candidate first level node.

6. The method of claim 5, further comprising: determining, by the LLM, the one or more graph manipulation commands comprise one or more commands to contract any candidate first level nodes on a failure to identify relevant next level nodes.

7. The method of any one of claims 4 to 6, wherein identifying, by the LLM, relevant successive level nodes based on the FF NL instruction and the one or more graph manipulation commands, comprises: determining, by the LLM, the one or more graph manipulation commands to comprise one or more commands to expand the one or more candidate successive level nodes to reveal a next level of nodes; identifying, by the LLM, for each of the one or more expanded candidate successive level nodes whether any of the next level of nodes includes one or more relevant next level nodes based on the FF NL instruction; assigning, by the LLM, a candidate successive level node to be a relevant successive level node on identification of one or more relevant next level nodes for the candidate successive level node.

8. The method of claim 7, further comprising: determining, by the LLM, the one or more graph manipulation commands comprise one or more commands to contract any candidate successive level node on failure to identify relevant next level nodes.

9. The method of any one of claims 4 to 8, wherein one or more successive level nodes comprises items associated with regions of the hierarchical multi-region environment.

10. The method of any one of claims 1 to 9, wherein processing, by the LLM, the FF NL instruction and the hierarchical graph data structure to generate a subgraph of the hierarchical graph data structure relevant to the task comprises initially prompting the LLM to divide the processing into intermediate steps.

11. The method of any one of claims 2 to 10, wherein the one or more graph manipulation commands are processable by the LLM.

12. The method of any one of claims 2 to 11, wherein the one or more graph manipulation commands comprises a command for the robot to explore a region of the hierarchical multi-region environment corresponding to an unpopulated portion of the hierarchical graph data structure to identify items located in the region.

13. The method of claim 12, wherein the hierarchical graph data structure is further populated based on items identified by the robot from exploring the region.

14. The method of any one of claims 1 to 13, further comprising: processing, by the LLM, the FF NL instruction and the subgraph to generate a task plan comprising a sequence of actions for the robot to carry out the task.

15. The method of claim 14, wherein processing, by the LLM, the FF NL instruction and the subgraph to determine a task plan comprising a sequence of actions for the robot to carry out the task comprises: generating, by the LLM, an initial sequence of actions comprising robot locations and manipulation of any items identified in the subgraph to carry out the task; processing the initial sequence of actions by a path planner to supplement the initial sequence of actions with robot navigational instructions to generate the sequence of actions for the robot to carry out the task.

16. The method of claim 15, further comprising: simulating the sequence of actions to determine if the sequence of actions is actionable by the robot and provide feedback indicating conflicting actions; on identification of any conflicting actions in the sequence of actions generating, by the LLM, based on the conflicting actions a modified initial sequence of actions comprising robot locations and manipulation of items identified in the subgraph to carry out the task; andprocessing the modified initial sequence of actions by the path planner to supplement the modified initial sequence of actions with robot navigational instructions to update the sequence of actions for the robot to carry out the task.

17. The method of any one of the preceding claims, wherein the hierarchical multi-region environment comprises regions defined to be in at least two levels of a hierarchy characterising the hierarchical multi-region environment.

18. The method of any one of the preceding claims, wherein the hierarchical graph data structure is a scene graph.

19. A system, comprising memory storing instructions and one or more processors configmed or operable to execute the instructions to perform the method of any one of claims 1 to 18.

20. A robot, comprising: one or more actuators; one or more end effectors; one or more processors operable or configured to carry out the task plan of any one of claims 14 to 18.

Images