Methods and systems for planning complex tasks

Abstract

This provides a method and system for planning complex tasks using multiple software tools based on natural language input. [Solution] The method includes searching a tool graph. The tool graph comprises multiple nodes representing multiple tools and multiple edges between the multiple nodes indicating compatibility between the multiple tools. The method also includes receiving a natural language request from a user that describes a complex task, determining a subset of multiple tools by using the natural language request to determine the subset of multiple tools that is most similar to the request, generating an initial plan using a reasonable large language model "LLM" 405 that uses the natural language request and the subset of multiple tools as input, and generating a final plan from the initial plan.

Description

[Technical Field]

[0001] The embodiments described herein relate to methods and systems for planning complex tasks. [Background technology]

[0002] The automated tool invocation method allows for the formation of a sequence of multiple software tools or multiple application programming interfaces (APIs) to solve a complex task given by natural language input.

[0003] The rapid development of multiple powerful large-scale language models (LLMs) has led to significant improvements in the performance of such automated task planning. Training multiple powerful LLMs from scratch is costly and difficult. [Brief explanation of the drawing]

[0004] Embodiments will be described with reference to the following drawings.

[0005] [Figure 1] Figure 1 is a schematic diagram of a request suitable for calling an automated tool according to the embodiment. [Figure 2] Figure 2 is a schematic diagram of a further request suitable for calling an automated tool according to the embodiment. [Figure 3] Figure 3 is a flowchart showing the steps of the method according to the embodiment. [Figure 4] Figure 4 is a schematic diagram of the tool graph. [Figure 5] Figure 5 is a schematic diagram of several tools identified in the initial plan. [Figure 6] Figure 6 is a schematic diagram of the flowchart for the final planning stage according to the embodiment. [Figure 7] Figure 7 is a schematic diagram of the two basic steps of the method according to the embodiment. [Figure 8] Figure 8 is a schematic diagram of a request suitable for calling an automated tool according to the embodiment. [Figure 9] Figure 9 is a flowchart showing the initial and final planning tools for the request in Figure 8. [Figure 10] Figure 10 is a schematic diagram of hardware that may be used in the method according to the embodiment. [Modes for carrying out the invention]

[0006] In one embodiment, a method is provided for planning a complex task using multiple software tools from natural language input, and this method is Searching the tool graph, where the tool graph comprises multiple nodes representing multiple tools and multiple edges between the multiple nodes indicating compatibility between the multiple tools. Receiving natural language requests from users that indicate complex tasks, This involves determining a first subset of tools using natural language requests, and then determining the subset of tools that most closely resembles the request. An initial plan is generated using a large-scale language model (LLM) capable of reasoning, which takes a natural language request and a first subset of multiple tools as input, wherein the initial plan comprises a sequence of multiple subtasks, each subtask having at least one tool from the first subset of multiple tools assigned to it. Generating the final plan from the initial plan and To have and generate the final plan is This involves determining whether the tools in the initial plan are compatible with one adjacent tool in the sequence of multiple subtasks in the initial plan, thereby processing each subtask of the initial plan. Use the tool graph to select one or more new tools to resolve multiple incompatibility issues, Adding one or more new tools to the final plan, Using LLM to determine the tool call path for the final plan, The final plan is to execute and output a response to natural language input. It is equipped with.

[0007] The method described above addresses a technical problem related to computer technology and arising in the domain of multiple computer networks, namely, the technical problem of providing accurate and complex task plans while reducing the multiple computing resources required to plan a task. The disclosed method solves this technical problem by a task planner comprising four stages. The first stage is to build a tool graph, which may be performed offline but may not be performed in the guessing stage. The second stage is to search for multiple relevant tools or multiple APIs to resolve the request. It is possible to reduce the processing required by reducing the number of multiple tools initially considered to resolve the task. The third stage is to perform an initial task decomposition. The third stage does not need to be used in the tool graph. The fourth stage is to generate tool execution paths to resolve the input query.

[0008] In this embodiment, the tool graph is initially generated automatically based on multiple tool descriptions and then refined by a language model. Initial task decomposition is performed by the language model using a combination of ReAct and reflection-style prompts. To determine the tool execution path, the tool graph is used to assist the LLM in identifying the multiple tools required to complete the task. The search is performed sequentially to satisfy the multiple input requirements of the current tool, prioritizing previously used tool lists before broadening the search scope.

[0009] Through language comprehension and reasoning capabilities, multiple LLMs can decompose multiple complex user queries into multiple subtasks, select multiple suitable tools to solve them, identify multiple task dependencies, and generate multiple execution paths. Some limitations of multiple direct LLM inference planners can be mitigated by introducing graph-based methods, improving the ability of multiple LLMs to understand multiple relationships between multiple subtasks and multiple tools. To do this, a Thoughts-on-Graph (ToG) paradigm can be used, where multiple optimal paths are searched on multiple pre-built tool graphs. First, the user request is decomposed by the LLM into multiple subtasks, and for each subtask, the input and output data types required to solve them are estimated. Second, using the previous decomposition, for all subtasks, the pre-built graphs are searched for multiple candidate tools that can form a solution path that satisfies the input and output constraints. Multiple scores generated by the LLM are used to facilitate the decision process when multiple solution paths are found.

[0010] Multiple task planners based on direct LLM inference can suffer from several drawbacks. For example, multiple LLMs struggle to reason through long text information consisting of multiple user requests, multiple tool descriptions, and multiple related instructions. Furthermore, task planning using multiple tools often requires dealing with graph-structured data, but due to the autoregressive training strategies and network layouts of multiple LLMs, multiple current models are unable to adequately reason through these structures or distinguish between the multiple relationships they encode. For example, rearranging multiple nodes in a graph can yield different results; i.e., multiple LLMs lack invariance under graph isomorphism. These problems can lead to multiple hallucinations and multiple inaccurate tool calls, difficulties in complex task decomposition, or ineffective handling of multiple task dependencies, resulting in suboptimal plans or complete failure.

[0011] Compared to multiple direct LLM inference planners, the above embodiment simply utilizes a tool graph to enhance the plan generated by the language model. Unlike multiple tool graph search LLM planners, the method of the above embodiment is more robust because it does not require estimating multiple input-output types for each subtask. For each subtask, the method of deriving a solution path from the tool graph and the input and output types of the prediction is often prone to multiple errors, leading to multiple unusable plans, especially with multiple ambiguous user requests.

[0012] In an embodiment, when generating a final plan, a tool in an initial plan is compatible with one adjacent tool in a sequence of multiple subtasks when multiple inputs required by the tool match multiple outputs provided by multiple earlier tools in the sequence of multiple subtasks. In this example, a data conversion tool is required for the solution to function, however, such a tool cannot be inferred from the input requests (e.g., converting a binary mask to a bounding box or converting a video to a sequence of multiple images to perform object detection).

[0013] When a tool from an initial plan does not meet multiple compatibility requirements, the method further comprises selecting one or more new tools using a tool graph to resolve multiple incompatibility issues. This selection of multiple new tools may comprise a first stage of searching the tool graph for multiple tools that meet the multiple compatibility requirements of the current tool, the search being limited to a depth of d, where d is an integer of at least 2, and the search starting from multiple tools in previous subtasks of the initial plan and leading to the current tool. "d" can be set to any value that allows multiple compatible tools to be found. However, the larger "d" is, the more processing is required, and thus the smallest "d" should be used.

[0014] If the above does not return compatible tools, a second stage of searching the tool graph can be provided for multiple tools that meet the multiple compatibility requirements, the second stage of the search starting from multiple current tools and using all input edges to the current tool.

[0015] In an embodiment, the initial plan is refined by processing each subtask in a sequence and checking the compatibility of multiple tools. However, multiple other methods can be used, such as searching from predefined parts of a tool graph, from a specific tool, or from the end of a sequence. As the refinement of the plan proceeds in a sequence of multiple subtasks, the LLM can be used to generate a call path after each subtask is processed. Finally, after all subtasks are processed, a final call path is generated.

[0016] Since the above method functions for a restricted tool set, it has a first stage of filtering multiple tools that can be used before generating an initial plan to accelerate the processing of the initial plan. This first subset of multiple tools can be determined by comparing multiple embeddings of a natural language request with multiple embeddings of tool metadata of multiple tools in a tool graph.

[0017] The initial plan can be generated using an LLM, a first subset of multiple tools, and a natural language enquiry. Multiple prompts are used to generate the initial plan, and a ReAct prompt technique interleaved with a reflection stage can be used.

[0018] The initial plan is generated without access to a tool graph and without multiple relationships between multiple tools. However, a filtered list of multiple tools is provided to the LLM.

[0019] The tool graph is generated offline and can be searched when a user query is received. In a further embodiment, a tool graph is generated. Generating the graph can include receiving metadata M regarding the collection of multiple tools V, where the metadata includes a description of the function of the tool and a list of its input and output parameters, and generating a tool graph G=(V,E,M) can be E={(T i ,T j )|Ti ,T j Constructed for ∈V, tool T i The output is from tool T j This is a data type compatible with the input.

[0020] In a further embodiment, LLM is used to score the relevance of multiple previously found edges E based on multiple existing tool descriptions, and rejects multiple edges that score below a threshold.

[0021] The method may further comprise receiving an image or audio file, or a natural language query related to the image or audio file. Furthermore, at least one of a plurality of subtasks may comprise capturing image data or audio data. At least one of a plurality of subtasks may comprise the use of a neural network, such as a convolutional neural network. In a further embodiment, outputting a response comprises outputting at least one selected from the image or audio signal.

[0022] At least one of the multiple tools comprises an operation on at least one of the following: multiple pixels, multiple voxels, multiple phonemes, multiple graphemes, multiple frequencies, or multiple other basic building blocks of an image and / or audio file.

[0023] In the embodiments described above, the plan may be output for immediate execution, or it may be output for later execution. Thus, in further embodiments, the step of executing the final plan and outputting a response to natural language input may not be performed, and the method outputs a plan in response to a natural language query.

[0024] In the second aspect, a system is provided for planning complex tasks using multiple software tools from natural language input, and this system is: The tool graph is retrieved from memory, where the tool graph comprises multiple nodes representing multiple tools and multiple edges between the multiple nodes indicating compatibility between the multiple tools. Receiving natural language requests from users that indicate complex tasks, This involves determining a first subset of tools using natural language requests, and then determining the subset of tools that most closely resembles the request. An initial plan is generated using a large-scale language model (LLM) capable of reasoning, which takes a natural language request and a first subset of multiple tools as input, wherein the initial plan comprises a sequence of multiple subtasks, each subtask having at least one tool from the first subset of multiple tools assigned to it. Generating the final plan from the initial plan and Having a processor configured to perform the following, generating the final plan is This involves determining whether the tools in the initial plan are compatible with one adjacent tool in the sequence of multiple subtasks in the initial plan, thereby processing each subtask of the initial plan. Use the tool graph to select one or more new tools to resolve multiple incompatibility issues, Adding one or more new tools to the final plan, Using LLM to determine the tool call path for the final plan, The final plan is to execute and output a response to natural language input. It is equipped with.

[0025] Figure 1 is a schematic diagram of a user query provided in a task planning system. In this example, the user query concerns monitoring people in a queue, specifically, "Tell me who in this queue has a bag and what size that bag is." Due to several safety reasons and staffing planning, it may be desirable to know how many bags may need to be inspected. This is a complex task and will likely require the use of multiple APIs / software tools to complete it. It may also be necessary to modify the task; for example, further user queries about the same device might be, "Tell me who in the queue has a water bottle," "Tell me who in the queue is wearing a red jacket," etc.

[0026] Requests can be entered via keyboard, or they can be received as speech input via microphone.

[0027] In each of the multiple requests described above, an image is processed. An image may be captured in response to a request, or the request may be used to analyze a previously captured image. The image may be a still image or a video. Once the image data is received, it is processed by multiple tools to complete a complex task, and the results are output. The results may be an image, a text file, an audio file, or an audio output. In some embodiments, the output may be an alarm.

[0028] When instructions require image processing, various tools can be used, such as object recognition, depth estimation, bounding box estimation, and object registration. One or more of these tools may incorporate a neural network. A neural network will require a specific multi-dimensional input vector and, if applicable, one or more additional parameters. When arranging a sequence of multiple tools to perform a task, the multiple inputs of each tool must be compatible with the multiple outputs from multiple earlier tools in the sequence. Thus, determining a viable sequence of multiple tools is complex in that the multiple inputs and multiple outputs of multiple potentially cascaded neural networks must be compatible with one another.

[0029] Figure 2 is a further schematic diagram illustrating a user query provided within the task planning system, which is located on a mobile phone. The user enters a question: "Search for yesterday's video, remove the audio, play the video to make it look like a vintage film, and add music appropriate for a vintage film." The user can type the input or provide the question via the microphone. This is a complex task requiring multiple subtasks such as "search for the video," "remove the audio from the video," "provide a description of the video," "apply filters," and "add music based on the description."

[0030] Here, in addition to image processing, audio processing is also used. Various tools for audio processing may include noise reduction, voice recognition, speech recognition, frequency shifting, and boundary marking, etc.

[0031] The method described herein allows multiple complex tasks to be performed with reduced processing power because the LLM does not reason across the entire tool graph. Therefore, the method can be used on multiple mobile phones or multiple other devices where processing power may be limited. Smaller GPU memory, and thus smaller context sizes or multiple smaller LLMs, can be used with multiple smaller and less expensive GPU cards.

[0032] Figure 3 is a schematic diagram showing the four basic stages in the process. However, it should be noted that the first stage S403, which constructs the tool graph, can be performed offline and before any user queries are received.

[0033] The process is used by multiple tools / multiple APIs 401 and LLM 405. LLM 405 does not need to be specifically trained to work with the generated tool graph and can be a general-purpose LLM. In embodiments, the LLM is a powerful LLM, such as OpenAI's GPT-4. Several other reasoning-enabled language models, such as Llama, Claude, and Gemini, may be used.

[0034] In the tool graph in S403, a brief overview of the tool's functionality is available, accompanied by a detailed list of its input and output parameters, along with some basic metadata for multiple tools / APIs, such as their respective data types (e.g., string, numeric, depth map). In embodiments, a directed graph topology capable of storing multiple multi-edges is used, as schematically shown in Figure 4. Such a topology supports multiple edges E between two nodes, which is advantageous when multiple outputs from one tool T are required as multiple inputs for another tool T. Each edge E holds multiple data type attributes for distinguishing purposes. Considering the collection of multiple tools V and their metadata M, tool T i The output of tool T jThe initial version of the tool graph G=(V,E,M) is created such that it has data types compatible with the input, where E={(T i ,T j )|T i ,T j ∈V}. A refined version of G can be obtained by leveraging an LLM to score the relevance of multiple previously found edges E based on multiple existing tool descriptions. Multiple edges that result in scores below a threshold are discarded. For example, if the scoring scale ranges from 0 to 100, the threshold can be, say, 50. In further embodiments, when determining the multiple graph edges E, the knowledge of human experts can also be included to handle multiple special scenarios. This can be achieved via human interaction in the process or via a previously defined set of instructions. In many embodiments, the graph is automatically generated without human intervention.

[0035] Examples of multiple potential tools are those used in image and / or audio processing. For example, noise removal, semantic segmentation, depth estimation, edge identification, sequence identification, object recognition, object registration, speech recognition, voice recognition, etc. Multiple other tools can also be used to control multiple peripheral devices, such as multiple cameras, multiple robotics, etc.

[0036] The second sub-stage S407 is a search stage that reduces the processing required later on the tool graph. The number of tools that can access multiple task planners can be very large, and therefore designing a way to use all of them to search for a plan may be impractical. In the second sub-stage S407, the top K tools related to the input query are selected. In embodiments, this is achieved by matching the query with the metadata content of the multiple tools. This may be done by converting the metadata into multiple embedding vectors, which are then matched to the embeddings of the query using similarity scores. Many embedding and similarity measures may be used. For example, the BGE model [S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff, “C-pack: Packaged resources to advance general Chinese embedding,” 2023] may be used to generate multiple embedding vectors, and the FAISS score [M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy et al., “The Faiss library,” 2024. [Online].Available: https: / / arxiv.org / abs / 2401.08281] may be used to perform similarity searches. In embodiments, LLM is used to create multiple plaintext descriptions of multiple tools in order to maintain a consistent structure of multiple tool profiles, and multiple large documents are divided into multiple overlapping chunks (e.g., 2048 chunk sizes with 25% overlap). For example, searching K=25 tools provides a good trade-off between tool search performance and LLM token utilization. Therefore, in some embodiments, 25 or fewer tools may be found, and in other embodiments, 15 or fewer tools may be found.

[0037] Next, in the third sub-stage, task decomposition in S409 is performed. Here, a custom prompt is created that asks LLM405 to decompose the query into multiple subtasks that can be resolved using multiple given tools, using multiple previously selected tools. This can be achieved using the reflection phase and interleaved ReAct prompting technique [S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran et al., “React: Synergizing reasoning and acting in language models,” 2023. [Online]. Available: https: / / arxiv.org / abs / 2210.03629], and [N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan et al., “Reflexion: Language agents with verbal reinforcement learning,” 2023. [Online]. Available: https: / / arxiv.org / abs / 2303.11366] is used to both decompose the input task and suggest multiple candidate tools suitable for solving each subtask. However, several other prompting techniques may be used. The output of this stage (illustrated as Δ in Figure 3) will serve as the initial input for the fourth “planner” substage. In embodiments, a powerful LLM (e.g., OpenAI’s GPT-4) is used to effectively perform task decomposition. In embodiments, Δ comprises a list / sequence of multiple subtasks of form:('subtask': “description_text”, 'tools': [candidate_tools]). Thus, the subtask list also provides multiple candidate tools to be refined later.

[0038] Figure 5 is a schematic diagram of the initial plan associated with the user query in Figure 1, for example, "Tell me who in this column has a bag and what size their bag is." Several subtasks related to this plan are: (i) Detect bags in the image, (ii) Estimate the size of the bag, (iii) Render an image of a person holding a bag. It is possible.

[0039] In this scenario, Tool 1 could be an "object detection model," Tool 2 could be "object size estimation," and Tool 3 could be "image rendering." However, the multiple inputs for Tool 2 are not compatible with the multiple outputs from Tool 1, and therefore the initial plan needs to be refined. This is done in the planning stage when trying to find new Tools 1 and 2 (illustrated by multiple bright circles in Figure 5), for example, the new Tool 1 could be related to providing a bounding box, and the new Tool 2 could be a depth estimation tool.

[0040] Figure 6 is a flowchart of the planning phase. In the planning phase, the LLM, supported by tool graph G, aims to check the validity of the draft plan Δ, perform multiple corrections (if necessary), generate tool call paths, and resolve user queries.

[0041] In the first step S501, the LLM determines the input data type from a user query that will later be used to finalize the tool call path. If the planner knows the data to be received from the user query, the planner can ensure that the first tool selected to process this data is compatible with the input data. In further embodiments, the output data may also be determined from the user query or from the task decomposition stage.

[0042] Next, in the second step S503, for each subtask in Δ, the tool selection mechanism S505 is used to identify the most suitable (current) tool for each subtask when multiple candidates are available. For example, the first tool in the list may be selected, but several other selection mechanisms may also be applied. In many situations, only a single tool will exist for each subtask in the initial plan.

[0043] In this embodiment, the tool selection in S505 is greedy in that once a tool is selected, it is used, and multiple other selected tools for the same subtask in the initial plan are not used.

[0044] The previous list of multiple tools is used to verify in S507 whether multiple input arguments of multiple compatible types can be generated for the current tool. If the multiple arguments are incompatible, multiple new tools from the tool graph are then sought in step S509. The search within the tool graph at this stage is limited to a depth of d, for example, d=2. In embodiments, multiple paths from multiple previously identified tools to multiple current tools are first considered, where the current tool is the tool selected in S505, and the multiple previous tools are the multiple tools prior to the current tool in the plan. If multiple such paths are not available, all valid input edges to the current tool are used instead. The LLM is then used in S511 to select the most promising path when multiple options are available. Multiple newly identified tools are attached to the updated plan, and the process continues in step S513 for the next entry of Δ. If the multiple input arguments of the current tool are compatible with multiple previous tools in S507, no further search for multiple tools is performed, and the method proceeds to step S513.

[0045] Once all entries have been parsed, the LLM generates a tool call path using a revised list of multiple arguments and multiple tools in S515. The LLM is used at the end of the tool selection to generate a final execution plan from the updated list of multiple tools in S515. However, the LLM is also used during the graph-based tool update process when it is found that multiple nodes produce desired missing arguments in S509. In this case, the LLM is asked to choose the best tool. It is better to make the selection here rather than later saturating the LLM with multiple redundant tools.

[0046] Multiple solutions can be generated by adjusting the number of paths retained from S511, the depth of the graph search, or the tool selection mechanism.

[0047] Figure 7 shows a summary of the multiple processes described above, in which the tool graph provides an enhanced LLM-based task planner. Firstly, in S551, the language model breaks down the request into multiple subtasks, and for each subtask, one or more suitable candidate tools are selected (initial plan). Secondly, in S553, a pre-generated tool graph is used to assist the LLM in improving the initial plan by identifying the multiple tools required to complete the task.

[0048] The methods and systems described above can also be applied to multiple queries that require multiple responses through multiple different modalities. Figure 8 is a schematic diagram of an exemplary request for security monitoring. The user asks, "I have video recordings from security cameras. Can you help me identify if there is a person wearing a hat in these recordings? Furthermore, please generate an audio response for anything found. Also, please check for any anomalies in sound (such as screaming) within 30 seconds before or after the detection and save the results to disk in a file named 'anomaly.txt'."

[0049] Figure 9 is a flowchart showing the final plan generated from the request. The multiple steps S905, S907, S915, and S917 in the multiple gray boxes are multiple tools linked to multiple subtasks of the initial plan before refinement.

[0050] In step S901, an input video is received, which includes both an image stream and an audio stream that need to be processed separately. The image stream is separated in S903, and "Visual Question Answering" is performed in S905 to answer the question, "Can you find the person wearing the hat?". Although shown here as a single tool, this could involve multiple tools, such as semantic segmentation, audio-to-text conversion of the message, etc.

[0051] The video-to-image step S903 has two outputs: one arrow corresponds to the image output (a list of multiple images), and the other arrow corresponds to the second output, which is a list of multiple timestamps from the original video from which the multiple images were acquired.

[0052] The response from S905 is output as text, which is then converted to speech in S907, which is then output as audio in step S909, indicating to the user that a hat was found on the input video.

[0053] The audio from the video is processed separately in step S911. The audio stream is then cropped in step S913 using audio editing. Audio editing crops the audio file over a 60-second window centered around the time the hat was observed. The size of the window is set based on an input query from the user. The center of the window is set based on the time the hat was observed in step S905.

[0054] The cropped audio file is then processed in S915 to determine the presence of multiple anomalies, such as "screams." This output is then captured during the file writing step S917, and the file is output in S919.

[0055] The initial plan is shown in several gray boxes. Here, we can see that the visual question answering tool S905 is provided first. However, this tool is not suitable for directly processing input video. Instead, the visual question answering tool S905 requires multiple inputs, each consisting of a list of timestamps and a list of images, provided by the video-to-image tool S903.

[0056] The abnormal sound detection tool S915 requires an audio file to be input, but this input is not available from multiple tools in the original plan. Therefore, during refinement, the tool graph search provides an audio editing tool used to generate a window of audio data for which multiple abnormalities should be searched. The audio editing tool also requires an audio stream input available from tool S911.

[0057] From the above, it is clear that multiple additional tools are used to provide data in the correct format, enabling multiple complex video and audio editing tools to work together.

[0058] Figure 10 shows a schematic diagram of hardware that may be used to carry out the method according to the embodiment. The hardware comprises a computing system 1200. In this particular example, several components of the system are described together. However, it will be noted that the multiple components are not necessarily collocated.

[0059] The computing system 1200 has multiple components, including, but is not limited to, a processing unit 1213 (a central processing unit, i.e., a CPU, also known as a "processor"), system memory 1201, and a system bus 1211 that connects various system components, including the system memory 1201, to the processing unit 1213. The system bus 1211 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of the various bus architectures. The computing system 1200 may also have memory 1215 connected to the system bus 1211. Memory 1215 may be external memory.

[0060] The system memory 1201 comprises a computer storage medium in the form of volatile and / or non-volatile memory, such as read-only memory (ROM). The system memory 1201 stores a number of computer-readable instructions that, when executed by the processing unit 1213 (e.g., the processor), cause the processing unit 1213 (also called the "processor") to perform the functions described herein, in particular, in the manner discussed in relation to Figures 3-9.

[0061] The system memory 1201 may also contain an operating system 1205 used by the processing unit 1213, multiple application programs 1207, and program data 1209. In some embodiments, a Basic Input / Output System (BIOS) 1203, which includes multiple routines that help transfer information between multiple elements within the computer, such as during startup, is typically stored in the system memory 1201.

[0062] The computing system further comprises an interface 1225 that is communicatively connected to the system bus 1211. In this example, interface 1225 is particularly capable of communicating via wired and / or wireless connections to receive information from and transmit information to multiple other devices. Additionally or alternatively, interface 1225 comprises a camera input for receiving input (e.g., multiple images of a scene) and an output for providing the user with multiple results of location recognition / image retrieval. In another example, interface 1225 could be various forms of user interfaces that allow the user to interact with the system, for example, by generating multiple commands and / or multiple questions and receiving multiple responses.

[0063] In the example shown in Figure 10, a video interface 1217 is provided. The video interface 1217 includes a graphics processing unit (GPU) 1219 that is communicatively connected to a graphics processing memory 1221.

[0064] The graphics processing unit (GPU) 1219 is particularly well-suited for training multiple machine learning models, such as neural network training, due to its suitability for multiple data parallel operations. Therefore, in this embodiment, the processing for training multiple machine learning models can be divided between the CPU 1213 and the GPU 1219.

[0065] It should be noted that in some embodiments, different hardware may be used to train multiple machine learning models and perform multiple state updates. For example, training multiple machine learning models may occur on one or more local desktop or workstation computers or on multiple devices in a cloud computing system, which may include one or more separate desktop or workstation GPUs, one or more separate desktop or workstation CPUs, such as processors with a PC architecture, and a considerable amount of volatile system memory, such as 16 GB or more. On the other hand, for example, the performance of the machine learning models (e.g., inference) may use other hardware.

[0066] Figure 10 shows a specific example of hardware that may be used to implement the method described herein, but it should be recognized that this is just one example and other configurations may also be used.

[0067] While certain configurations have been described, these are presented only as examples and are not intended to limit the scope of protection. The concepts of the invention described herein can be implemented in various other forms. In addition, various omissions, substitutions, and modifications to the specific implementations described herein can be made without departing from the scope of protection as defined in the following claims.

Claims

1. A method for planning a complex task using multiple software tools from natural language input, wherein the method is: Searching a tool graph, wherein the tool graph comprises multiple nodes representing multiple tools and multiple edges between the multiple nodes indicating compatibility between the multiple tools. Receiving a natural language request from the user indicating the aforementioned complex task, Determining a subset of the multiple tools by using the aforementioned natural language request, and determining the subset of the multiple tools that is most similar to the aforementioned natural language request, An initial plan is generated using a large-scale language model (LLM) capable of reasoning, which takes the natural language request and the subset of the plurality of tools as input, wherein the initial plan comprises a sequence of subtasks, each subtask having at least one tool from the subset of the plurality of tools assigned to it. To generate the final plan from the aforementioned initial plan and Having the above-mentioned final plan is, Processing each subtask of the initial plan by determining whether the tool in the initial plan is compatible with one adjacent tool in the sequence of multiple subtasks of the initial plan, Using the aforementioned tool graph, select one or more new tools to resolve multiple incompatibility issues, Adding one or more of the aforementioned new tools to the final plan, Using the aforementioned LLM, the tool call path for the final plan is determined, The final plan is to execute the aforementioned plan and output a response to the natural language input. A method that includes [a certain feature].

2. The method according to claim 1, wherein, when generating the final plan, the tool in the initial plan is compatible with one adjacent tool in the sequence of multiple subtasks when the multiple inputs required by the tool match the multiple outputs provided by the multiple earlier tools in the sequence of multiple subtasks.

3. The method according to claim 1, wherein selecting one or more new tools using the tool graph to resolve multiple incompatibility issues comprises a first step of searching the tool graph for multiple tools that satisfy multiple compatibility requirements of the current tool, wherein the search is limited to a depth of d, where d is an integer of at least 2, and the search starts from multiple tasks in previous subtasks of the initial plan and arrives at the current tool.

4. The method according to claim 3, wherein selecting one or more new tools using the tool graph to resolve multiple incompatibility issues comprises a second step of searching the tool graph for multiple tools that satisfy the multiple compatibility requirements, the second step of starting with the current tool and using all input edges to the current tool.

5. The method according to claim 1, wherein generating the final plan comprises processing each subtask in the sequence and checking the compatibility of the multiple tools.

6. The method according to claim 5, further comprising using the LLM to generate a call path after each subtask has been processed.

7. The method according to claim 5, wherein the LLM is used to generate a call path after all subtasks have been processed.

8. The method according to claim 1, wherein determining the subset of the plurality of tools by using the natural language request and determining the subset of the plurality of tools that is most similar to the natural language request comprises comparing a plurality of embeddings of the natural language request with a plurality of embeddings of metadata of the plurality of tools in the tool graph.

9. The method according to claim 1, wherein the initial plan is generated using a reflection phase and an interleaved ReAct prompt technique.

10. The method according to claim 1, wherein the initial plan is generated without access to the tool graph.

11. The method according to claim 1, wherein the tool graph is generated offline.

12. The method according to claim 1, further comprising generating the tool graph.

13. Generating the aforementioned tool graph means The system includes receiving metadata M relating to the collection of multiple tools V, wherein the metadata comprises a description of the tool's function and a list of the tool's input and output parameters, and generating the tool graph G = (V, E, M) such that E = {(T ｉ , T ｊ )|T ｉ , T ｊ Created for ∈V, tool T ｉ The output is from tool T ｊ The method according to claim 12, wherein the data type is compatible with the aforementioned input.

14. The method according to claim 13, further comprising using the LLM to score the relevance of multiple previously found edges E based on multiple existing tool descriptions, and rejecting multiple edges that score below a threshold.

15. The method according to claim 1, further comprising receiving an image or audio file, wherein the natural language request relates to the image or audio file.

16. The method according to claim 1, wherein at least one of the plurality of subtasks comprises capturing image data or audio data.

17. The method according to claim 1, wherein at least one of the plurality of subtasks comprises the use of a neural network.

18. The method according to claim 1, wherein the output of the response comprises outputting at least one selected from an image or an audio signal.

19. The method according to claim 1, wherein at least one of the plurality of tools comprises an operation on at least one of a plurality of pixels, a plurality of voxels, a plurality of phonemes, a plurality of graphemes, and a plurality of frequencies.

20. A system for planning complex tasks using multiple software tools based on natural language input, wherein the system is Searching for a tool graph in memory, wherein the tool graph comprises multiple nodes representing multiple tools and multiple edges between the multiple nodes indicating compatibility between the multiple tools. Receiving a natural language request from the user indicating the aforementioned complex task, Determining a subset of the multiple tools by using the aforementioned natural language request, and determining the subset of the multiple tools that is most similar to the aforementioned natural language request, An initial plan is generated using a large-scale language model (LLM) capable of reasoning, which takes the natural language request and the subset of the plurality of tools as input, wherein the initial plan comprises a sequence of subtasks, each subtask having at least one tool from the subset of the plurality of tools assigned to it. To generate the final plan from the aforementioned initial plan and A processor configured to perform the following actions is provided, and generating the final plan is: The process involves determining whether the tool in the initial plan is compatible with one adjacent tool in the sequence of multiple subtasks of the initial plan, thereby processing each subtask of the initial plan. Using the aforementioned tool graph, select one or more new tools to resolve multiple incompatibility issues, Adding one or more of the aforementioned new tools to the final plan, Using the aforementioned LLM, the tool call path for the final plan is determined, The final plan is to execute the aforementioned plan and output a response to the natural language input. A system that includes these features.

Images