Tool-augmented large language model-based multi-modal processing method and apparatus

Abstract

The present disclosure relates to the technical field of artificial intelligence, and in particular to a tool-augmented large language model-based multi-modal processing method and apparatus. The method comprises: acquiring a user request to be processed, wherein said user request is used for indicating multi-modal tasks that need to be processed and multi-modal data required for processing each multi-modal task; performing task decomposition on the multi-modal tasks to obtain a task decomposition result; on the basis of the task decomposition result, performing searching in a preset tool graph to obtain all feasible solutions, and evaluating all the feasible solutions to obtain a target solution; using each target tool to execute a corresponding sub-task to obtain a tool execution result for each sub-task; and on the basis of said user request, the target solution, and each tool execution result, determining a processing result corresponding to said user request. The tool-augmented large language model-based multi-modal processing method and apparatus provided by embodiments of the present disclosure can implement efficient and accurate multi-modal processing.

Description

Multimodal Processing Method and Apparatus Based on Tool Enhancement for Large Language Models

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202410798631.6, filed on June 20, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of artificial intelligence technology, and in particular to a multimodal processing method and apparatus for large language models based on tool enhancement. Background Technology

[0004] In today's information age, Large Language Models (LLMs) play a crucial role in natural language processing and artificial intelligence. However, traditional LLMs have certain limitations, such as the inability to directly invoke external tools to perform specific multimodal tasks. To address this issue, tool-augmented LLMs have emerged, which endow LLMs with the ability to invoke tools.

[0005] While this technology has made some progress, it still has some limitations. For example, tool invocation is not accurate enough. Existing large language models often encounter difficulties when dealing with complex multimodal tasks due to the ambiguity of user requests, tool selection, and inefficient tool scheduling. Furthermore, existing large language models still face challenges in task decomposition, task planning, and tool scheduling when dealing with complex multimodal tasks. Summary of the Invention

[0006] In view of this, this disclosure proposes a multimodal processing method and apparatus based on a large language model with tool enhancement. It can search and evaluate the optimal target solution from a preset tool graph based on the task decomposition results obtained from the decomposition of multimodal tasks, and then call the target tool based on the target solution to achieve efficient and accurate multimodal processing.

[0007] According to one aspect of this disclosure, a multimodal processing method based on a tool-enhanced large language model is provided. The method utilizes a tool-enhanced large language model to process multimodal tasks. The processing includes the following steps: obtaining a user request to be processed, the user request indicating the multimodal task to be processed and the multimodal data required for each multimodal task; decomposing the multimodal task to obtain a task decomposition result, the task decomposition result including multiple sub-tasks; the multimodal data including various input data of different types; and, based on the task decomposition result, searching a preset tool graph using depth-first search or breadth-first search to obtain all feasible solutions, and evaluating all feasible solutions to obtain a target solution. The tool graph includes multiple input nodes, multiple tool nodes, multiple output nodes, and multiple directed edges, the multiple directed edges including multiple first directed edges, multiple second directed edges, and multiple third directed edges. The system uses directed edges, where each input node corresponds to a different input data type, each tool node corresponds to a different tool, and each output node corresponds to a different output data type. Input nodes and tool nodes with dependencies are connected by a first directed edge, with the direction of the first directed edge pointing from the input node to the tool node. Tool nodes and output nodes with dependencies are connected by a second directed edge, with the direction of the second directed edge pointing from the tool node to the output node. Two tool nodes with dependencies are connected by a third directed edge, with the direction of the third directed edge pointing from one tool node to the other. The target scheme indicates the target tools used in each subtask and the order in which they are used. The corresponding subtasks are executed using each target tool to obtain tool execution results for each subtask. The processing result for the user request to be processed is determined based on the user request to be processed, the target scheme, and the execution results of each tool.

[0008] In this way, by decomposing complex multimodal tasks into multiple subtasks, the processing difficulty is reduced, which is conducive to improving the efficiency of subsequent processing. By searching and evaluating in a preset tool graph, the target tools suitable for each subtask and their usage order can be automatically found, ensuring the accuracy and efficiency of tool invocation. This graph search method can take into account the dependencies between tools, avoiding the problem of task failure or inefficiency caused by improper tool selection. By using tools adapted to each subtask to execute the corresponding subtask, the needs of the actual task can be better met, improving the accuracy and reliability of the processing results of each subtask. By using a large language model to obtain user-friendly processing results, efficient and accurate multimodal processing can be achieved.

[0009] In one possible implementation, the task decomposition result further includes the correlation between the multiple subtasks; the step of searching in a preset tool graph based on the task decomposition result using a depth-first search or a breadth-first search to obtain all feasible solutions, and evaluating all feasible solutions to obtain a target solution, includes: when the correlation indicates that each subtask is independent, performing a parallel search for each subtask in the tool graph using the depth-first search or the breadth-first search to obtain all feasible solutions corresponding to each subtask; and performing a parallel evaluation of all feasible solutions for each subtask according to preset evaluation rules to obtain a target sub-solution corresponding to each subtask; wherein each target sub-solution indicates the target tools used for the corresponding subtask and the order in which the target tools are used, and the target solution includes all target sub-solutions.

[0010] This can greatly improve the system's processing efficiency.

[0011] In one possible implementation, the task decomposition result also includes the association relationship between the multiple subtasks; the step of searching in a preset tool graph to obtain all feasible solutions based on the task decomposition result using a depth-first search or a breadth-first search method includes: when the association relationship indicates that each of the subtasks is executed in a preset execution order, searching in the tool graph in the execution order using the depth-first search method or the breadth-first search method to obtain all feasible solutions.

[0012] In this way, by clearly defining the relationships and execution order between subtasks, searches can be performed according to the logical order of task processing, avoiding unnecessary search paths and thus significantly improving the efficiency and accuracy of task planning. Furthermore, the search strategy based on relationships can plan the required resources in advance, thereby achieving the rational allocation and effective utilization of resources.

[0013] In one possible implementation, the task decomposition result further includes the correlation between the multiple subtasks; the step of searching in a preset tool graph to obtain all feasible solutions based on the task decomposition result using a depth-first search or a breadth-first search method includes: when the correlation indicates that the multiple subtasks include some subtasks executed in a preset execution order and some independently executed subtasks, for the sequentially executed subtasks, searching each subtask in the tool graph according to the execution order using the depth-first search or the breadth-first search method to obtain all feasible solutions; for the independently executed subtasks, performing a parallel search for each subtask in the tool graph using the depth-first search or the breadth-first search method to obtain all feasible solutions corresponding to each subtask; wherein, all feasible solutions for the multiple subtasks include: all feasible solutions for the sequentially executed subtasks and all feasible solutions for each independently executed subtask.

[0014] In this way, by clarifying the relationships and execution order between subtasks, the search process can be executed more effectively, achieving a reasonable allocation and efficient use of resources.

[0015] In one possible implementation, the task decomposition result further includes a task description, input data type, and output data type for each subtask. The task description includes task content and task domain. The task content indicates the processing procedure of the corresponding subtask. The task domain includes one of image processing, speech processing, video processing, and text processing. The input data type indicates the type of input data processed by the corresponding subtask. The input data processed by each subtask comes from at least the multimodal data. The output data type indicates the type of output data of the corresponding subtask. Based on the task decomposition result, a search is performed in a preset tool graph using either depth-first search or breadth-first search to obtain all feasible solutions. This includes performing the following processing on each subtask: based on the input of the current subtask... The search begins by identifying input nodes of the same data type from multiple input nodes in the tool graph, and ends by identifying output nodes of the same data type from multiple output nodes in the tool graph, based on the data type of the current subtask. A search is then conducted among all tool nodes in the tool graph that are directly or indirectly connected to the current search starting point via directed edges to identify target tool nodes that match the task description of the current subtask. Each target tool node includes at least one tool node, and at least one of these target tool nodes is connected to the current search ending point via a second directed edge. Based on the current search starting point, the target tool node, and the current search ending point, all feasible solutions for the current subtask are determined.

[0016] In this way, by breaking down complex multimodal tasks into multiple subtasks, each with a clear task description, input data type, and output data type, the subsequent search target becomes clearer, allowing for the rapid identification of suitable tool nodes and improving task processing efficiency.

[0017] In one possible implementation, the step of searching all tool nodes in the tool graph that are directly or indirectly connected to the current search starting point through directed edges to determine the target tool node that matches the task description of the current subtask includes: scoring the degree of matching between all tool nodes that are directly or indirectly connected to the current search starting point through directed edges and the task description of the current subtask, and determining the target tool node that matches the task description of the current subtask based on the scoring results.

[0018] By utilizing the directed edge relationships of the tool graph for searching, the search process is ensured to be systematic and structured, avoiding the inefficiencies and high error rates that may result from random or disordered searches. This improves search efficiency, reduces unnecessary time and resource consumption, and the introduction of a scoring mechanism allows for a quantitative assessment of the matching degree between tool nodes and task descriptions, providing a clear basis for determining target tool nodes. This helps reduce the influence of subjective judgment and improves the accuracy and reliability of search results.

[0019] In one possible implementation, the method further includes a process for constructing the tool graph, the construction process comprising: determining multiple different sample tasks and the task description, input data type, and output data type of each sample task using a preset adjacency matrix or adjacency list; determining multiple input nodes based on all different input data types, determining multiple output nodes based on all different output data types, and determining multiple tool nodes based on all different tools; determining multiple directed edges based on the task description of each sample task, the multiple directed edges including multiple first directed edges, multiple second directed edges, and multiple third directed edges; and obtaining the tool graph based on the multiple input nodes, the multiple tool nodes, the multiple output nodes, all first directed edges, all second directed edges, and all third directed edges.

[0020] By representing different sample tasks, input data types, output data types, and tool nodes in a structured manner, the task representation becomes clearer and easier to understand. This representation not only improves the accuracy and efficiency of task decomposition but also provides a foundation for subsequent task search and solution determination. Furthermore, once the tool graph is constructed, task searches can be performed based on it without manually analyzing the dependencies of each tool, significantly improving the automation of task planning. In addition, the tool graph construction process is based on a pre-defined adjacency matrix or adjacency list, allowing for the easy addition of new sample tasks, input data types, output data types, and tool nodes to adapt to task requirements in different domains and scenarios.

[0021] According to another aspect of this disclosure, a multimodal processing apparatus for a large language model based on tool enhancement is provided, comprising: a task decomposition module, configured to acquire user requests to be processed, wherein the user requests indicate multimodal tasks to be processed and multimodal data required for processing each multimodal task; decomposing the multimodal tasks to obtain task decomposition results, wherein the task decomposition results include multiple sub-tasks; and the multimodal data includes various types of input data, the different types of input data being different. A task planning module, configured to, based on the task decomposition results, search a preset tool graph to obtain all feasible solutions using a depth-first search or breadth-first search approach, and evaluate all feasible solutions to obtain a target solution. The tool graph includes multiple input nodes, multiple tool nodes, multiple output nodes, and multiple directed edges, wherein the multiple directed edges include multiple first directed edges, multiple second directed edges, and multiple third directed edges, and each input node corresponds to a different input. The data types are defined as follows: each tool node corresponds to a different tool, and each output node corresponds to a different output data type. Input nodes and tool nodes with dependencies are connected by a first directed edge, with the direction of the first directed edge pointing from the input node to the tool node. Tool nodes and output nodes with dependencies are connected by a second directed edge, with the direction of the second directed edge pointing from the tool node to the output node. Two tool nodes with dependencies are connected by a third directed edge, with the direction of the third directed edge pointing from one tool node to the other. The target scheme indicates the target tools used in each subtask and the order in which the target tools are used. A task execution module is used to execute the corresponding subtask using each target tool to obtain the tool execution result for each subtask. A result determination module is used to determine the processing result corresponding to the user request to be processed based on the user request to be processed, the target scheme, and the tool execution results.

[0022] In this way, by decomposing complex multimodal tasks into multiple subtasks, the processing difficulty is reduced, which is conducive to improving the efficiency of subsequent processing. By searching and evaluating in a preset tool graph, the target tools suitable for each subtask and their usage order can be automatically found, ensuring the accuracy and efficiency of tool invocation. This graph-based search method can take into account the dependencies between tools and avoid the problem of task failure or inefficiency caused by improper tool selection. By using tools adapted to each subtask to execute the corresponding subtask, the needs of the actual task can be better met, and the accuracy and reliability of the processing results of each subtask can be improved. By using a large language model, user-friendly processing results are obtained, thereby achieving efficient and accurate multimodal processing.

[0023] In one possible implementation, the task decomposition result further includes the correlation between the multiple subtasks; the step of searching in a preset tool graph based on the task decomposition result using a depth-first search or a breadth-first search to obtain all feasible solutions, and evaluating all feasible solutions to obtain a target solution, includes: when the correlation indicates that each subtask is independent, performing a parallel search for each subtask in the tool graph using the depth-first search or the breadth-first search to obtain all feasible solutions corresponding to each subtask; and performing a parallel evaluation of all feasible solutions for each subtask according to preset evaluation rules to obtain a target sub-solution corresponding to each subtask; wherein each target sub-solution indicates the target tools used for the corresponding subtask and the order in which the target tools are used, and the target solution includes all target sub-solutions.

[0024] This can greatly improve the system's processing efficiency.

[0025] In one possible implementation, the task decomposition result also includes the association relationship between the multiple subtasks; the step of searching in a preset tool graph to obtain all feasible solutions based on the task decomposition result using a depth-first search or a breadth-first search method includes: when the association relationship indicates that each of the subtasks is executed in a preset execution order, searching in the tool graph in the execution order using the depth-first search method or the breadth-first search method to obtain all feasible solutions.

[0026] In this way, by clearly defining the relationships and execution order between subtasks, searches can be performed according to the logical order of task processing, avoiding unnecessary search paths and thus significantly improving the efficiency and accuracy of task planning. Furthermore, the search strategy based on relationships can plan the required resources in advance, thereby achieving the rational allocation and effective utilization of resources.

[0027] In one possible implementation, the task decomposition result further includes the correlation between the multiple subtasks; the step of searching in a preset tool graph to obtain all feasible solutions based on the task decomposition result using a depth-first search or a breadth-first search method includes: when the correlation indicates that the multiple subtasks include some subtasks executed in a preset execution order and some independently executed subtasks, for the sequentially executed subtasks, searching each subtask in the tool graph according to the execution order using the depth-first search or the breadth-first search method to obtain all feasible solutions; for the independently executed subtasks, performing a parallel search for each subtask in the tool graph using the depth-first search or the breadth-first search method to obtain all feasible solutions corresponding to each subtask; wherein, all feasible solutions for the multiple subtasks include: all feasible solutions for the sequentially executed subtasks and all feasible solutions for each independently executed subtask.

[0028] In this way, by clarifying the relationships and execution order between subtasks, the search process can be executed more effectively, achieving a reasonable allocation and efficient use of resources.

[0029] In one possible implementation, the task decomposition result further includes a task description, input data type, and output data type for each subtask. The task description includes task content and task domain. The task content indicates the processing procedure of the corresponding subtask. The task domain includes one of image processing, speech processing, video processing, and text processing. The input data type indicates the type of input data processed by the corresponding subtask. The input data processed by each subtask comes from at least the multimodal data. The output data type indicates the type of output data of the corresponding subtask. Based on the task decomposition result, a search is performed in a preset tool graph using either depth-first search or breadth-first search to obtain all feasible solutions. This includes performing the following processing on each subtask: based on the input of the current subtask... The search begins by identifying input nodes of the same data type from multiple input nodes in the tool graph, and ends by identifying output nodes of the same data type from multiple output nodes in the tool graph, based on the data type of the current subtask. A search is then conducted among all tool nodes in the tool graph that are directly or indirectly connected to the current search starting point via directed edges to identify target tool nodes that match the task description of the current subtask. Each target tool node includes at least one tool node, and at least one of these target tool nodes is connected to the current search ending point via a second directed edge. Based on the current search starting point, the target tool node, and the current search ending point, all feasible solutions for the current subtask are determined.

[0030] In this way, by breaking down complex multimodal tasks into multiple subtasks, each with a clear task description, input data type, and output data type, the subsequent search target becomes clearer, allowing for the rapid identification of suitable tool nodes and improving task processing efficiency.

[0031] In one possible implementation, the step of searching all tool nodes in the tool graph that are directly or indirectly connected to the current search starting point through directed edges to determine the target tool node that matches the task description of the current subtask includes: scoring the degree of matching between all tool nodes that are directly or indirectly connected to the current search starting point through directed edges and the task description of the current subtask, and determining the target tool node that matches the task description of the current subtask based on the scoring results.

[0032] By utilizing the directed edge relationships of the tool graph for searching, the search process is ensured to be systematic and structured, avoiding the inefficiencies and high error rates that may result from random or disordered searches. This improves search efficiency, reduces unnecessary time and resource consumption, and the introduction of a scoring mechanism allows for a quantitative assessment of the matching degree between tool nodes and task descriptions, providing a clear basis for determining target tool nodes. This helps reduce the influence of subjective judgment and improves the accuracy and reliability of search results.

[0033] In one possible implementation, the apparatus further includes a construction module for constructing the tool graph. The construction process includes: determining multiple different sample tasks and their task descriptions, input data types, and output data types using a preset adjacency matrix or adjacency list; determining multiple input nodes based on all different input data types, multiple output nodes based on all different output data types, and multiple tool nodes based on all different tools; determining multiple directed edges based on the task descriptions of each sample task, the multiple directed edges including multiple first directed edges, multiple second directed edges, and multiple third directed edges; and obtaining the tool graph based on the multiple input nodes, the multiple tool nodes, the multiple output nodes, all first directed edges, all second directed edges, and all third directed edges.

[0034] By representing different sample tasks, input data types, output data types, and tool nodes in a structured manner, the task representation becomes clearer and easier to understand. This representation not only improves the accuracy and efficiency of task decomposition but also provides a foundation for subsequent task search and solution determination. Furthermore, once the tool graph is constructed, task searches can be performed based on it without manually analyzing the dependencies of each tool, significantly improving the automation of task planning. In addition, the tool graph construction process is based on a pre-defined adjacency matrix or adjacency list, allowing for the easy addition of new sample tasks, input data types, output data types, and tool nodes to adapt to task requirements in different domains and scenarios.

[0035] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing instructions stored in the memory.

[0036] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0038] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0039] Figures 1 and 2 illustrate schematic diagrams of a multimodal processing method for a large language model based on tool enhancement provided according to embodiments of the present disclosure.

[0040] Figures 3 and 4 show block diagrams of a multimodal processing apparatus for a large language model based on tool enhancement provided according to embodiments of the present disclosure. Detailed Implementation

[0041] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0042] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0043] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0044] To facilitate understanding of the technical solutions provided by the embodiments of this disclosure by those skilled in the art, the technical environment for implementing the technical solutions will be described below.

[0045] Large language models have demonstrated powerful generalization capabilities in various natural language processing tasks such as machine translation, intelligent dialogue, text generation, and code development. Tool-enhanced large language models combine large language models with external tools. The main idea is to embed external tools into the large language model to enhance its capabilities and flexibility. Tool-enhanced large language models can be implemented in various ways, one common approach being to design appropriate interfaces that allow the large language model to directly call the interfaces of external tools to obtain necessary information or perform specific tasks.

[0046] Specifically, the workflow of a tool-enhanced large language model can include: the large language model receives input data; based on the content of the input data, the large language model determines whether to call external tools; if external tools are needed, the large language model communicates with the external tools using predefined interfaces to obtain the required information or perform specific tasks using the tools; and the large language model integrates the results returned by the external tools into its own output to complete the entire processing. Because it may call external tools, the tool-enhanced large language model can be viewed as a tool-enhanced language model framework.

[0047] Based on this, many pre-known tools for multimodal understanding and processing sample tasks can be integrated into the system to develop a multimodal agent. This agent uses the technology of tool-enhanced large language models and has the ability to process multimodal data. This means that the multimodal agent can not only process text data, but also other forms of data, such as images and audio, thereby achieving more comprehensive and intelligent human-computer interaction.

[0048] While current tool-enhanced large language model techniques have made some progress in handling multimodal tasks, several limitations remain. These limitations include: a low success rate for external tool invocation, restricting the feasibility of practical application; limited multimodal data processing capabilities, particularly in understanding and processing non-textual data such as images and audio; task planning constrained by the input token length limit of the language model; and difficulty in forming complex thought networks, especially in task planning, to explore the tool invocation scenarios required to solve complex multimodal tasks. Therefore, tool-enhanced large language model techniques are of great significance for achieving more flexible and intelligent human-computer interaction and multimodal data processing.

[0049] To address the aforementioned technical problems, this disclosure provides a multimodal processing method based on a tool-enhanced large language model. This method utilizes a tool-enhanced large language model to process multimodal tasks. The processing includes acquiring a user request to be processed, which indicates the multimodal task to be processed and the multimodal data required for each task. The multimodal task is then decomposed to obtain a task decomposition result, which includes multiple sub-tasks. The multimodal data includes various types of input data. This decomposition of complex multimodal tasks into multiple sub-tasks reduces processing difficulty and is advantageous. To improve subsequent processing efficiency, based on the task decomposition results, a search is performed in a pre-defined tool graph using either depth-first search or breadth-first search to obtain all feasible solutions. All feasible solutions are then evaluated to obtain the target solution. The tool graph includes multiple input nodes, multiple tool nodes, multiple output nodes, and multiple directed edges. These directed edges include multiple first directed edges, multiple second directed edges, and multiple third directed edges. Each input node corresponds to a different input data type, each tool node corresponds to a different tool, and each output node corresponds to a different output data type. Input nodes and tool nodes with dependencies are connected via first directed edges. Furthermore, the direction of the first directed edge is from the input node to the tool node. Tool nodes with dependencies are connected to the output node via a second directed edge, with the direction of the second directed edge also pointing from the tool node to the output node. Two tool nodes with dependencies are connected via a third directed edge, with the direction of the third directed edge pointing from one tool node to the other. The target solution indicates the target tools used in each subtask and the order in which they are used. Thus, by searching and evaluating within a pre-defined tool graph, the appropriate target tools and their usage order for each subtask can be automatically found, ensuring the accuracy and efficiency of tool invocation. This graph search-based method takes into account the dependencies between tools, avoiding task failures or inefficiencies caused by inappropriate tool selection. It utilizes target tools to execute corresponding subtasks, obtaining tool execution results for each subtask. By using tools adapted to each subtask, it better meets the needs of the actual task, improving the accuracy and reliability of the subtask processing results. Based on the user request to be processed, the target solution, and the execution results of each tool, it determines the processing result for the corresponding user request. This leverages a large language model to obtain user-friendly processing results, thereby achieving efficient and accurate multimodal processing.

[0050] Figures 1 and 2 illustrate schematic diagrams of a multimodal processing method for a large language model based on tool enhancement according to embodiments of the present disclosure. The following description, in conjunction with Figures 1 and 2, provides an illustrative explanation of the multimodal processing method for a large language model based on tool enhancement according to embodiments of the present disclosure.

[0051] As shown in Figure 1, the multimodal processing method uses a tool-enhanced large language model (hereinafter referred to as the large language model) to process multimodal tasks. The processing process may include four consecutive stages: task decomposition stage, task planning stage, solution execution stage, and response generation stage. Corresponding to these four consecutive stages, the processing may include the following steps S101 to S104.

[0052] S101. Obtain user requests to be processed. User requests to be processed indicate the multimodal tasks that need to be processed and the multimodal data required for processing each multimodal task. Decompose the multimodal tasks to obtain the task decomposition results.

[0053] The user requests to be processed serve as input to the large language model that enhances the tool, enabling the large language model to extract multimodal tasks and multimodal data from these requests. Multimodal data includes various types of input data. The task decomposition results in multiple sub-tasks. For example, from a user request to "generate a new image similar to image A.png and describe this image in detail, and predict tomorrow's weather," the multimodal tasks can be identified as image generation, image description, and weather prediction. The multimodal data includes two types of input data: images for image generation and image description, and time and location for weather prediction. This results in three sub-tasks: image generation, image description, and weather prediction.

[0054] Step S101 corresponds to the task decomposition stage, which aims to break down the user requests to be processed into a series of sub-tasks, reducing the difficulty of subsequent task planning. In the task decomposition stage, pre-trained large language models such as ChatGPT or instruction-tuning LLaMA models can be used to automatically decompose the user requests to be processed into a list of sub-tasks, without needing to call external tools, thereby reducing processing difficulty and improving subsequent processing efficiency. The task decomposition results can be represented in JSON format.

[0055] S102. Based on the task decomposition results, search the preset tool graph using either depth-first search or breadth-first search to obtain all feasible solutions, and evaluate all feasible solutions to obtain the target solution.

[0056] The tool graph can be pre-constructed using an adjacency matrix or adjacency list, as detailed below. The tool graph illustrates the inherent relationships between tools, serving as a fundamental basis for analyzing and optimizing interactions between tools. Each tool shown corresponds to a specific function. The functions performed by each tool may be different, or there may be two or more tools performing the same function.

[0057] The tool graph comprises multiple nodes and multiple directed edges. The nodes include multiple input nodes, multiple tool nodes, and multiple output nodes. The directed edges include multiple first directed edges, multiple second directed edges, and multiple third directed edges. Each input node corresponds to a different input data type; for example, in the tool graph, there is input node 1 corresponding to image type and input node 2 corresponding to text type. Each tool node corresponds to a different tool; for example, in the tool graph, there is tool node 1 corresponding to tool 1 and tool node 2 corresponding to tool 2. Each output node corresponds to a different output data type; for example, in the tool graph, there is output node 1 corresponding to text type and output node 2 corresponding to image type. Nodes and tools with dependencies are connected by directed edges, where a dependency means that one of the connected nodes is either an input or an output of the other. Input nodes and tool nodes with dependencies are connected by a first directed edge, with the direction of the first directed edge pointing from the input node to the tool node. Tool nodes and output nodes with dependencies are connected by a second directed edge, with the direction of the second directed edge pointing from the tool node to the output node. Two tool nodes with dependencies are connected by a third directed edge, with the direction of the third directed edge pointing from one tool node to the other. For example, for an image description task, the processing includes processing image-type input data to obtain text-type output data. This processing can be implemented by using tool 1 first and then tool 2. In other words, the tool... The input of tool 1 is image-type input data, the output of tool 1 is the input of tool 2, and the output of tool 2 is text-type output data. Based on this, the input node 1 corresponding to the image type is connected to the tool node 1 corresponding to tool 1 through the first directed edge with a solid line and arrow in Figure 2. The tool node 1 is connected to the tool node 2 corresponding to tool 2 through the third directed edge with a dashed line and arrow in Figure 2. The tool node 2 is connected to the output node 1 corresponding to the text type through the second directed edge with a double-dotted line and arrow in Figure 2. This process can also be achieved by using tool 1, tool 2, tool 3, and tool 4 in sequence. The connection between each node and tool is the same as the previous method, which will not be repeated here.

[0058] Step S102 corresponds to the task planning stage, which aims to find feasible solutions in the tool graph using a graph-based thinking paradigm and determine the most suitable target solution. This graph-based thinking paradigm takes into account the dependencies between tools, avoiding task failures or inefficiencies caused by inappropriate tool selection, and helps to achieve efficient and accurate task planning.

[0059] During the task planning phase, a depth-first search (DFS) approach can be used to search the tool graph. Taking Figure 2 as an example, starting from input node 1, a node connected to (or adjacent to) the starting point, such as tool node 1, is visited and marked as visited. Then, an unvisited node connected to tool node 1, such as tool node 2, is visited and marked as visited. Next, an unvisited node connected to tool node 2, such as output node 1, is visited and marked as visited. Since no other node is connected to output node 1, the process backtracks to the previous node, tool node 2, and visits the remaining unvisited node connected to tool node 2, such as tool node 3, marking it as visited. This process continues until all nodes are visited or the desired solution is found. For a detailed implementation of DFS, please refer to relevant technologies; this will not be elaborated upon here.

[0060] Alternatively, during the task planning phase, a breadth-first search (BFS) approach can be used to search the tool graph. Taking Figure 2 as an example, starting with input node 1, all nodes connected to input node 1 (tool node 1 and tool node 5) are retrieved. After traversing these two nodes, all nodes connected to tool node 1 are retrieved and traversed in the same way. This process continues until all nodes have been visited or the desired solution is found. It should be noted that since not all tools shown in the tool graph can solve the multimodal tasks in the user request, a large language model can be used for auxiliary searching. This involves scoring the tools corresponding to each tool node involved in each search step and filtering out tools irrelevant to the multiple subtasks using a preset search strategy. Simultaneously, it determines which tool nodes to visit during the search on the tool graph, thus finding a balance between time and space complexity, improving search efficiency while finding the most suitable target solution.

[0061] Based on the task breakdown results, a search is performed in the tool graph to obtain all feasible solutions, with one or more feasible solutions. If multiple feasible solutions are found, a large language model can be used to score each solution. The scoring criteria can be set according to actual needs (e.g., computational complexity or computational speed). The feasible solution with the highest score is selected as the target solution. If the user is not satisfied with the processing result of the corresponding user request based on the target solution, the second-highest-scoring feasible solution can be selected as the new target solution to return a new processing result to the user. The target solution indicates the target tools used in each subtask and the order in which they are used. This ensures the accuracy and efficiency of tool invocation by sequentially calling the target tools applicable to each subtask in the large language model. For example, for the subtask of image description, two feasible paths are found in the tool graph. The first path is composed of input node 1, tool node 1, tool node 2, and output node 1 in Figure 2. The second path is composed of input node 1, tool node 1, tool node 2, tool node 3, tool node 4, and output node 1 in Figure 2. Each feasible path represents a feasible solution. The feasible solution represented by the first feasible path is to use tool 1 first and then tool 2. The feasible solution represented by the second feasible path is to use tool 1, tool 2, tool 3, and tool 4 in sequence. The target solution can then be determined according to the actual needs.

[0062] S103. Execute the corresponding sub-tasks using each target tool to obtain the tool execution results for each sub-task.

[0063] Taking the first feasible solution for the subtask of image description (using tool 1 first and then tool 2) as an example, the large language model can first call tool 1 to process the input data and obtain the output data of tool 1. Then, the output data of tool 1 is used as the input data of tool 2, and tool 2 is called to process it to obtain the tool execution result for the subtask of image description. The other subtasks are similar in this way.

[0064] Step S103 corresponds to the solution execution phase, which aims to execute each subtask based on the target solution determined in the task planning phase. By using tools adapted to each subtask to execute the corresponding subtask, the needs of the actual task can be better met, improving the accuracy and reliability of the subtask processing results. The solution execution phase involves parsing and executing the target solution, as well as scheduling and executing actions. The system's execution engine can first parse the target solution into a series of actions, each action associated with a specific target tool service. This can be achieved through manually created mapping tables or automatic schedulers based on certain strategies, thus giving the system flexibility to arrange various tools according to user preferences. The parsed actions are then scheduled for execution on local, remote, or hybrid endpoints by a scheduler. Multiple independent subtasks can be executed in parallel to improve efficiency. Furthermore, a state memory can be maintained in the system to store all intermediate results, including their values ​​and types, allowing for automatic runtime correction of action parameters.

[0065] S104. Determine the processing result of the corresponding user request based on the user request to be processed, the target solution, and the execution results of each tool.

[0066] Step S104 corresponds to the response generation stage, which aims to use the large language model to summarize the tool execution results for each subtask according to the target scheme, and generate a user-friendly response result, i.e., the processing result, based on the user request to be processed.

[0067] During the response generation phase, large language models such as ChatGPT, upon receiving the user request, the target solution, and the execution results of various tools, will understand and analyze them, and intelligently generate corresponding answers. In the process of generating answers, the large language model comprehensively considers the content of the user request, the target solution, and the execution results of each tool, thereby ensuring that the generated response or answer is comprehensive, clear, and easy to understand, while also being user-friendly.

[0068] The generated responses can include summaries of the tool execution results, extraction of key information, explanations of the tool execution results, and suggestions. The large language model can understand the user's intent based on context and output appropriate responses in natural language, helping the user better understand the execution results and take further action. Thus, by leveraging the powerful language understanding and generation capabilities of the large language model, the execution results of various tools can be transformed into user-friendly responses, thereby improving the system's interactive experience, usability, flexibility, and scalability.

[0069] In one possible implementation, the multimodal processing method may also include a tool graph construction process, which may include the following four steps.

[0070] First, multiple different sample tasks are determined using a pre-defined adjacency matrix or adjacency list, along with their task descriptions, input data types, and output data types. The number and types of sample tasks can be flexibly set according to actual needs. The task description includes the task content and task domain of the corresponding sample task. The task content indicates the processing procedure of the corresponding sample task, i.e., a brief summary of the sample task. The task domain of the corresponding sample task includes one of the following: image processing, speech processing, video processing, and text processing. Image processing may include image generation, image editing, image description, and visual processing; video processing may include video generation and video understanding; speech processing may include speech generation and speech conversion; and text processing may include text translation and text generation. The input data type indicates the type of input data processed by the corresponding sample task, and the output data type indicates the type of output data of the corresponding sample task. The type of input data or output data can be image, mask, video, speech, text, etc.

[0071] Secondly, multiple input nodes are determined based on all different input data types, multiple output nodes are determined based on all different output data types, and multiple tool nodes are determined based on all different tools. For example, if there are three different input data types, three input nodes are determined to represent the three different input data types. The same applies to output nodes and tool nodes, which will not be elaborated further here.

[0072] Next, multiple directed edges are determined based on the task descriptions of each sample task. These multiple directed edges include multiple first directed edges, multiple second directed edges, and multiple third directed edges. For example, for a sample task describing an image, the task description includes processing input data of type image to obtain output data of type text, and the processing procedure is to use tool 1 first and then tool 2. Based on this, three directed edges can be determined: one first directed edge, one second directed edge, and one third directed edge. The first directed edge is used to connect the corresponding input node of type image with the corresponding tool node of tool 1, and the other directed edges are determined in the same way.

[0073] Finally, the tool graph is obtained based on multiple input nodes, multiple tool nodes, multiple output nodes, all first directed edges, all second directed edges, and all third directed edges. Taking the image description sample task as an example, as shown in Figure 2, the first directed edges with arrows (solid lines in Figure 2) connect the input node 1 of the corresponding image type with the tool node 1 of the corresponding tool 1. The third directed edges with arrows (dashed lines in Figure 2) connect tool node 1 with the tool node 2 of the corresponding tool 2. The second directed edges with arrows (double-dotted lines in Figure 2) connect tool node 2 with the output node 1 of the corresponding text type. The processing for other sample tasks is similar, ultimately yielding the tool graph.

[0074] By representing different sample tasks, input data types, output data types, and tool nodes in a structured manner, the task representation becomes clearer and easier to understand. This representation not only improves the accuracy and efficiency of task decomposition but also provides a foundation for subsequent task search and solution determination. Furthermore, once the tool graph is constructed, task searches can be performed based on it without manually analyzing the dependencies of each tool, significantly improving the automation of task planning. In addition, the tool graph construction process is based on a pre-defined adjacency matrix or adjacency list, allowing for the easy addition of new sample tasks, input data types, output data types, and tool nodes to adapt to task requirements in different domains and scenarios.

[0075] When constructing the tool graph, it's crucial to acquire as many sample tasks and feasible solutions as possible for the same task, incorporating them all into the graph. This ensures that the most suitable feasible solution, i.e., the target solution, can be selected when addressing actual user requests. Because of this, the constructed tool graph often contains tasks unrelated to the subtasks in the task decomposition results. Therefore, during the task planning phase, when searching the tool graph using depth-first search or breadth-first search, a large language model can be used to score the tools corresponding to the tool nodes, and a pre-defined search strategy can be embedded into the depth-first search or breadth-first search to obtain the target solution.

[0076] Search strategies can include one of the following: greedy search, bundle search, adaptive search, and heuristic search. In a greedy search strategy, the node with the highest score at each step is selected, where the score represents the relevance of the tool to the subtask. Greedy search is fast and simple, but may not find the optimal solution or even any solution. In a bundle search strategy, only the top-ranked tools based on their scores are retained. Bundle search can expand the search space but slightly reduces search efficiency. In an adaptive strategy, the bundle size is dynamically adjusted by selecting tools with scores above a fixed threshold, representing a trade-off between exploration and exploitation. In a heuristic search strategy, an evaluation function f(n) = g(n) + h(n) is designed, where g(n) is the actual cost from the starting node to node n, and h(n) is the estimated cost from node n to the ending node. During the search based on the heuristic strategy, each time the next node is visited, the actual cost and the estimated cost are weighed, favoring the node n with the smallest f(n). The scores used in the search strategy can be determined by a large language model. The specific determination method is set according to actual needs, and this disclosure does not limit this.

[0077] In addition to multiple subtasks, the task breakdown results may also include a task description, input data type, and output data type for each subtask. The task description includes the task content and task domain of the corresponding subtask. The task content indicates the processing procedure of the corresponding subtask, i.e., a brief summary of the subtask, which can guide the process of finding the target solution using the graph thinking paradigm. The task domain of the corresponding subtask is similar to that of the corresponding sample task, and will not be elaborated further in this paper. The task domain of the corresponding subtask can narrow the search scope in the process of finding the target solution using the graph thinking paradigm, so as to find the most relevant and suitable tools for each subtask. The input data type indicates the type of input data processed by the corresponding subtask. The output data type indicates the type of output data of the corresponding subtask. The input data processed by each subtask can come from multimodal data and the output data obtained after performing one or more subtasks based on multimodal data.

[0078] In one possible implementation, step S102, based on the task decomposition results, searches the preset tool graph to obtain all feasible solutions using either depth-first search or breadth-first search. This may include performing the following processing for each subtask: Based on the input data type of the current subtask, input nodes of the same input data type determined from multiple input nodes in the tool graph are used as the current search starting point; based on the output data type of the current subtask, output nodes of the same output data type determined from multiple output nodes in the tool graph are used as the current search ending point; searching all tool nodes in the tool graph directly or indirectly connected to the current search starting point via directed edges to determine target tool nodes that match the task description of the current subtask, where each target tool node includes at least one tool node connected to the current search ending point via a second directed edge; and determining all feasible solutions for the current subtask based on the current search starting point, target tool nodes, and current search ending point. In this way, by decomposing the complex multimodal task into multiple subtasks, each with a clear task description, input data type, and output data type, the subsequent search targets become clearer, allowing for faster identification of suitable tool nodes and improving task processing efficiency.

[0079] In one possible implementation, the search is performed on all tool nodes in the tool graph that are directly or indirectly connected to the current search starting point through directed edges to determine the target tool node that matches the task description of the current subtask. This may include: scoring the degree of matching between all tool nodes that are directly or indirectly connected to the current search starting point through directed edges and the task description of the current subtask, and determining the target tool node that matches the task description of the current subtask based on the scoring results.

[0080] Taking the current subtask of image description as an example, based on the input data type of image (image) and the output data type of text for this current subtask, we determine the input nodes corresponding to images (e.g., input node 1 in Figure 2) and the output nodes corresponding to text (e.g., output node 2) from the tool graph. Thus, we take input node 1 as the current search starting point and output node 2 as the current search ending point. As shown in Figure 2, all tool nodes directly or indirectly connected to the current search starting point (output node 2) through directed edges are tool node 1, tool node 2, tool node 3, and tool node 4. We score the degree of matching between these tool nodes and the task description of image description, obtaining the score for each tool node. According to a preset search strategy, such as a greedy search strategy, we determine the tool node with the highest score (tool node 5) as the target tool node. Thus, based on the current search starting point (output node 2), the target tool node (tool node 5), and the current search ending point (output node 2), we can determine the feasible solution for the current subtask image description. That is, the target tool used in the current subtask is only tool 5 corresponding to tool node 5.

[0081] By utilizing the directed edge relationships of the tool graph for searching, the search process is ensured to be systematic and structured, avoiding the inefficiencies and high error rates that may result from random or disordered searches. This improves search efficiency, reduces unnecessary time and resource consumption, and the introduction of a scoring mechanism allows for a quantitative assessment of the matching degree between tool nodes and task descriptions, providing a clear basis for determining target tool nodes. This helps reduce the influence of subjective judgment and improves the accuracy and reliability of search results.

[0082] The task decomposition results may also include the relationships between multiple subtasks. These relationships can be understood as whether there is any data exchange between the subtasks. For example, if multiple subtasks are image enhancement and weather forecasting, then these two subtasks are unrelated, meaning they are independent. In one possible implementation, S102 may include: when the relationships indicate that the subtasks are independent, performing a parallel search for each subtask in the tool graph based on a depth-first search or a breadth-first search to obtain all feasible solutions for each subtask; and performing a parallel evaluation of all feasible solutions for each subtask according to preset evaluation rules to obtain the target sub-solution for each subtask; wherein each target sub-solution indicates the target tools used in the corresponding subtask and the order in which the target tools are used, and the target solution includes all target sub-solutions. In other words, if there are no dependencies between multiple subtasks, the target solution can be searched in parallel and each subtask can be executed in parallel, which can greatly improve the system processing efficiency.

[0083] For example, if multiple subtasks are image generation and image description, and the image to be described in the image description subtask is determined by the image generation subtask, then there is a relationship between these two subtasks. That is, the image generation subtask must be executed first, and then the image description subtask must be executed, because the input of the latter subtask depends on the output of the former subtask. In one possible implementation, the step in S102, based on the task decomposition results, searching in a preset tool graph to obtain all feasible solutions using either depth-first search or breadth-first search, may include: when the relationship indicates that each subtask is executed in a preset execution order, searching each subtask in the tool graph sequentially according to the execution order using either depth-first search or breadth-first search to obtain all feasible solutions. In this way, by clearly defining the relationship and execution order between subtasks, the search can be performed according to the logical order of task processing, avoiding unnecessary search paths, thereby significantly improving the efficiency and accuracy of task planning. Furthermore, the search strategy based on relationships can plan the required resources in advance, thereby achieving reasonable allocation and effective utilization of resources.

[0084] For example, if multiple subtasks include both independent subtasks and subtasks that need to be executed sequentially, corresponding search strategies can be used for each. In one possible implementation, the step in S102, based on the task decomposition results, searches the preset tool graph to obtain all feasible solutions using either depth-first search or breadth-first search. This can include: when the relationship indicates that multiple subtasks include some subtasks executed in a preset execution order and some independently executed subtasks, for the sequentially executed subtasks, searching the tool graph sequentially according to the execution order using either depth-first search or breadth-first search to obtain all feasible solutions; for the independently executed subtasks, performing parallel searches for each subtask in the tool graph using either depth-first search or breadth-first search to obtain all feasible solutions for each subtask; wherein, all feasible solutions for multiple subtasks include: all feasible solutions for the sequentially executed subtasks and all feasible solutions for each independently executed subtask. In this way, by clarifying the relationships and execution order between subtasks, the search process can be executed more effectively, achieving a reasonable allocation and efficient use of resources.

[0085] In the multimodal processing methods provided in this disclosure, the choice of a large language model is crucial to the overall performance of the system. Two different options are offered: using a pre-trained large language model and fine-tuning a large language model. The preferred option is to use a pre-trained large language model, such as ChatGPT or Llama2. These models have been pre-trained on large-scale text corpora and can handle various natural language processing tasks. By carefully designing a series of cues to guide the processes of task decomposition, task planning, especially tool evaluation and solution execution, the work of training a large language model from scratch can be avoided. However, since pre-trained large language models are not trained for real-world needs, they may lead to poor performance. Another option is to fine-tune a large language model, such as Llama2. A self-guided approach can be used, which has the advantage of achieving high performance of the large language model by adapting it to the data and task. However, it requires a large amount of graphics processing units (GPUs) to train the large language model and may encounter overfitting problems. If sufficient computational resources and training time are available, and optimal performance is desired, then fine-tuning the model may be a good choice. However, if time and resources are limited, or if only a fast solution is needed, then using a pre-trained model may be more appropriate. The choice of model depends on the specific application scenario and available resources, and this disclosure does not limit this choice.

[0086] The multimodal processing method based on a tool-enhanced large language model provided in this disclosure aims to promote the development of human-computer interaction by utilizing a tool-enhanced large language model capable of multimodal interaction. It can interact with various modal data through tools, handle various types of tasks, and achieve efficient task execution through automated processes. It features flexibility, high applicability, and high performance, and has made significant progress in tool invocation. In particular, when handling complex multimodal tasks, it searches for the optimal path, i.e., the target solution, through a pre-built tool graph, avoiding the token restriction problem in task planning in related technologies, and can easily expand the toolbox.

[0087] This disclosure also provides a multimodal processing apparatus for a large language model based on tool enhancement. Figure 3 shows a block diagram of the multimodal processing apparatus for a large language model based on tool enhancement provided according to an embodiment of this disclosure. As shown in Figure 3, the multimodal processing apparatus 300 may include the following modules: task decomposition module 301, task planning module 302, task execution module 303, and result determination module 304.

[0088] The task decomposition module 301 is used to obtain user requests to be processed. The user requests to be processed indicate the multimodal tasks to be processed and the multimodal data required for processing each multimodal task. The multimodal tasks are decomposed to obtain task decomposition results. The task decomposition results include multiple sub-tasks. The multimodal data includes multiple types of input data. Different types of input data have different types.

[0089] The task planning module 302 is used to search for all feasible solutions in a preset tool graph based on the task decomposition results, using either depth-first search or breadth-first search, and to evaluate all feasible solutions to obtain a target solution. The tool graph includes multiple input nodes, multiple tool nodes, multiple output nodes, and multiple directed edges. The multiple directed edges include multiple first directed edges, multiple second directed edges, and multiple third directed edges. Each input node corresponds to a different input data type, each tool node corresponds to a different tool, and each output node corresponds to a different output data type. Input nodes and tool nodes with dependencies are connected by the first directed edge, with the direction of the first directed edge pointing from the input node to the tool node. Tool nodes and output nodes with dependencies are connected by the second directed edge, with the direction of the second directed edge pointing from the tool node to the output node. Two tool nodes with dependencies are connected by the third directed edge, with the direction of the third directed edge pointing from one tool node to the other. The target solution indicates the target tools used in each subtask and the order in which the target tools are used.

[0090] The task execution module 303 is used to execute corresponding sub-tasks using each of the target tools to obtain tool execution results for each of the sub-tasks.

[0091] The result determination module 304 is used to determine the processing result corresponding to the user request to be processed based on the user request to be processed, the target scheme, and the execution results of each tool.

[0092] In this way, by decomposing complex multimodal tasks into multiple subtasks, the processing difficulty is reduced, which is conducive to improving the efficiency of subsequent processing. By searching and evaluating in a preset tool graph, the target tools suitable for each subtask and their usage order can be automatically found, ensuring the accuracy and efficiency of tool invocation. This graph-based search method can take into account the dependencies between tools and avoid the problem of task failure or inefficiency caused by improper tool selection. By using tools adapted to each subtask to execute the corresponding subtask, the needs of the actual task can be better met, and the accuracy and reliability of the processing results of each subtask can be improved. By using a large language model, user-friendly processing results are obtained, thereby achieving efficient and accurate multimodal processing.

[0093] In one possible implementation, the task decomposition result further includes the correlation between the multiple subtasks; the step of searching in a preset tool graph based on the task decomposition result using a depth-first search or a breadth-first search to obtain all feasible solutions, and evaluating all feasible solutions to obtain a target solution, includes: when the correlation indicates that each subtask is independent, performing a parallel search for each subtask in the tool graph using the depth-first search or the breadth-first search to obtain all feasible solutions corresponding to each subtask; and performing a parallel evaluation of all feasible solutions for each subtask according to preset evaluation rules to obtain a target sub-solution corresponding to each subtask; wherein each target sub-solution indicates the target tools used for the corresponding subtask and the order in which the target tools are used, and the target solution includes all target sub-solutions.

[0094] This can greatly improve the system's processing efficiency.

[0095] In one possible implementation, the task decomposition result also includes the association relationship between the multiple subtasks; the step of searching in a preset tool graph to obtain all feasible solutions based on the task decomposition result using a depth-first search or a breadth-first search method includes: when the association relationship indicates that each of the subtasks is executed in a preset execution order, searching in the tool graph in the execution order using the depth-first search method or the breadth-first search method to obtain all feasible solutions.

[0096] In this way, by clearly defining the relationships and execution order between subtasks, searches can be performed according to the logical order of task processing, avoiding unnecessary search paths and thus significantly improving the efficiency and accuracy of task planning. Furthermore, the search strategy based on relationships can plan the required resources in advance, thereby achieving the rational allocation and effective utilization of resources.

[0097] In one possible implementation, the task decomposition result further includes the correlation between the multiple subtasks; the step of searching in a preset tool graph to obtain all feasible solutions based on the task decomposition result using a depth-first search or a breadth-first search method includes: when the correlation indicates that the multiple subtasks include some subtasks executed in a preset execution order and some independently executed subtasks, for the sequentially executed subtasks, searching each subtask in the tool graph according to the execution order using the depth-first search or the breadth-first search method to obtain all feasible solutions; for the independently executed subtasks, performing a parallel search for each subtask in the tool graph using the depth-first search or the breadth-first search method to obtain all feasible solutions corresponding to each subtask; wherein, all feasible solutions for the multiple subtasks include: all feasible solutions for the sequentially executed subtasks and all feasible solutions for each independently executed subtask.

[0098] In this way, by clarifying the relationships and execution order between subtasks, the search process can be executed more effectively, achieving a reasonable allocation and efficient use of resources.

[0099] In one possible implementation, the task decomposition result further includes a task description, input data type, and output data type for each subtask. The task description includes task content and task domain. The task content indicates the processing procedure of the corresponding subtask. The task domain includes one of image processing, speech processing, video processing, and text processing. The input data type indicates the type of input data processed by the corresponding subtask. The input data processed by each subtask comes from at least the multimodal data. The output data type indicates the type of output data of the corresponding subtask. Based on the task decomposition result, a search is performed in a preset tool graph using either depth-first search or breadth-first search to obtain all feasible solutions. This includes performing the following processing on each subtask: based on the input of the current subtask... The search begins by identifying input nodes of the same data type from multiple input nodes in the tool graph, and ends by identifying output nodes of the same data type from multiple output nodes in the tool graph, based on the data type of the current subtask. A search is then conducted among all tool nodes in the tool graph that are directly or indirectly connected to the current search starting point via directed edges to identify target tool nodes that match the task description of the current subtask. Each target tool node includes at least one tool node, and at least one of these target tool nodes is connected to the current search ending point via a second directed edge. Based on the current search starting point, the target tool node, and the current search ending point, all feasible solutions for the current subtask are determined.

[0100] In this way, by breaking down complex multimodal tasks into multiple subtasks, each with a clear task description, input data type, and output data type, the subsequent search target becomes clearer, allowing for the rapid identification of suitable tool nodes and improving task processing efficiency.

[0101] In one possible implementation, the step of searching all tool nodes in the tool graph that are directly or indirectly connected to the current search starting point through directed edges to determine the target tool node that matches the task description of the current subtask includes: scoring the degree of matching between all tool nodes that are directly or indirectly connected to the current search starting point through directed edges and the task description of the current subtask, and determining the target tool node that matches the task description of the current subtask based on the scoring results.

[0102] By utilizing the directed edge relationships of the tool graph for searching, the search process is ensured to be systematic and structured, avoiding the inefficiencies and high error rates that may result from random or disordered searches. This improves search efficiency, reduces unnecessary time and resource consumption, and the introduction of a scoring mechanism allows for a quantitative assessment of the matching degree between tool nodes and task descriptions, providing a clear basis for determining target tool nodes. This helps reduce the influence of subjective judgment and improves the accuracy and reliability of search results.

[0103] In one possible implementation, the apparatus further includes a construction module for constructing the tool graph. The construction process includes: determining multiple different sample tasks and their task descriptions, input data types, and output data types using a preset adjacency matrix or adjacency list; determining multiple input nodes based on all different input data types, multiple output nodes based on all different output data types, and multiple tool nodes based on all different tools; determining multiple directed edges based on the task descriptions of each sample task, the multiple directed edges including multiple first directed edges, multiple second directed edges, and multiple third directed edges; and obtaining the tool graph based on the multiple input nodes, the multiple tool nodes, the multiple output nodes, all first directed edges, all second directed edges, and all third directed edges.

[0104] By representing different sample tasks, input data types, output data types, and tool nodes in a structured manner, the task representation becomes clearer and easier to understand. This representation not only improves the accuracy and efficiency of task decomposition but also provides a foundation for subsequent task search and solution determination. Furthermore, once the tool graph is constructed, task searches can be performed based on it without manually analyzing the dependencies of each tool, significantly improving the automation of task planning. In addition, the tool graph construction process is based on a pre-defined adjacency matrix or adjacency list, allowing for the easy addition of new sample tasks, input data types, output data types, and tool nodes to adapt to task requirements in different domains and scenarios.

[0105] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described multimodal processing method based on a tool-enhanced large language model when executing the instructions stored in the memory. In some embodiments, the electronic device provided by this disclosure may have functions or include modules that can be used to execute the method described in the above method embodiments. Its specific implementation can be referred to the description of the above-described multimodal processing method based on a tool-enhanced large language model embodiments, which will not be repeated here for brevity.

[0106] This disclosure also proposes a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the aforementioned multimodal processing method for a large language model based on tool enhancement. The computer-readable storage medium can be volatile or non-volatile. In some embodiments, the functions or modules included in the computer-readable storage medium provided in this disclosure can be used to execute the method described in the above method embodiments. The specific implementation can be referred to the description of the above embodiments of the multimodal processing method for a large language model based on tool enhancement; for brevity, it will not be repeated here.

[0107] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device performs the above-described multimodal processing method based on a tool-enhanced large language model. In some embodiments, the functions or modules included in the computer program product provided in this disclosure can be used to execute the methods described in the above method embodiments. Specific implementations can refer to the description of the above-described multimodal processing method embodiments based on a tool-enhanced large language model; for brevity, these details will not be repeated here.

[0108] Figure 4 is a block diagram of a multimodal processing apparatus based on a tool-enhanced large language model according to an embodiment of the present disclosure. For example, apparatus 1900 may be provided as a server or terminal device. Referring to Figure 4, apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions executable by processing component 1922, such as application programs. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, processing component 1922 is configured to execute instructions to perform the methods described above.

[0109] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0110] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0111] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0112] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0113] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0114] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0115] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0116] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0117] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0119] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal processing method for large language models based on tool enhancement, characterized in that, The method utilizes a tool-enhanced large language model for multimodal task processing, which includes the following steps: Obtain user requests to be processed, which indicate the multimodal tasks to be processed and the multimodal data required for each multimodal task. Decompose the multimodal tasks to obtain task decomposition results, which include multiple sub-tasks. The multimodal data includes various types of input data. Based on the task decomposition results, all feasible solutions are obtained by searching a preset tool graph using either depth-first search or breadth-first search. All feasible solutions are then evaluated to obtain the target solution. The tool graph includes multiple input nodes, multiple tool nodes, multiple output nodes, and multiple directed edges. The directed edges include multiple first directed edges, multiple second directed edges, and multiple third directed edges. Each input node corresponds to a different input data type, each tool node corresponds to a different tool, and each output node corresponds to a different output data type. Input nodes and tool nodes with dependencies are connected by the first directed edge, with the direction of the first directed edge pointing from the input node to the tool node. Tool nodes and output nodes with dependencies are connected by the second directed edge, with the direction of the second directed edge pointing from the tool node to the output node. Two tool nodes with dependencies are connected by the third directed edge, with the direction of the third directed edge pointing from one tool node to the other. The target solution indicates the target tools used in each subtask and the order in which the target tools are used. Execute the corresponding subtasks using each of the target tools to obtain the tool execution results for each of the subtasks; The processing result for the user request to be processed is determined based on the user request to be processed, the target solution, and the execution results of each tool.

2. The method according to claim 1, characterized in that, The task decomposition result also includes the relationships between the multiple subtasks; based on the task decomposition result, the search is performed in a preset tool graph using either depth-first search or breadth-first search to obtain all feasible solutions, and all feasible solutions are evaluated to obtain the target solution, including: When the relationship indicates that each subtask is independent of the others, a parallel search is performed on the tool graph based on the depth-first search method or the breadth-first search method to obtain all feasible solutions corresponding to each subtask. Then, a parallel evaluation of all feasible solutions for each subtask is performed according to the preset evaluation rules to obtain the target sub-solution corresponding to each subtask. Each of the target sub-schemes indicates the target tools used in the corresponding sub-task and the order in which the target tools are used. The target scheme includes all the target sub-schemes.

3. The method according to claim 1, characterized in that, The task decomposition result also includes the relationships between the multiple subtasks; based on the task decomposition result, a search is performed in a preset tool graph using either depth-first search or breadth-first search to obtain all feasible solutions, including: When the association relationship indicates that each of the subtasks is executed in a preset execution order, the tool graph searches for each of the subtasks in the execution order according to the depth-first search method or the breadth-first search method to obtain all feasible solutions.

4. The method according to claim 1, characterized in that, The task decomposition result also includes the relationships between the multiple subtasks; based on the task decomposition result, a search is performed in a preset tool graph using either depth-first search or breadth-first search to obtain all feasible solutions, including: When the association relationship indicates that the multiple subtasks include some subtasks executed in a preset execution order and some subtasks executed independently, For the sequentially executed subtasks, based on the depth-first search method or the breadth-first search method, the subtasks are searched sequentially in the tool graph according to the execution order to obtain all feasible solutions; For independently executed subtasks, a parallel search is performed on the tool graph based on the depth-first search method or the breadth-first search method to obtain all feasible solutions corresponding to each subtask. Among them, all feasible solutions for the multiple subtasks include: all feasible solutions for some subtasks executed sequentially and all feasible solutions for each subtask executed independently.

5. The method according to claim 1, characterized in that, The task decomposition result also includes the task description, input data type, and output data type of each subtask. The task description includes the task content and task domain. The task content indicates the processing procedure of the corresponding subtask. The task domain includes one of image processing, speech processing, video processing, and text processing. The input data type indicates the type of input data processed by the corresponding subtask. The input data processed by each subtask comes from at least the multimodal data. The output data type indicates the type of output data of the corresponding subtask. Based on the task decomposition results, all feasible solutions are obtained by searching a preset tool graph using either depth-first search or breadth-first search, including performing the following processing on each subtask: Based on the input data type of the current subtask, the input node with the same input data type determined from multiple input nodes in the tool graph is taken as the current search starting point; Based on the output data type of the current subtask, the output node with the same output data type determined from multiple output nodes of the tool graph is taken as the current search end point; A search is performed on all tool nodes in the tool graph that are directly or indirectly connected to the current search starting point through directed edges to determine the target tool node that matches the task description of the current subtask. The target tool node includes at least one tool node, and at least one of the target tool nodes is connected to the current search ending point through the second directed edge. Based on the current search start point, the target tool node, and the current search end point, all feasible solutions for the current subtask are determined.

6. The method according to claim 5, characterized in that, The process of searching all tool nodes in the tool graph that are directly or indirectly connected to the current search starting point through directed edges to determine the target tool node that matches the task description of the current subtask includes: The matching degree between all tool nodes directly or indirectly connected to the current search starting point through directed edges and the task description of the current subtask is scored, and the target tool node that matches the task description of the current subtask is determined based on the scoring results.

7. The method according to claim 5, characterized in that, The method further includes a process for constructing the tool graph, the construction process comprising: Multiple different sample tasks and their task descriptions, input data types, and output data types are determined using a pre-defined adjacency matrix or adjacency list. The plurality of input nodes are determined based on all different input data types, the plurality of output nodes are determined based on all different output data types, and the plurality of tool nodes are determined based on all different tools; Multiple directed edges are determined based on the task descriptions of each sample task, including multiple first directed edges, multiple second directed edges, and multiple third directed edges; The tool graph is obtained based on the plurality of input nodes, the plurality of tool nodes, the plurality of output nodes, all first directed edges, all second directed edges, and all third directed edges.

8. A multimodal processing device for a large language model based on tool enhancement, characterized in that, include: The task decomposition module is used to obtain user requests to be processed. The user requests to be processed indicate the multimodal tasks to be processed and the multimodal data required for processing each multimodal task. The multimodal tasks are decomposed to obtain task decomposition results. The task decomposition results include multiple sub-tasks. The multimodal data includes multiple types of input data. Different types of input data have different types. The task planning module is used to search for all feasible solutions in a preset tool graph based on the task decomposition results, using either depth-first search or breadth-first search, and to evaluate all feasible solutions to obtain a target solution. The tool graph includes multiple input nodes, multiple tool nodes, multiple output nodes, and multiple directed edges. The multiple directed edges include multiple first directed edges, multiple second directed edges, and multiple third directed edges. Each input node corresponds to a different input data type, each tool node corresponds to a different tool, and each output node corresponds to a different output data type. Input nodes and tool nodes with dependencies are connected by a first directed edge, with the direction of the first directed edge pointing from the input node to the tool node. Tool nodes and output nodes with dependencies are connected by a second directed edge, with the direction of the second directed edge pointing from the tool node to the output node. Two tool nodes with dependencies are connected by a third directed edge, with the direction of the third directed edge pointing from one tool node to the other. The target solution indicates the target tools used in each subtask and the order in which the target tools are used. The task execution module is used to execute corresponding sub-tasks using each of the target tools, and obtain tool execution results for each of the sub-tasks. The result determination module is used to determine the processing result corresponding to the user request to be processed based on the user request to be processed, the target scheme, and the execution results of each tool.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 7 when executing instructions stored in the memory.

10. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images