Multi-agent driven industrial software component assembling method and system

Abstract

The invention discloses a multi-agent driven industrial software component assembly method and system, and the method comprises the steps: obtaining a subtask sequence corresponding to a target task instruction and a first component sequence corresponding to the first subtask sequence; according to the target task instruction, performing regeneration optimization on the first subtask sequence and the first component sequence to obtain a second subtask sequence and a second component sequence; querying a historical task log corresponding to the second subtask sequence and the second component sequence; according to a historical task log, performing regeneration optimization on the second subtask sequence and the second component sequence to obtain an optimization response representation; and according to the historical task log, performing parameter reasoning on the optimization response representation to obtain a target task workflow. According to the assembly method, the development time consumption of the task workflow can be reduced, the working intensity of workers is reduced, and the generation quality of the task workflow is improved. The invention relates to the technical field of deep learning.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a multi-agent driven industrial software component assembly method and system. Background Art

[0002] In industrial software scenarios, the traditional development method is usually to manually write and configure a large number of highly repetitive code structures and interaction logic between various components to achieve the generation of task workflows. This method requires a lot of development time and high workload.

[0003] In addition, there is a development method based on a large language model (LLM), which specifically fine-tunes the large language model based on pre-collected fine-tuning data, and then develops and generates task workflows through the fine-tuned large language model. In practice, the data quality and scale of fine-tuning data are often limited. When faced with novel or complex task scenarios, the large language model after fine-tuning training often cannot understand specific tasks well, has insufficient generalization capabilities, and the generation quality of task workflows is unsatisfactory.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] To this end, an object of an embodiment of the present invention is to provide a multi-agent driven industrial software component assembly method and system, wherein the assembly method can reduce the development time of task workflows, reduce the workload of staff, and improve the generation quality of task workflows.

[0007] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0008] In a first aspect, an embodiment of the present application provides a multi-agent driven industrial software component assembly method, comprising:

[0009] The first language agent obtains a target task instruction, and decomposes the target task instruction to obtain a first subtask sequence;

[0010] The component agent performs component retrieval on the first subtask sequence according to the target task instruction to obtain a first component sequence;

[0011] The first language agent performs a first regeneration optimization on the first subtask sequence and the first component sequence according to the target task instruction to obtain a second subtask sequence and a second component sequence, wherein each subtask in the second subtask sequence corresponds to a task component in the second component sequence;

[0012] The retrieval agent queries the task log corresponding to the second subtask sequence and the second component sequence to obtain a first historical task log and a second historical task log, wherein the first historical task log is used to record historical task instructions, and historical subtask sequences and historical component sequences corresponding to the historical task instructions, and the second historical task log is used to record component information of each task component of the second component sequence, and a component execution log corresponding to the component information;

[0013] The first language agent performs a second regeneration optimization on the second subtask sequence and the second component sequence according to the first historical task log to obtain an optimized response representation, wherein the optimized response representation is used to record a third subtask sequence, a third component sequence and scene graph information corresponding to the target task instruction;

[0014] The second language agent performs parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow, where the target task workflow is used to characterize the third component sequence and component parameters corresponding to each task component of the third component sequence.

[0015] In addition, the assembly method according to the above embodiment of the present application may also have the following additional technical features:

[0016] Further, in one embodiment of the present application, decomposing the target task instruction to obtain a first subtask sequence includes:

[0017] Acquire a first prompt and an initial scene graph corresponding to the target task instruction, wherein the first prompt is used to instruct to generate a plurality of subtasks corresponding to the target task instruction;

[0018] The target task instruction is decomposed according to the first prompt and the initial scene graph to obtain the first subtask sequence.

[0019] Further, in one embodiment of the present application, performing component retrieval on the first subtask sequence according to the target task instruction to obtain a first component sequence includes:

[0020] Get the preset component library;

[0021] Component similarity calculation is performed on the first subtask sequence according to the component library and the target task instruction to obtain the first component sequence.

[0022] Further, in one embodiment of the present application, the querying of the task logs corresponding to the second subtask sequence and the second component sequence to obtain the first historical task log and the second historical task log includes:

[0023] Obtain the preset knowledge graph database and knowledge vector database;

[0024] According to the second subtask sequence and the second component sequence, a first log query is performed on at least one of the knowledge graph database and the knowledge vector database to obtain a target history log, where the target history log is the first history task log and the second history task log.

[0025] Further, in one embodiment of the present application, performing parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow includes:

[0026] Obtaining a preset second prompt, where the second prompt is used to indicate a method for obtaining component parameters corresponding to each task component of the third component sequence;

[0027] According to the second prompt, the second historical task log, the third subtask sequence and the scene graph information, parameter prediction is performed on the third component sequence to obtain the target task workflow.

[0028] Furthermore, in one embodiment of the present application, the assembly method further comprises:

[0029] The reflective agent obtains execution data corresponding to the target task workflow, wherein the execution data includes an execution result and execution information, and the execution result is used to indicate whether the target task component in the target task workflow is successfully executed;

[0030] If the execution result is execution failure, the reflective agent obtains a third historical task log corresponding to the target task workflow, and extracts information from the execution information according to the third historical task log to obtain an execution summary;

[0031] The first language agent performs a third regeneration optimization on the optimized response representation according to the execution summary and the target task instruction to obtain a fourth subtask sequence;

[0032] The second language agent performs repair reasoning on the target task workflow according to the execution summary and the fourth subtask sequence to obtain a repaired target task workflow.

[0033] Further, in the embodiment of the present application, the target task workflow is repaired and inferred according to the execution summary and the fourth subtask sequence to obtain the repaired target task workflow, including:

[0034] Acquire a fourth component sequence corresponding to the fourth subtask sequence, and determine a corresponding fourth historical task log according to the fourth component sequence;

[0035] According to the execution summary, the fourth historical task log, the fourth subtask sequence and the fourth component sequence, parameters of the target task workflow are repaired to obtain the repaired target task workflow.

[0036] Further, in the embodiment of the present application, the execution information is refined according to the third historical task log to obtain an execution summary, including:

[0037] Acquire an expected scene graph, a first scene graph, and a second scene graph corresponding to the execution result, wherein the first scene graph is an actual scene graph corresponding to the target task component, and the second scene graph is a scene graph of a previous frame of the first scene graph;

[0038] According to the expected scene graph, performing execution deviation reflection on the first scene graph to obtain a scene reflection result;

[0039] According to the second scene graph, performing event reflection on the first scene graph to obtain an event reflection result;

[0040] The scenario reflection results and the event reflection results are summarized and refined according to the third historical task log to obtain the execution summary.

[0041] Further, in the embodiment of the present application, the fourth historical task log is obtained by the following steps:

[0042] According to the third component sequence, the fourth component sequence is subjected to component screening to obtain an update task component, so that the second language agent calls the retrieval agent to perform a second log query on the update task component to obtain the fourth historical task log.

[0043] In a second aspect, an embodiment of the present application provides a multi-agent driven industrial software component assembly system, including:

[0044] The first language agent is used to obtain a target task instruction and perform instruction decomposition on the target task instruction to obtain a first subtask sequence; according to the target task instruction, the first subtask sequence and the first component sequence are first regenerated and optimized to obtain a second subtask sequence and a second component sequence, wherein each subtask in the second subtask sequence corresponds to a task component in the second component sequence; according to the first historical task log, the second subtask sequence and the second component sequence are second regenerated and optimized to obtain an optimized response representation, wherein the optimized response representation is used to record a third subtask sequence, a third component sequence and scene graph information corresponding to the target task instruction;

[0045] A component agent, configured to perform component retrieval on the first subtask sequence according to the target task instruction to obtain the first component sequence;

[0046] A retrieval agent is used to query the task log corresponding to the second subtask sequence and the second component sequence to obtain the first historical task log and the second historical task log, wherein the first historical task log is used to record historical task instructions, and the historical subtask sequence and the historical component sequence corresponding to the historical task instructions, and the second historical task log is used to record the component information of each task component of the second component sequence, and the component execution log corresponding to the component information;

[0047] The second language agent is used to perform parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow, wherein the target task workflow is used to characterize the third component sequence and component parameters corresponding to each task component of the third component sequence.

[0048] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0049] at least one processor;

[0050] at least one memory for storing at least one program;

[0051] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0052] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the above method when executed by the processor.

[0053] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0054] The embodiment of the present application discloses a multi-agent driven industrial software component assembly method and system, wherein the assembly method obtains a target task instruction through a first language agent, and decomposes the target task instruction to obtain a first subtask sequence; the component agent performs component retrieval on the first subtask sequence according to the target task instruction to obtain a first component sequence; the first language agent performs a first regeneration optimization on the first subtask sequence and the first component sequence according to the target task instruction to obtain a second subtask sequence and a second component sequence, wherein each subtask in the second subtask sequence corresponds to a task component in the second component sequence; the retrieval agent queries the task log corresponding to the second subtask sequence and the second component sequence to obtain a first historical task log and a second historical task log, wherein the first historical task The log is used to record historical task instructions, as well as historical subtask sequences and historical component sequences corresponding to the historical task instructions. The second historical task log is used to record component information of each task component of the second component sequence, as well as a component execution log corresponding to the component information. The first language agent performs a second regeneration optimization on the second subtask sequence and the second component sequence according to the first historical task log to obtain an optimized response representation, and the optimized response representation is used to record a third subtask sequence, a third component sequence and scene graph information corresponding to the target task instruction. The second language agent performs parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow, and the target task workflow is used to characterize the third component sequence and component parameters corresponding to each task component of the third component sequence. This assembly method is based on the mutual collaboration among the first language agent, the component agent, the retrieval agent and the second language agent. Specifically, the component agent retrieves the components of the subtask sequence, which can reduce the impact of irrelevant components on the quality of task workflow generation and effectively avoid the phenomenon of hallucinations caused by the language agent. In addition, the assembly method obtains historical task logs by querying the task logs corresponding to the subtask sequence and the component sequence through the retrieval agent, and performs inference and prediction on the component sequence and parameters of the target task workflow based on the language agent, which can effectively reduce the time spent on task workflow development, reduce the workload of staff, and improve the quality of task workflow generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the embodiments of the present application or the drawings of the related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 A flowchart of a multi-agent driven industrial software component assembly method provided in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of the structural framework of a multi-agent driven industrial software component assembly system provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] In industrial software scenarios, the traditional development method is usually to manually write and configure a large number of highly repetitive code structures and interaction logic between various components to achieve the generation of task workflows. This method requires a lot of development time and high workload.

[0062] In addition, there is a development method based on a large language model (LLM), which specifically fine-tunes the large language model based on pre-collected fine-tuning data, and then develops and generates task workflows through the fine-tuned large language model. In practice, the data quality and size of fine-tuning data are often limited. When faced with novel or complex task scenarios, the large language model after fine-tuning often cannot understand specific tasks well, has insufficient generalization capabilities, and the quality of task workflow generation is unsatisfactory. In addition, when the generated task workflow does not meet expectations, this method often requires manual intervention to redo or continue further iterative optimization, and it lacks an effective feedback adjustment mechanism.

[0063] In view of this, an embodiment of the present invention provides a multi-agent driven industrial software component assembly method, which is based on the mutual cooperation between the first language agent, the component agent, the retrieval agent and the second language agent. Specifically, the component agent retrieves the subtask sequence by the component agent, which can reduce the impact of irrelevant components on the quality of task workflow generation and effectively avoid the phenomenon of hallucination by the language agent. In addition, the assembly method obtains historical task logs by querying the task logs corresponding to the subtask sequence and the component sequence by the retrieval agent, and performs inference and prediction on the component sequence and parameters of the target task workflow based on the language agent, which can effectively reduce the time consumption of task workflow development, reduce the work intensity of the staff, and improve the generation quality of the task workflow. In addition, the assembly method also extracts the execution information of the target task workflow that fails to execute based on the reflective agent, and repairs and optimizes the optimized response representation and the target task workflow based on the refined execution summary, which can improve the adaptability of the target task workflow generation to complex or novel tasks and improve the generation quality of the target task workflow.

[0064] Reference Figure 1 In an embodiment of the present application, a multi-agent driven industrial software component assembly method includes:

[0065] Step 110: The first language agent obtains a target task instruction and decomposes the target task instruction to obtain a first subtask sequence;

[0066] In some embodiments, the step 110 of decomposing the target task instruction to obtain a first subtask sequence includes:

[0067] A1. Obtaining a first prompt and an initial scene graph corresponding to the target task instruction, wherein the first prompt is used to instruct the generation of a plurality of subtasks corresponding to the target task instruction;

[0068] A2. Decompose the target task instruction according to the first prompt and the initial scene graph to obtain the first subtask sequence.

[0069] In the embodiment of the present application, the first language agent, component agent, retrieval agent, second language agent and reflective agent can be single agents that implement specific functions in a multi-agent system (Multi-Agent Systems) based on a large language model (LLM).

[0070] It is understandable that step 110 can specifically be that the user provides a complex target task instruction to the first language agent in the multi-agent system, and then uses the first prompt to decompose the target task instruction into a plurality of first subtasks, and determines the first subtask sequence based on all the decomposed first subtasks.

[0071] It should be noted that, for robot tasks in industrial software application scenarios, the first language agent can first obtain the first prompt language constructed based on the prompt engineering, and then mobilize the environment observation module, which can usually be a device with an image shooting function, to obtain the RGB-D image corresponding to the target task instruction (i.e., the initial scene graph); then, the decomposed prompt text of the first language agent is constructed according to the first prompt language and the initial scene graph, and the target task instruction is decomposed through the first language agent, so as to obtain several executable first subtasks, which can be the execution operation planned by the robot to achieve the target task instruction.

[0072] Exemplarily, one of the decomposition prompt texts about the first language agent in the embodiments of the present application can be: #First language agent task description. You are a high-level planner for industrial software workflows. Your task is to decompose complex user instructions into a series of executable subtasks, each of which should be independently completed by a specific component. User instructions may involve multiple steps and conditions, and they need to be decomposed into simple and easy-to-understand subtasks to ensure that each subtask is clearly stated and independently executable, and that each subtask is functionally self-consistent for execution by robots or software components. For task instructions that are robot tasks, you will also accept environmental states. Please combine the task instructions and environmental states to generate subtask sequences, related objects that correspond to the subtask sequences one by one, and their expected states (such as positions). Subtasks should have clear descriptions of goals, actions, and expected movements so that the robot or system can understand and execute them.

[0073] Step 120: The component agent performs component retrieval on the first subtask sequence according to the target task instruction to obtain a first component sequence;

[0074] In some embodiments, the step 120, performing component retrieval on the first subtask sequence according to the target task instruction to obtain a first component sequence, includes:

[0075] B1. Get the preset component library;

[0076] B2. Calculate component similarity of the first subtask sequence according to the component library and the target task instruction to obtain the first component sequence.

[0077] In the embodiment of the present application, the component library records multiple task components for executing subtasks; after generating the first subtask sequence, the first language agent can send the first subtask sequence and the obtained target task instruction to the component agent. For a first subtask in the first subtask sequence, step B2 can be to obtain the task description corresponding to the first subtask in the target task instruction; then the first subtask and the task description are input into the component agent as retrieval prompt text, so that the component agent calculates the similarity of each task component in the component library based on the prompt text. The specific similarity calculation method can be any one of cosine similarity, Euclidean distance, etc., and the task component matching the first subtask is determined by comparing the similarity between the first subtask and the task component, and the similarity between the task description and the task component description.

[0078] It is understandable that the remaining first subtasks in the first subtask sequence are similar and can be simply deduced by analogy. There are many ways to construct the retrieval prompt text. For example, it can be simply deduced by referring to the aforementioned decomposition prompt text, which will not be elaborated in this application. After obtaining the task components corresponding to each first subtask, the first component sequence can be determined based on all the obtained task components. It is worth mentioning that the embodiment of the present application retrieves the task components related to the first subtask from the component library based on the component agent, which can reduce the phenomenon of too long prompt text caused by too many irrelevant components, avoid the situation where the prompt text exceeds the input upper limit of the large language model, and at the same time reduce the impact of irrelevant components on the quality of the target task workflow generation, effectively reducing the possibility of hallucinations in the large language model.

[0079] Step 130: the first language agent performs a first regeneration optimization on the first subtask sequence and the first component sequence according to the target task instruction to obtain a second subtask sequence and a second component sequence, wherein each subtask in the second subtask sequence corresponds to a task component in the second component sequence;

[0080] In the embodiment of the present application, after the component agent retrieves the first component sequence matching the first subtask sequence, the component agent can send the first component sequence to the first language agent. The first regeneration optimization in step 130 can be to construct a first optimization prompt text with the target task instruction, the first subtask sequence and the first component sequence, and input the first optimization prompt text into the first language agent, and generate the component execution order and other relationships of each task in the first component sequence through the first language agent, so as to obtain the second subtask sequence and the second component sequence with the component execution order and other relationships.

[0081] Exemplarily, the first optimization prompt text in the embodiment of the present application can be obtained by adding "clearly indicating the mapping relationship between each subtask and the corresponding component in the results" on the basis of the aforementioned decomposition prompt text. There can be many specific ways to construct the first optimization prompt text, and the example of the present application does not limit the first optimization prompt text.

[0082] Step 140: Retrieve the task log corresponding to the second subtask sequence and the second component sequence by the retrieval agent to obtain a first historical task log and a second historical task log, wherein the first historical task log is used to record historical task instructions, and historical subtask sequences and historical component sequences corresponding to the historical task instructions, and the second historical task log is used to record component information of each task component of the second component sequence, and a component execution log corresponding to the component information;

[0083] In some embodiments, the step 140 of querying the task logs corresponding to the second subtask sequence and the second component sequence to obtain the first historical task log and the second historical task log includes:

[0084] C1. Obtain the preset knowledge graph database and knowledge vector database;

[0085] C2. According to the second subtask sequence and the second component sequence, a first log query is performed on at least one of the knowledge graph database and the knowledge vector database to obtain a target historical log, where the target historical log is the first historical task log and the second historical task log.

[0086] In an embodiment of the present application, after generating the second subtask sequence and the second component sequence, the first language agent can send the second subtask sequence and the second component sequence to the retrieval agent. Step 140 can be to construct a log retrieval prompt text based on the second subtask sequence and the second component sequence. There can be many ways to construct the log retrieval prompt text, and the present application does not limit it here. Then, the log retrieval prompt text is input into the retrieval agent so that the retrieval agent generates the corresponding first historical task log and second historical log from the vector data and / or the knowledge graph database, and the vector data and the knowledge graph database both record multiple historical task logs.

[0087] It can be understood that, for step C1, after the retrieval agent receives the log retrieval prompt text, the first search method can be to extract the knowledge graph information in the knowledge graph database through the retrieval agent based on the knowledge graph search method, and then convert the obtained knowledge graph information into a knowledge graph prompt, so that the retrieval agent can generate the corresponding historical task log based on the knowledge graph prompt. Specifically, the data structure of the knowledge graph database is based on the form of a graph, which can be represented by node and edge types. The nodes include target user instruction nodes, subtask nodes, component nodes, etc. The target user instruction node is used to represent the natural language instruction input by the user, the subtask node is used to represent the specific subtask corresponding to the target user instruction, and the component node is used to represent the task component and task component parameters corresponding to the subtask.

[0088] It should be noted that edge types include edges from target user instructions to subtasks, edges from subtasks to task components, edges between components, edges between different target user instructions, edges between subtasks and subtasks, etc. Among them, the edge from target user instructions to subtasks is used to represent multiple subtasks corresponding to a target user instruction. Each edge under this edge type has a sequence number, which is used to represent the execution order of the corresponding subtasks under the target user instruction; the edge from subtask to task component is used to represent the components and component parameters required to call to complete a subtask, which include parameter passing, input, output, etc.; the edge between components is used to represent possible relationships between component parameters, such as data flow, dependency or call relationship; the edge of different target user instructions is used to represent the relationship between different target user instructions, such as sequential relationship, context dependency relationship, correction relationship, etc.; the edge between subtasks and subtasks is used to represent the relationship between subtasks, such as sequential execution relationship, parallel execution relationship, data dependency relationship, etc.

[0089] It should be added that, for step C1, after the retrieval agent receives the log retrieval prompt text, the second search method can be to extract the knowledge vector information in the knowledge vector database by the retrieval agent based on the vector similarity search method, specifically, the approximate nearest neighbor (ANN) algorithm, etc., and then convert the obtained knowledge vector information into a knowledge vector prompt, so that the retrieval agent can generate the corresponding historical task log based on the knowledge vector prompt. The third search method can be to extract the knowledge vector information in the knowledge vector database and the knowledge graph information in the knowledge graph database by the retrieval agent based on the graph retrieval enhanced generation (GraphRAG) technology, and then integrate the obtained knowledge vector information and knowledge graph information into knowledge prompts, so that the retrieval agent generates the corresponding historical task log based on the knowledge prompt, thereby obtaining the first historical task log and the second historical task log. In addition, in actual applications, the search method used by the retrieval agent each time can be obtained by the retrieval agent based on the information level requirements corresponding to the log retrieval prompt text. This application will not go into details here.

[0090] It is worth mentioning that step C2 can be to first use a community detection algorithm (such as the Leiden algorithm) to cluster the knowledge graph database into several entity communities of different granularity levels, and then based on the graph retrieval enhanced generation (GraphRAG) technology, perform a global log query on the entity community to find the first historical task log and the second historical task log corresponding to the second subtask sequence and the second component sequence. The first historical task log is used to represent the log data that is the same or similar to the second subtask sequence and the second component sequence, that is, the first historical task log records historical task instructions, historical subtask sequences and historical component sequences; the second historical task log is used to record the component information and component execution log corresponding to each task component of the second component sequence.

[0091] It should be added that the embodiment of the present application obtains the first historical task log and the second historical task log corresponding to the target task instructions, the second subtask sequence, the second component sequence, etc. based on the retrieval agent, which can better assist the planning and execution of subtasks, thereby generating a target task workflow with better generation quality; in addition, the embodiment of the present application obtains the knowledge graph database through the retrieval agent, and can realize semantic associations across work logs through the knowledge graph, discover implicit relationships, and form a more comprehensive log collection, which can improve the accuracy and practicality of the first historical task log matching.

[0092] Step 150: the first language agent performs a second regeneration optimization on the second subtask sequence and the second component sequence according to the first historical task log to obtain an optimized response representation, wherein the optimized response representation is used to record the third subtask sequence, the third component sequence and the scene graph information corresponding to the target task instruction;

[0093] In the embodiment of the present application, after the retrieval agent generates the first historical task log, the retrieval agent can send the first historical task log to the first language agent. Step 150 can be to construct a log prompt text according to the first historical log, so that the first language agent reprocesses and optimizes the execution order of the second subtask sequence and the execution order of the second component sequence based on the log prompt text, thereby improving the quality of the output response of the first language agent and obtaining an optimized response representation, which includes a third subtask sequence, a third component sequence and scene graph information, and the scene graph information is used to indicate the environment and system status when each subtask of the third subtask sequence is executed.

[0094] Step 160: The second language agent performs parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow, where the target task workflow is used to characterize the third component sequence and component parameters corresponding to each task component of the third component sequence.

[0095] In some embodiments, the step 160 of performing parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow includes:

[0096] D1. Obtain a preset second prompt, where the second prompt is used to indicate a method for obtaining component parameters corresponding to each task component of the third component sequence;

[0097] D2. According to the second prompt, the second historical task log, the third subtask sequence and the scene graph information, parameter prediction is performed on the third component sequence to obtain the target task workflow.

[0098] In the embodiment of the present application, the first prompt is used to indicate the way in which the second language agent obtains the component parameters of each task component, which may specifically be "there are multiple sources of component parameters: ① inferring from the user instruction context; ② relying on the output of the previous component; ③ asking the user", and the example of this application is for illustration only.

[0099] It can be understood that step D2 can be to construct a parameter reasoning prompt text based on the second prompt, the second historical task log, the third subtask sequence and the scene graph information, and then use the second language agent to reason about the parameter type and description of each task component in the third component sequence based on the parameter reasoning prompt text to obtain the target task workflow.

[0100] Exemplarily, the parameter reasoning prompt text in the embodiment of the present application may specifically be "#Second language agent task description. You are a manipulator with a mobile base. Given user instructions, environment and system information and logs, you need to optimize and refine the matching of each subtask with the component, and generate corresponding parameters. You need to give reasons for the selected parameters. There are many sources of parameters: ① Reasoning from the context of user instructions; ② Depend on the output of the previous component; ③ Ask the user. #Robot constraints. You can only grab one object at a time."

[0101] In some embodiments, the assembly method further comprises:

[0102] E1. The reflective agent obtains execution data corresponding to the target task workflow, wherein the execution data includes execution results and execution information, and the execution results are used to indicate whether the target task component in the target task workflow is successfully executed;

[0103] In the embodiment of the present application, after the second language agent generates the target task workflow, the platform can gradually execute the component actions of the target task workflow and record the execution status, thereby generating execution data corresponding to the target task workflow and sending it to the reflective agent. The target task component can be any task component in the target task workflow.

[0104] E2. If the execution result is execution failure, the reflective agent obtains a third historical task log corresponding to the target task workflow, and extracts information from the execution information according to the third historical task log to obtain an execution summary;

[0105] Furthermore, the step E2, according to the third historical task log, extracts the execution information to obtain an execution summary, including:

[0106] E21. Acquire an expected scene graph, a first scene graph, and a second scene graph corresponding to the execution result, wherein the first scene graph is an actual scene graph corresponding to the target task component, and the second scene graph is a scene graph of a previous frame of the first scene graph;

[0107] E22. According to the expected scene graph, perform execution deviation reflection on the first scene graph to obtain a scene reflection result;

[0108] E23. Perform event reflection on the first scene graph according to the second scene graph to obtain an event reflection result;

[0109] E24. Summarize and refine the scenario reflection results and the event reflection results according to the third historical task log to obtain the execution summary.

[0110] In an embodiment of the present application, if the execution result is an execution failure, the reflective agent can first obtain the third historical task log corresponding to the target task workflow, and the third historical task log is used for log data that is the same or similar to the third component sequence and component parameters, and the log data records the historical component sequence that is the same or similar to the third component sequence, and the component parameters corresponding to the historical component sequence and the log of execution success or failure; then the reflective agent can extract information from the execution information based on the acquired third historical task log. Specifically, the expected scene graph in step E1 can be the expected environmental state corresponding to the target task component that failed to execute; when the execution result of the target task component is an execution failure, the first scene graph can be the actual environmental state corresponding to the failure of the target task component to execute; the second scene graph can be the scene graph of the previous frame of the first scene graph.

[0111] It can be understood that the execution deviation reflection of step E22 can be to calculate the deviation between the actual environmental state represented by the first scene graph and the expected environmental state represented by the expected scene graph, so as to obtain the scene reflection result, and the scene reflection result is used to characterize the deviation between the subtask execution result corresponding to the target task component and the task plan; the event reflection of step E23 is similar, and its event reflection result is used to provide task event information when the fault is sent, which will not be elaborated in this application.

[0112] It should be noted that step E24 can be based on the third historical task log, by constructing a reflection prompt for the event reflection result and the scene reflection result and inputting it into the reflection agent, and the reflection agent summarizes and refines the above two reflection results to obtain an execution summary in the form of a triple (error type, cause, suggestion), so that the reflection agent can think and obtain whether the reason for the failure or success of the subtask corresponding to the component is due to the failure of some component conditions to be met based on the component sequence and the execution success or failure log in the third historical task log, combined with the scene reflection result and the event reflection result; in addition, the reflection agent can also think and obtain whether the reason for the failure or success of the subtask corresponding to the component is due to the existence of specific usage restrictions of the component (for example, the frequency, power consumption, temperature, etc. of the components corresponding to the component in the actual execution environment) based on the component parameters and the execution success or failure log in the third historical task log, combined with the scene reflection result and the event reflection result. After the reflection agent thinks and obtains the reason for the failure or success of the subtask corresponding to the component, the execution summary can be determined based on the obtained component condition reasons and component restrictions or constraints.

[0113] Exemplarily, the reflection prompt may specifically be "On the premise that you already know the components and execution results, please explain the task failure. The task name is [TASK_NAME]. In the [k]th round, an error occurred that caused the task execution to fail. Please determine the error type based on the provided historical task logs, event reflection results, and scenario reflection results, explain the cause of the error, and give suggestions for improvement. Please output the following information in the form of a triple (error type, cause, suggestion): Error type: Identify the specific error type in task execution, such as semantic errors, grammatical errors, logical errors, physical errors (such as obstructed movement), program running errors (such as communication errors), planning errors (such as LLM illusions, task component selection errors, component parameter errors), etc. Cause: Explain in detail the root cause of the error, including component configuration problems, incorrect input parameters, and the execution environment does not meet expectations. Suggestions: Make targeted improvement suggestions to resolve current errors and optimize subsequent task execution, including adjusting component parameters, modifying task planning, and improving execution strategies."

[0114] E3. The first language agent performs a third regeneration optimization on the optimized response representation according to the execution summary and the target task instruction to obtain a fourth subtask sequence;

[0115] In an embodiment of the present application, a planning repair prompt instruction can be constructed according to the execution summary and the target task instruction, and the first language agent can be enabled to regenerate reasoning on the subtask plan in the third subtask sequence in the optimization response representation based on the planning repair prompt instruction, so as to obtain a fourth subtask sequence after the adjustment of the subtask plan, and the planning adjustment includes subtask adjustment and subtask execution order adjustment, etc. Specifically, the subtasks of the fourth subtask sequence in the embodiment of the present application may be the same as or different from the subtasks of the third subtask sequence. For example, the subtasks of the third subtask sequence are task A, task B, task C and task D, while the subtasks of the fourth subtask sequence are task A, task B, task D and task E, respectively. The examples in this application are for illustration only.

[0116] E4. The second language agent performs repair reasoning on the target task workflow according to the execution summary and the fourth subtask sequence to obtain a repaired target task workflow.

[0117] Furthermore, the step E4, performing repair reasoning on the target task workflow according to the execution summary and the fourth subtask sequence to obtain a repaired target task workflow, includes:

[0118] E41. Acquire a fourth component sequence corresponding to the fourth subtask sequence, and determine a corresponding fourth historical task log according to the fourth component sequence;

[0119] Furthermore, the fourth historical task log in step E41 is obtained by the following steps:

[0120] E411. Perform component screening on the fourth component sequence according to the third component sequence to obtain an update task component, so that the second language agent calls the retrieval agent to perform a second log query on the update task component to obtain the fourth historical task log.

[0121] In an embodiment of the present application, after obtaining the corresponding fourth component sequence according to the fourth subtask sequence, the component screening in step E411 may first obtain the common task components between the fourth component sequence and the third component sequence, and then screen the fourth component sequence based on the common task components, and determine the remaining task components in the fourth component sequence except the common task components as update task components.

[0122] It is understandable that the second log query of the retrieval agent for the update task component is similar to the content of the aforementioned first log query, and can be simply inferred to obtain the fourth historical task log corresponding to the fourth component sequence, which is not elaborated in this application.

[0123] E42. Repair parameters of the target task workflow according to the execution summary, the fourth historical task log, the fourth subtask sequence and the fourth component sequence to obtain the repaired target task workflow.

[0124] In an embodiment of the present application, step E42 can specifically be to construct a parameter repair prompt text based on the execution summary, the fourth historical task log, the fourth subtask sequence and the fourth component sequence. There are many ways to construct the parameter repair prompt text, so that the second language intelligent agent can optimize and regenerate the task components and component parameters of the target task workflow, thereby completing the parameter repair of the target task workflow and obtaining the repaired target task workflow.

[0125] It can be understood that if the execution data corresponding to the repaired target task workflow still has an execution result of failure, steps E1 to E4 can be executed again, which can improve the adaptability of multiple agents to novel or complex task scenarios and effectively ensure the generation quality of the target task workflow.

[0126] A multi-agent driven industrial software component assembly system proposed according to an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0127] Reference Figure 2 , a multi-agent driven industrial software component assembly system proposed in the embodiment of the present application includes a first language agent 101, a component agent 102, a retrieval agent 103, and a second language agent 104;

[0128] The first language agent is used to obtain a target task instruction and perform instruction decomposition on the target task instruction to obtain a first subtask sequence; according to the target task instruction, the first subtask sequence and the first component sequence are first regenerated and optimized to obtain a second subtask sequence and a second component sequence, wherein each subtask in the second subtask sequence corresponds to a task component in the second component sequence; according to the first historical task log, the second subtask sequence and the second component sequence are second regenerated and optimized to obtain an optimized response representation, wherein the optimized response representation is used to record a third subtask sequence, a third component sequence and scene graph information corresponding to the target task instruction;

[0129] A component agent, configured to perform component retrieval on the first subtask sequence according to the target task instruction to obtain the first component sequence;

[0130] A retrieval agent is used to query the task log corresponding to the second subtask sequence and the second component sequence to obtain the first historical task log and the second historical task log, wherein the first historical task log is used to record historical task instructions, and the historical subtask sequence and the historical component sequence corresponding to the historical task instructions, and the second historical task log is used to record the component information of each task component of the second component sequence, and the component execution log corresponding to the component information;

[0131] The second language agent is used to perform parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow, wherein the target task workflow is used to characterize the third component sequence and component parameters corresponding to each task component of the third component sequence.

[0132] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] Reference Figure 3 , the embodiment of the present application further provides an electronic device, including:

[0134] at least one processor 201;

[0135] At least one memory 202, used to store at least one program;

[0136] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiment.

[0137] Similarly, it can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0139] Similarly, the contents of the above method embodiments are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0141] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0142] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0143] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0144] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0145] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0146] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0147] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0148] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A multi-agent driven industrial software component assembly method, characterized in that: include: The first language agent obtains a target task instruction, and decomposes the target task instruction to obtain a first subtask sequence; The component agent performs component retrieval on the first subtask sequence according to the target task instruction to obtain a first component sequence; The first language agent performs a first regeneration optimization on the first subtask sequence and the first component sequence according to the target task instruction to obtain a second subtask sequence and a second component sequence, wherein each subtask in the second subtask sequence corresponds to a task component in the second component sequence; The retrieval agent queries the task log corresponding to the second subtask sequence and the second component sequence to obtain a first historical task log and a second historical task log, wherein the first historical task log is used to record historical task instructions, and historical subtask sequences and historical component sequences corresponding to the historical task instructions, and the second historical task log is used to record component information of each task component of the second component sequence, and a component execution log corresponding to the component information; The first language agent performs a second regeneration optimization on the second subtask sequence and the second component sequence according to the first historical task log to obtain an optimized response representation, wherein the optimized response representation is used to record a third subtask sequence, a third component sequence and scene graph information corresponding to the target task instruction; The second language agent performs parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow, where the target task workflow is used to characterize the third component sequence and component parameters corresponding to each task component of the third component sequence.

2. The assembly method according to claim 1, characterized in that: Decomposing the target task instruction to obtain a first subtask sequence includes: Acquire a first prompt and an initial scene graph corresponding to the target task instruction, wherein the first prompt is used to instruct to generate a plurality of subtasks corresponding to the target task instruction; The target task instruction is decomposed according to the first prompt and the initial scene graph to obtain the first subtask sequence.

3. The assembly method according to claim 1, characterized in that: The step of performing component retrieval on the first subtask sequence according to the target task instruction to obtain a first component sequence includes: Get the preset component library; Component similarity calculation is performed on the first subtask sequence according to the component library and the target task instruction to obtain the first component sequence.

4. The assembly method according to claim 1, characterized in that: The querying of the task log corresponding to the second subtask sequence and the second component sequence to obtain the first historical task log and the second historical task log includes: Obtain the preset knowledge graph database and knowledge vector database; According to the second subtask sequence and the second component sequence, a first log query is performed on at least one of the knowledge graph database and the knowledge vector database to obtain a target history log, where the target history log is the first history task log and the second history task log.

5. The assembly method according to claim 1, characterized in that: The step of performing parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow includes: Obtaining a preset second prompt, where the second prompt is used to indicate a method for obtaining component parameters corresponding to each task component of the third component sequence; According to the second prompt, the second historical task log, the third subtask sequence and the scene graph information, parameter prediction is performed on the third component sequence to obtain the target task workflow.

6. The assembly method according to claim 1, characterized in that: The assembly method further comprises: The reflective agent obtains execution data corresponding to the target task workflow, wherein the execution data includes an execution result and execution information, and the execution result is used to indicate whether the target task component in the target task workflow is successfully executed; If the execution result is execution failure, the reflective agent obtains a third historical task log corresponding to the target task workflow, and extracts information from the execution information according to the third historical task log to obtain an execution summary; The first language agent performs a third regeneration optimization on the optimized response representation according to the execution summary and the target task instruction to obtain a fourth subtask sequence; The second language agent performs repair reasoning on the target task workflow according to the execution summary and the fourth subtask sequence to obtain a repaired target task workflow.

7. The assembly method according to claim 6, characterized in that: The step of performing repair reasoning on the target task workflow according to the execution summary and the fourth subtask sequence to obtain a repaired target task workflow includes: Acquire a fourth component sequence corresponding to the fourth subtask sequence, and determine a corresponding fourth historical task log according to the fourth component sequence; According to the execution summary, the fourth historical task log, the fourth subtask sequence and the fourth component sequence, parameters of the target task workflow are repaired to obtain the repaired target task workflow.

8. The assembly method according to claim 6, characterized in that: The extracting the execution information according to the third historical task log to obtain an execution summary includes: Acquire an expected scene graph, a first scene graph, and a second scene graph corresponding to the execution result, wherein the first scene graph is an actual scene graph corresponding to the target task component, and the second scene graph is a scene graph of a previous frame of the first scene graph; According to the expected scene graph, performing execution deviation reflection on the first scene graph to obtain a scene reflection result; According to the second scene graph, performing event reflection on the first scene graph to obtain an event reflection result; The scenario reflection results and the event reflection results are summarized and refined according to the third historical task log to obtain the execution summary.

9. The assembly method according to claim 7, characterized in that: The fourth historical task log is obtained by the following steps: According to the third component sequence, the fourth component sequence is subjected to component screening to obtain an update task component, so that the second language agent calls the retrieval agent to perform a second log query on the update task component to obtain the fourth historical task log.

10. A multi-agent driven industrial software component assembly system, characterized in that: It includes first language agent, component agent, retrieval agent and second language agent; The first language agent is used to obtain a target task instruction and perform instruction decomposition on the target task instruction to obtain a first subtask sequence; according to the target task instruction, the first subtask sequence and the first component sequence are first regenerated and optimized to obtain a second subtask sequence and a second component sequence, wherein each subtask in the second subtask sequence corresponds to a task component in the second component sequence; according to the first historical task log, the second subtask sequence and the second component sequence are second regenerated and optimized to obtain an optimized response representation, wherein the optimized response representation is used to record a third subtask sequence, a third component sequence and scene graph information corresponding to the target task instruction; A component agent, configured to perform component retrieval on the first subtask sequence according to the target task instruction to obtain the first component sequence; A retrieval agent is used to query the task log corresponding to the second subtask sequence and the second component sequence to obtain the first historical task log and the second historical task log, wherein the first historical task log is used to record historical task instructions, and the historical subtask sequence and the historical component sequence corresponding to the historical task instructions, and the second historical task log is used to record the component information of each task component of the second component sequence, and the component execution log corresponding to the component information; The second language agent is used to perform parameter reasoning on the optimized response representation according to the second historical task log to obtain a target task workflow, wherein the target task workflow is used to characterize the third component sequence and component parameters corresponding to each task component of the third component sequence.

Images