Task planning generation technology supporting process automation
The multimodal large language model generates intermediate representation and combines verification and interpretation tools to convert it into specific task scripts, which solves the problems of low generalization ability of large language models in task planning and incorrect task planning in intelligent agents, and realizes the automation and efficient and orderly completion of task planning.
Patent Information
- Application Number
- CN202411975778.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-09
AI Technical Summary
The existing large language models and multimodal large language models have low generalization capabilities in task planning, and the task planning generated by intelligent agents through these models is prone to errors, which require manual adjustment, and require construction of complex professional training data for fine-tuning training.
Design a task planning generation technology that supports process automation, generates intermediate representations through multimodal large language models, and converts intermediate representations into scripts required for specific tasks through verification and interpretation tools, simplifying the complexity of data preparation and manual adjustment.
It realizes the automation and efficient and orderly completion of task planning, reduces the training cost of large models for specific tasks planning, simplifies the workload of data preparation and manual adjustment, and improves the application efficiency of smart agents.
Smart Images
Figure CN119962484A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information technology, and more particularly to a task planning generation technology supporting process automation. Background Art
[0002] Large language models and multimodal large language models are pre-trained models built on the basis of multimodal data from around the world. By combining domain-specific data, they can achieve cognitive analysis capabilities for specific fields. As content generators for domain problems, they can be applied to fields including prototyping, text generation, image generation, and task planning. Task planning, as a major application field based on large model generation capabilities, has been successfully practiced in different industries and for different types of scenarios and problems. Task planning / content generation achieved by training based on pre-trained models combined with domain-specific data is often well adapted to specific problems and scenarios, but cannot effectively adapt to other related scenarios, that is, the generalization ability is low. One of the important reasons is that the output of large models is usually implemented in text form, and the text scripts generated by task planning for different problems are usually very different in content arrangement and format.
[0003] Existing intelligent agents generate task plans through large language models (LLM) or large multimodal language models (MLLM) technology, and complete task operations by calling related tools. With the help of the knowledge inference ability of LLM / MLLM, the automated planning of intelligent agents can simplify the workload of manual planning. However, it is limited by the capabilities of LLM / MLLM itself and still cannot avoid two types of problems: 1. Errors are prone to occur when generating task scheduling in an end-to-end manner (such as calling application programming interfaces through preset parameters, etc.), which require manual adjustment with professional tools; 2. For different tasks, it is necessary to build complex professional training data to fine-tune the LLM / MLLM to achieve the generation of planning content for specific tasks.
[0004] Therefore, this invention designs an effective intermediate expression, which has an extensible format design and the ability to be verified and corrected through tools. The large model outputs the intermediate expression in text form, and then combines the parsing tool to convert the intermediate expression into the script required for the corresponding specific problem. Summary of the invention
[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a task planning generation technology that supports process automation. On the one hand, it simplifies the complexity of data preparation and reduces the training cost of large models for specific types of task planning; on the other hand, it reduces the complexity of manual adjustment by combining intermediate expressions with verification and interpretation tools, thereby providing more effective support for building intelligent agent applications.
[0006] The above technical objectives of the present invention are achieved through the following technical solutions: a task planning generation technology supporting process automation, the method comprising: S0: Generate intermediate representation using a multimodal large language model; S1: Design and verify the intermediate representation; S2: Iteratively modify the model based on the validation results; S3: A conversion script that converts the intermediate representation into a domain-specific task.
[0007] The present invention is further configured as follows: in step S0, the multimodal large language model corrects the output format of the model by fine-tuning instructions, especially correcting it based on the intermediate representation format.
[0008] The present invention is further configured as follows: the method for fine-tuning the instruction comprises the following steps: A1: Select a typical task set and build a corpus from task prompts to command outputs, where the command outputs are in the intermediate representation format; A2: Fine-tune the selected pre-trained model M to obtain M'; A3: Test and verify in a test environment; A4: Sort out the error samples and perform iterative training until the error rate is lower than the threshold T.
[0009] The present invention is further configured as follows: in step S1, the intermediate representation serves as the basic interface for the machine learning model to interact with the tool, the intermediate representation includes the task name, task description, input data, output data, task configuration and action script, and the intermediate representation is expressed in JSON format, including at least name, task_desc, input, output, configuration and actions fields.
[0010] The present invention is further configured as follows: the verification of the intermediate representation is implemented by means of data flow verification in the form of a static verification tool, and the algorithm of the static verification tool includes the following steps: B1: Convert the generated task intermediate representation into a data flow graph; B2: Check the data flow rationality of each task’s input, output, configuration, and actions.
[0011] The present invention is further configured as follows: the specific method for checking the rationality of the data stream comprises the following steps: C1: Using data flow connectivity as a test method, verify whether the connectivity between tasks matches based on the input and output defined for each task; C2: Use the security of running scripts as a test method, and verify whether there is a situation where the script cannot be executed according to the configuration definition of the task configuration list based on the configuration and actions defined in each task.
[0012] The present invention is further configured such that: in step S2, the iterative correction includes continuously fine-tuning the configuration and script parts of the intermediate representation of the model output.
[0013] The present invention is further configured as follows: in step S3, the conversion script divides the adaptation tasks into the following categories: D1: Direct script generation: Generate executable scripts; D2: Indirect script generation: Generates indirectly executable scripts; D3: Tool content generation: Generates an intermediate representation that can be parsed and executed by other software tools; D4: Query content production: Produce scripts that can be queried by structured and unstructured data engines.
[0014] In summary, the present invention has the following beneficial effects: the present invention provides a large model task planning generation technology based on intermediate representation content generation, realizes task planning by generating intermediate representation, and can realize docking with task-related tools through verification and error correction to complete the planned tasks efficiently and orderly. At the same time, the technology proposed by the present invention also provides iterative planning optimization, which can continuously improve the accuracy and efficiency of operation planning according to the user-set goals. Through the present invention, users can greatly simplify the complexity of data preparation and reduce the workload of manual adjustment of generated task plans. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a logical schematic diagram of the intermediate representation generator in the present invention; Figure 2 A logical schematic diagram of a workflow generated based on the intermediate representation content in the present invention; Figure 3 A logical diagram of task conversion in the present invention; Figure 4 This is a flow chart of instruction fine-tuning in the present invention; Figure 5 It is an iterative correction flow chart of the present invention; Figure 6 It is a logical schematic diagram of the present invention. DETAILED DESCRIPTION
[0016] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention. Example
[0017] This invention provides a large model task planning generation technology based on intermediate representation content generation, which realizes task planning by generating intermediate representation, and can achieve docking with task-related tools through verification and error correction to complete the planned tasks efficiently and orderly. At the same time, the technology proposed by this invention also provides iterative planning optimization, which can continuously improve the accuracy and efficiency of operation planning according to the user's set goals. Through this invention, users can greatly simplify the complexity of data preparation and reduce the workload of manual adjustment of generated task plans.
[0018] The present invention specifically includes four parts: 1. Intermediate representation generator based on multimodal large language model; 2. Intermediate representation design and verification tools; 3. Iterative modification of the model based on verification; 4.Task conversion tools based on intermediate representation can realize content generation and conversion for tasks in different fields.
[0019] First, regarding the generation of intermediate representations based on multimodal large language models: The intermediate representation of this invention is expressed in text form, so a pre-trained model with text output capability is required as the basic model. The model needs to correct the output format of the model by fine-tuning instructions, especially based on the intermediate representation format (detailed in the next section). The instruction fine-tuning method is divided into 4 steps: A1. Select a typical task set and construct a corpus of task prompts to command output, where the command output is based on the intermediate representation format defined in this invention; A2. Fine-tune the selected pre-trained model M to obtain M'; A3. In the test environment T ’ (Based on a test set with similar corpus as 1) for test verification; A4. If the output accuracy is less than the threshold T (threshold), sort out the error samples, reconstruct the prediction library T, perform iterative training, set M=M', and repeat step A2; A5. Until the output accuracy is greater than the threshold T (threshold) (see Appendix Figure 1 ), release M'. The specific process is as attached Figure 4 .
[0020] Secondly, the intermediate representation design and verification, the intermediate representation is hierarchical in a structured way to achieve effective expression of tasks of different complexity. In order to adapt to the language model, the intermediate representation design provided by this invention is mainly in JSON format.
[0021] { name:name_string, task_desc:task_desc_string, input:[in_desc0,in_desc1,…], output:[out_desc0,out_desc1,…], configuration:[conf_desc0,conf_desc1,…], actions:[act_string0:act_body0,act_string1:act_body1,…] } Each item in the JSON file is described as follows: name: Task name task_desc: text-based task description Input: Input data or a list of input-related tasks output: output data or a list of output-related tasks configuration: Task configuration list actions: A list of scripts that need to be executed for a task. Each element in the list is defined in a key-value format, for example: act_stringi:act_bodyi. act_stringi represents the action type expressed by the content of act_bodyi, including: Execute scripts, such as API calls that interact with other software; Data description, which is the operation content specified for a specific software, such as the vector graphics primitives required by drawing tools; Knowledge base query script.
[0022] The workflow for content generation based on intermediate representation is attached. Figure 2 .
[0023] The verification of the intermediate representation is implemented through data flow verification using static verification tools. The specific verification algorithm includes the following two steps: Convert the generated task intermediate representation into a data flow graph; Perform data flow rationality verification on the input, output, configuration, and actions of each task.
[0024] The specific method for data flow rationality verification is as follows: Taking data flow connectivity as the test means, verify whether the tasks match the connectivity according to the input and output defined for each task. For example, check whether there is a situation where the output goes to an undefined task. Taking the security of the running script as the test means, verify whether there is a situation where the script cannot be executed according to the configuration definitions in the task configuration list, based on the configuration and actions defined for each task.
[0025] At the same time, iterative correction continuously improves the quality of the generated intermediate representation content, that is, continuously improves the content of the configuration and actions parts. Therefore, it is necessary to continuously fine-tune the configuration and script parts of the intermediate representation output by the model in the same way as the quality of the corrected output format. This invention constructs training data for each type of task separately and conducts fine-tuning training. The specific process is as follows: E1. Select n types of tasks and set i = 0; E2. Select the i-th type of task and test it through real tasks; E3. Based on static tools, verify the output quality. For unqualified outputs, make manual corrections and construct a training corpus Ti' at one time; E4. Instruct the fine-tuning training model M to obtain M'; E5. Let M = M', i = i + 1, and perform iterative operations; E6. If i < n, that is, the tasks are not trained yet, continue to execute B2. Otherwise, release M.
[0026] Finally, regarding the design of the task conversion tool based on the intermediate representation, as a general representation, the intermediate representation needs to adapt to different types of tasks (see Appendix Figure 3 ). This invention classifies the adapted tasks into 4 categories, which are reflected in the definition representation of the actions list: Direct script generation: Generate executable scripts, such as directly executable programs like Python and Ruby. The definition of the elements in the actions list is: act_string = DIRECT_EXEC, act_body = executable script; Indirect script generation: Generates an indirectly executable script, such as a software tool API call list. A compiler tool converts the call list into an executable program. The actions list element is defined as: act_string = INDIRECT_EXEC, act_body = script definition (converted by the compiler tool); Tool content generation: Generates an intermediate representation that can be parsed and executed by other software tools, such as generating a computer-aided design (CAD) script, and the CAD tool performs drawing operations based on the script content. The actions list element is defined as: act_string=INDIRECT_SCRIPT, act_body={script type, script content}; Query content production: Produce scripts that can be queried by structured and unstructured data engines, such as producing structured query elements. The structured query engine generates query language (SQL) scripts to execute database or knowledge base operations. The actions list elements are defined as: act_string=KG_QL, act_body={query type, query element}.
[0027] This invention proposes a large model task planning generation technology based on intermediate representation content generation. On the one hand, it simplifies the complexity of data preparation and reduces the training cost of large models for specific types of task planning; on the other hand, it reduces the complexity of manual adjustment by combining intermediate expressions with verification and interpretation tools, thereby providing more effective support for building intelligent agent applications.
[0028] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A task planning generation technology supporting process automation, characterized in that: The method includes: S0: Generate intermediate representation using a multimodal large language model; S1: Design and verify the intermediate representation; S2: Iteratively modify the model based on the validation results; S3: A conversion script that converts the intermediate representation into a domain-specific task.
2. The task planning generation technology supporting process automation according to claim 1, characterized in that: In step S0, the multimodal large language model corrects the output format of the model by fine-tuning the instructions, especially correcting it based on the intermediate representation format.
3. The task planning generation technology supporting process automation according to claim 2, characterized in that: The method for fine-tuning instructions comprises the following steps: A1: Select a typical task set and build a corpus from task prompts to command outputs, where the command outputs are in the intermediate representation format; A2: Fine-tune the selected pre-trained model M to obtain M'; A3: Test and verify in a test environment; A4: Sort out the error samples and perform iterative training until the error rate is lower than the threshold T.
4. The task planning generation technology supporting process automation according to claim 1, characterized in that: In step S1, the intermediate representation serves as the basic interface for the machine learning model to interact with the tool. The intermediate representation includes the task name, task description, input data, output data, task configuration and action script. The intermediate representation is expressed in JSON format and includes at least name, task_desc, input, output, configuration and actions fields.
5. The task planning generation technology supporting process automation according to claim 4, characterized in that: The verification of the intermediate representation is implemented by means of data flow verification in the form of a static verification tool, and the algorithm of the static verification tool includes the following steps: B1: Convert the generated task intermediate representation into a data flow graph; B2: Check the data flow rationality of each task’s input, output, configuration, and actions.
6. The task planning generation technology supporting process automation according to claim 5, characterized in that: The specific method for checking the rationality of the data stream comprises the following steps: C1: Using data flow connectivity as a test method, verify whether the connectivity between tasks matches based on the input and output defined for each task; C2: Use the security of running scripts as a test method, and verify whether there is a situation where the script cannot be executed according to the configuration definition of the task configuration list based on the configuration and actions defined in each task.
7. The task planning generation technology supporting process automation according to claim 1, characterized in that: In step S2, the iterative modification includes continuously fine-tuning the configuration and script parts of the intermediate representation of the model output.
8. The task planning generation technology supporting process automation according to claim 1, characterized in that: In step S3, the conversion script divides the adaptation tasks into the following categories: D1: Direct script generation: Generate executable scripts; D2: Indirect script generation: Generates indirectly executable scripts; D3: Tool content generation: Generates an intermediate representation that can be parsed and executed by other software tools; D4: Query content production: Produce scripts that can be queried by structured and unstructured data engines.