LLM-based AI Agent self-evolution tool chain system

Abstract

The invention discloses an LLM-based AI Agent self-evolution tool chain system, and mainly relates to the field of AI Agents. Comprising a Module and a Database, the Module can generate / evaluate a data module by using an LLM, the Database comprises data storage, screening and management, the Claymodel is a platform focusing on data generation, processing and evaluation, a certain single task is completed by using the LLM, the Claymodel currently has two modules, a Generator (generator) module can be created to generate data, and the Module and the Database can be used for storing, screening and managing the data. An Evaluator module is created to evaluate data, once the module is created and debugged, batch automatic generation and automatic evaluation tasks can be executed to replace a large amount of repeated labor in traditional manual annotation and data processing, the data working efficiency of a user is greatly improved, data generated in Generator and Evaluator can be stored in Database, and the data processing efficiency is greatly improved. And the management data is screened according to the score in the Evaluator, and a high-quality data set is sorted out for a production environment.

Description

Technical Field

[0001] The present invention relates to the field of AI Agent, and in particular to an AI Agent self-evolution tool chain system based on LLM. Background Art

[0002] AI Agent (Artificial Intelligence) is an intelligent entity that can perceive the environment, make decisions and perform actions. Unlike traditional artificial intelligence, AI Agent has the ability to gradually complete a given goal by thinking independently and calling tools. The difference between AI Agent and big models is that the interaction between big models and humans is based on prompts. Whether the user prompt is clear and unambiguous will affect the effect of the big model's answer. The work of AI Agent only needs to be given a goal, and it can think independently and take actions based on the goal. In principle, the core driving force of AI Agent is the big model, on which three key components are added: planning, memory and tool use. Therefore, an AI Agent self-evolution tool chain system based on LLM is particularly needed.

[0003] However, the existing LLM-based AI Agent self-evolution tool chain system has a huge demand for high-quality data in the process of integrating LLM (large language model) into products / businesses. Whether it is used as accurate retrieval and recall of external knowledge bases or as data sets required for model training, the quality of data affects the effect of LLM in actual usage scenarios to a certain extent. Currently, manual screening and labeling are still required, and a lot of repetitive work is required in data processing, which not only increases labor intensity but also reduces work efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide an AI Agent self-evolution tool chain system based on LLM, which can realize batch generation of data according to input text, process data format, and automatically evaluate and screen high-quality data to replace traditional manual annotation and a large amount of repetitive work in data processing, greatly improving the user's data work efficiency.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] The AI ​​Agent self-evolution tool chain system based on LLM includes two major functions: Module and Database. The Module is the basic unit in Claymore. The Module can use LLM to generate / evaluate data modules, and the Database includes storage, screening and management of data.

[0007] Furthermore, the overall use process of Claymore includes the following steps:

[0008] Step S1: Create Module / module;

[0009] Step S2: Prompt Test / Edit prompt words;

[0010] Step S3: Run Batch / batch generation;

[0011] Step S4: Evaluator Module / Evaluator module;

[0012] Step S5: Data Base / database data storage and export;

[0013] Step S6: Advanced use of Chat Test.

[0014] Furthermore, the specific steps of S1 are:

[0015] Step S11: On the Module level page, click the "Create" button in the upper right corner, enter the Module title, select Category and template, and you can complete the creation;

[0016] Step S12: Different templates correspond to different application scenarios and data generation types, and the specific content can also be modified on the Module details page;

[0017] Step S13: Use the Prompt Test and Run Batch functions in the Module Details page.

[0018] Furthermore, the specific steps of S2 are:

[0019] Step S21: On the Prompt test page, edit the Prompt in the template according to specific needs and scenarios;

[0020] Step S22: Currently, Claymore provides two different editing modes: normal mode and expert mode;

[0021] Step S23: After editing the prompt / filling in the form, enter text in the Input area on the right and click "Run Sample" to test. If the test result passes, proceed to the next step of batch generating data.

[0022] Furthermore, the specific steps of S3 are:

[0023] Step S31: mainly used to upload documents, using the Prompt tested in step S23 to generate data in batches;

[0024] Step S32: Currently only supports plain text formats such as pdf, txt, and structured csv files;

[0025] Step S33: The Candidate list on the right will display a list of generated results;

[0026] Step S34: The current status of the Module will be displayed below the title, namely Ready / Pending / Running.

[0027] Furthermore, the specific steps of S4 are:

[0028] Step S41: mainly used to automatically evaluate the data generated in the Generator, divided into two functional areas, evaluation standard form, and generation result area;

[0029] Step S42: Enter the automatic evaluation indicators, judgment basis, and output format sample in the evaluation criteria form. The indicators should be described as detailed as possible to help AI perform the evaluation task better.

[0030] Step S43: The current Auto Evaluate module uses the evaluation model inside Claymore and does not support prompt editing;

[0031] Step S44: In the early stage, you need to fill in the basic evaluation criteria for high-quality data in the prompt form.

[0032] Furthermore, the specific steps of S5 are:

[0033] Step S51: The data generated in the Module is only a temporary state and needs to be saved locally or in the Database by clicking the "SaveData" button in the upper right corner;

[0034] Step S52: Database is divided into two parts: Raw data and Validated data;

[0035] Step S53: Save the data in the Database as a Generator, enter a specific folder and filter the data according to relevant indicators;

[0036] Step S54: Save the filtered data to Validated data or save it locally or delete it.

[0037] Furthermore, the specific steps of S6 are:

[0038] Step S61: After completing data generation or evaluation on the Module details page, use Chat Test to perform a quick test on the generated data;

[0039] Step S62: Claymore will use the generated data set as the GPT plug-in knowledge base, and then test the relevant content in the question and answer data.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] The LLM-based AI Agent self-evolution tool chain system of the present invention, Claymore, is a platform focusing on data generation, processing and evaluation. Claymore currently has two modules. A Generator module can be created to generate data, and an Evaluator module can be created to evaluate data. The Database can store the data generated in the Generator and the Evaluator, and manage the data according to the score screening in the Evaluator, and organize high-quality data sets for production environments. By introducing LLM into the data work pipeline, batch generation of data according to input text, processing of data formats, and automatic evaluation and screening of high-quality data can be achieved to replace traditional manual annotation and a large amount of repetitive work in data processing, greatly improving the data work efficiency of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Attached Figure 1 is a flow chart of the overall use of the Claymore of the present invention.

[0043] Attached Figure 2 It is a schematic diagram of creating a Module / module of the present invention.

[0044] Attached Figure 3 It is a schematic diagram of the Prompt Test / edit prompt word of the present invention.

[0045] Attached Figure 4 It is a schematic diagram of Run Batch / batch generation of the present invention.

[0046] Attached Figure 5 It is a schematic diagram of the Evaluator Module of the present invention.

[0047] Attached Figure 6 It is a schematic diagram of the storage and export of the Data Base / database data of the present invention.

[0048] Attached Figure 7 It is a schematic diagram of the advanced use of Chat Test of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0050] The present invention discloses an AI Agent self-evolution tool chain system based on LLM, and the main structure includes two major functions: Module and Database. Module can use LLM to generate / evaluate data modules, and Database includes storage, screening and management of data. Module is the basic unit in Claymore, and LLM is used to complete a single task. Claymore currently has two modules. Generator modules can be created to generate data, and Evaluator modules can be created to evaluate data. Once the modules are created and debugged, batch automatic generation and automatic evaluation tasks can be performed, which greatly improves the user's data work efficiency. Database can store the data generated in Generator and Evaluator, and screen and manage data according to the scores in Evaluator, and organize high-quality data sets for production environments.

[0051] Preferably, the overall use process of Claymore includes the following steps:

[0052] Step S1: Create Module / module;

[0053] The specific steps of S1 are:

[0054] Step S11: On the Module level page, click the "Create" button in the upper right corner, enter the Module title, select the Category and template, and you can complete the creation;

[0055] Step S12: Different templates correspond to different application scenarios and data generation types. You can also modify the specific content on the Module details page later;

[0056] Step S13: Use the Prompt Test prompt word test and Run Batch data batch generation function in the Module details page;

[0057] Step S2: Prompt Test / Edit prompt words;

[0058] The specific steps of S2 are:

[0059] Step S21: On the Prompt test page, you can edit the Prompt in the template according to specific needs and scenarios;

[0060] Step S22: Claymore currently provides two different editing modes: normal mode and expert mode. In normal mode (beginner mode), you only need to fill in the content in the form (such as the rules and output QA sample fields in the figure), and you do not need to modify the prompt content. For users who are more familiar with prompt word engineering, you can use the expert mode to directly write prompt words, but you need to keep the corresponding keyword field (Note: Evaluator does not support expert mode);

[0061] Step S23: After editing the prompt / filling in the form, enter text in the Input area on the right and click "Run Sample" to test. If the test result passes, you can proceed to the next step of batch data generation;

[0062] Step S3: Run Batch / batch generation;

[0063] The specific steps of S3 are:

[0064] Step S31: mainly used to upload documents, using the Prompt tested in step S23 to generate data in batches;

[0065] Step S32: Currently only supports plain text formats such as pdf, txt, and structured csv files;

[0066] Step S33: The Candidate list on the right will display a list of generated results;

[0067] Step S34: The current status of the module, Ready / Pending / Running, will be displayed below the title;

[0068] Step S4: Evaluator Module / Evaluator module;

[0069] The specific steps of S4 are:

[0070] Step S41: mainly used to automatically evaluate the data generated in the Generator, divided into two functional areas, evaluation standard form, and generation result area;

[0071] Step S42: Enter the automatic evaluation indicators, judgment basis, and output format sample in the evaluation criteria form. The indicators should be described as detailed as possible to help AI perform the evaluation task better.

[0072] Step S43: The current Auto Evaluate module uses the evaluation model inside Claymore and does not support Prompt editing.

[0073] Step S44: Currently, users are required to fill in the basic evaluation criteria for high-quality data in the prompt form.

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] Step S5: Data Base / database data storage and export;

[0081] The specific steps of S5 are:

[0082] Step S51: The data generated in the Module is only a temporary state and needs to be saved locally or in the Database by clicking the "SaveData" button in the upper right corner;

[0083] Step S52: Database is divided into two parts: Raw data and Validated data. Raw data directly stores the data saved by the module, and Validated data stores high-quality data that has been manually screened and can be used in the production environment.

[0084] Step S53: Save the data in the Database in units of Generators, and enter a specific folder to filter the data according to relevant indicators;

[0085] Step S54: Save the filtered data to Validated data or save it locally or delete it;

[0086] Step S6: Advanced use of Chat Test;

[0087] The specific steps of S6 are:

[0088] Step S61: After completing data generation or evaluation on the Module details page, you can use Chat Test to do a quick test on the generated data;

[0089] Step S62: Claymore will use the generated data set as the GPT plug-in knowledge base, and then test the relevant content in the question and answer data.

[0090] By introducing LLM into the data work pipeline, the present invention can realize batch generation of data according to input text, process data format, and automatically evaluate and screen high-quality data, so as to replace traditional manual labeling and a large amount of repetitive work in data processing, thereby greatly improving the user's data work efficiency.

Claims

1. The AI ​​Agent self-evolution tool chain system based on LLM is characterized by: It includes two major functions: Module and Database. The Module is the basic unit in Claymore. The Module can use LLM to generate / evaluate data modules. The Database includes storage, screening and management of data.

2. The LLM-based AI Agent self-evolution tool chain system according to claim 1 is characterized by: The overall usage process of Claymore includes the following steps: Step S1: Create Module / module; Step S2: Prompt Test / Edit prompt words; Step S3: Run Batch / batch generation; Step S4: Evaluator Module / Evaluator module; Step S5: Data Base / database data storage and export; Step S6: Advanced use of Chat Test.

3. The LLM-based AI Agent self-evolution tool chain system according to claim 2 is characterized by: The specific steps of S1 are: Step S11: On the Module level page, click the "Create" button in the upper right corner, enter the Module title, select Category and template, and you can complete the creation; Step S12: Different templates correspond to different application scenarios and data generation types. The specific content can also be modified on the Module details page; Step S13: Use the Prompt Test and Run Batch functions in the Module Details page.

4. The LLM-based AI Agent self-evolution tool chain system according to claim 2 is characterized by: The specific steps of S2 are: Step S21: On the Prompt test page, edit the Prompt in the template according to specific needs and scenarios; Step S22: Currently, Claymore provides two different editing modes: normal mode and expert mode; Step S23: After editing the prompt / filling in the form, enter text in the Input area on the right and click "RunSample" to test. If the test result passes, proceed to the next step of batch generating data.

5. The LLM-based AI Agent self-evolution tool chain system according to claim 4 is characterized by: The specific steps of S3 are: Step S31: mainly used to upload documents, using the Prompt tested in step S23 to generate data in batches; Step S32: Currently only supports plain text formats such as pdf, txt, and structured csv files; Step S33: The Candidate list on the right will display a list of generated results; Step S34: The current status of the Module will be displayed below the title, namely Ready / Pending / Running.

6. The LLM-based AI Agent self-evolution tool chain system according to claim 2 is characterized by: The specific steps of S4 are: Step S41: mainly used to automatically evaluate the data generated in the Generator, divided into two functional areas, evaluation standard form, and generation result area; Step S42: Enter the automatic evaluation indicators, judgment basis, and output format sample in the evaluation criteria form. The indicators should be described as detailed as possible to help AI perform the evaluation task better. Step S43: The current Auto Evaluate module uses the evaluation model inside Claymore and does not support prompt editing; Step S44: In the early stage, you need to fill in the basic evaluation criteria for high-quality data in the prompt form.

7. The LLM-based AI Agent self-evolution tool chain system according to claim 2 is characterized by: The specific steps of S5 are: Step S51: The data generated in the module is only a temporary state and needs to be saved to the local computer or database by clicking the "Save Data" button in the upper right corner; Step S52: Database is divided into two parts: Raw data and Validated data; Step S53: Save the data in the Database as a Generator, enter a specific folder and filter the data according to relevant indicators; Step S54: Save the filtered data to Validated data or save it locally or delete it.

8. The LLM-based AI Agent self-evolution tool chain system according to claim 2 is characterized by: The specific steps of S6 are: Step S61: After completing data generation or evaluation on the Module details page, use Chat Test to perform a quick test on the generated data; Step S62: Claymore will use the generated data set as the GPT plug-in knowledge base, and then test the relevant content in the question and answer data.

Images