A method and system for constructing an entity extraction agent based on a large language model

Abstract

The present invention relates to the technical field of entity extraction and discloses a method for constructing an entity extraction intelligent agent based on a large language model. The method comprises the following steps: classifying and slicing entity label files, converting label names and corresponding entity words into vectors to be queried, organizing them into a label vector library, and finally merging them into a knowledge base; compiling prompt words according to a preset paradigm; selecting an adapted large language model from the model library, evaluating the accuracy and recall of candidate models using a test sample set, and selecting the optimal model based on the evaluation results; developing a tool according to downstream task requirements, completing service URL testing, tool usability testing, and automated call process verification; inputting the knowledge base and prompt words into the large language model for entity extraction, calling the tool to process the extraction results, verifying the output correctness, and iteratively optimizing the configuration of each stage based on the debugging results. The present invention solves the problems of poor general model effect and cumbersome use in existing intelligent agent construction methods.

Description

Technical Field

[0001] The present invention relates to the field of entity extraction technology, and in particular to a method and system for constructing an entity extraction intelligent agent based on a large language model. Background Art

[0002] The development of named entity recognition (NER) technology can be divided into several major stages.

[0003] (1) Early stage: rule-based and dictionary-based methods

[0004] Rule-based methods: In the early days of natural language processing, entity extraction primarily relied on hand-crafted rules. These rules, often based on the grammatical structure and lexical features of a language, defined a series of patterns to identify specific entity types. For example, a rule could be used to identify phrases beginning with "Mr." as human names.

[0005] Dictionary matching: Another common approach is to use a dictionary for matching. The dictionary contains known entity names, and extraction is achieved by searching for these known entities in the text. This approach has the advantage of being simple and direct, but has the disadvantages of limited dictionary coverage and difficulty handling newly discovered entity names.

[0006] (2) Statistical learning method stage

[0007] Feature Engineering: With the development of machine learning, statistical methods have emerged. These methods learn entity features from labeled training data. Common features include word context, affixes, parts of speech, and contextual features. Feature engineering is a key step in this phase, requiring researchers to carefully design and select features based on task requirements and data characteristics.

[0008] Model Application: Common statistical models include the Hidden Markov Model (HMM), Maximum Entropy Model (MaxEnt), Support Vector Machine (SVM), and Conditional Random Field (CRF). These models can effectively capture the contextual information and sequential dependencies of entities, improving the accuracy of entity extraction.

[0009] Sequence labeling: Entity extraction tasks are often transformed into sequence labeling problems, using annotation methods such as IO, BIO, and BIOES to label entities. For each word or character in the text, there are several candidate labels. The model needs to predict the label of each word or character to identify the boundaries and types of entities.

[0010] (3) Deep learning method stage

[0011] Neural Network Models: In recent years, the rapid development of deep learning technology has brought new breakthroughs to entity extraction. Deep neural networks such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), and Transformers are widely used in entity extraction tasks. These models can automatically learn deep semantic features and complex contextual information in text, reducing reliance on manual feature engineering.

[0012] Pre-trained large language models: The emergence of pre-trained language models (such as BERT and GPT) has further advanced entity extraction technology. These models are pre-trained on large text corpora to learn rich linguistic features and knowledge. Fine-tuning then adapts them to downstream NER tasks. Pre-trained large language models have achieved impressive accuracy in entity extraction tasks.

[0013] However, there are some problems when applying large language models to downstream specific domain tasks: (1) Based on prior knowledge, there is currently no universal large model, that is, a model that can achieve the best results in all fields. This actually brings great limitations to downstream task processing. The corpus that needs to be processed in each specific field is different, and the entity words that need to be extracted and the given labels are also different. Therefore, when applying large language models to entity extraction tasks in different fields, the final extraction accuracy is not high, and it takes a long time to fine-tune the model. (2) In some fields with entity extraction needs, practitioners may not understand large language models. They just want an intelligent method to quickly use large language models to process downstream tasks. (3) After using the large language model to complete the entity extraction task, it is generally necessary to manually further process the extracted entity words before processing the downstream tasks. Summary of the Invention

[0014] The present invention provides a method and system for constructing an entity extraction intelligent agent based on a large language model to solve the problems of poor universal model effect and cumbersome use in existing intelligent agent construction methods.

[0015] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0016] In a first aspect, the present invention provides a method for constructing an entity extraction agent based on a large language model, comprising:

[0017] Obtain entity label files for a specific field, classify and slice the entity label files, convert the label names and entity words corresponding to the label names in the entity label files into query vectors, organize the query vectors into a label vector library, and finally merge the label vector library into a knowledge base;

[0018] Write prompt words according to the preset paradigm, where prompt words include: role definition, task content, Chinese and English labels, extraction instructions and standard output format;

[0019] Selecting a large language model adapted to the specific domain from a model library, generating a test sample set based on the specific domain, evaluating the accuracy and recall of the selected large language model using the test sample set, and selecting the optimal model based on the evaluation results;

[0020] Develop tools based on the downstream task requirements of the specific field, and conduct service URL testing, usability testing, and automated call process verification on the developed tools. Once all tests and verifications pass, the developed tools will be considered the optimal tools.

[0021] The knowledge base and prompt words are input into the optimal model to extract entities, the optimal tool is called to process the results of entity extraction, and the correctness of the output is verified. When the correctness meets the requirements, the optimal model and the optimal tool are constructed into an intelligent entity.

[0022] Optionally, the entity label file includes: a text file or a table file.

[0023] Optionally, the preset paradigm includes:

[0024] The default pattern for defining a character's prompt is: a model character starting with the pound sign and the character name;

[0025] The default prompt format for the task content is: an entity extraction task that begins with the pound sign and the task name;

[0026] The default format for prompts for Chinese and English labels is: a mapping relationship between example labels starting with the pound sign plus the Chinese label and its English name;

[0027] The default pattern of the prompt word for the extraction instruction is: the extraction rule starts with the pound character plus the instruction for entity extraction;

[0028] The default format for the prompt word of the standard output format is: the output format starts with the pound sign and standard output.

[0029] Optionally, generating a test sample set based on the specific domain includes:

[0030] Compile a test sample library containing domain questions and standard answers in the specific domain;

[0031] Randomly sampling the test sample library according to a preset ratio to generate a test sample set;

[0032] The selected large language model is evaluated for accuracy and recall using a test sample set, and the optimal model is selected based on the evaluation results, including:

[0033] Input the domain questions in the test sample set into the selected large language model one by one and record the operation results of the large language model;

[0034] The running results of the large language model are compared with the standard answers in the test sample set, the accuracy and recall of the running results and the standard answers are calculated, and the large language model with the highest accuracy and recall is regarded as the optimal model.

[0035] Optionally, the automated calling process verification of the tool includes:

[0036] Ensure that the service URL response status code is 200 and the returned data format meets the requirements;

[0037] Verify that the tool performs its functions correctly and the results are as expected;

[0038] Integrate tools into agents and validate end-to-end call flows.

[0039] Optionally, the process of verifying the correctness of the output includes: verifying the output result, verifying the output format, and verifying the tool processing result;

[0040] When a question in the specific field is input, if the output result is a correct answer, then the output result is verified to be correct; if the output correct answer conforms to the format, then the output format is verified to be correct; if the output correct answer is not an entity extraction result, then the processing result of the verification tool is correct;

[0041] Correctness meets the requirements when the verification output is correct, the verification output format is correct, and the verification tool processes the result correctly;

[0042] When there are inaccuracies in the verification output results, verification output format, and verification tool processing results, if the output result is wrong, reconfigure the knowledge base; if the output format does not match, adjust the model or prompt words; if the output result is an entity extraction result, redevelop the tool.

[0043] In a second aspect, an embodiment of the present application provides an entity extraction agent construction system based on a large language model, the system comprising:

[0044] The knowledge base module is used to store the label vector library and knowledge base of a specific field;

[0045] The prompt word module is used to generate entity extraction instructions according to the preset paradigm;

[0046] Model adaptation module, used to select an adapted large language model from the model library;

[0047] Tool integration module, used to connect to downstream task tools and verify the tools;

[0048] The execution engine module is used to call the large language model to extract entities based on the label vector library and entity extraction instructions, and input the entity extraction results into the tool to complete downstream tasks;

[0049] The debugging and optimization module is used to iteratively optimize the knowledge base, prompt words, large language model, and tool configuration based on the output results.

[0050] Optionally, the tool integration module includes a tool development submodule and a tool verification submodule;

[0051] The tool development submodule is used to develop tools according to the downstream task requirements of the specific field;

[0052] The tool verification submodule is used to perform service URL testing, usability testing, and automated call process verification on the developed tools. When all tests and verifications pass, the developed tools will be used as tools for connecting to downstream tasks.

[0053] Optionally, the execution engine module includes a model processing submodule and a result output submodule;

[0054] The model processing submodule is used to input the knowledge base and prompt words into the large language model to extract entities;

[0055] The result output submodule is used to call the tool to process the result of entity extraction and output the result as an output result.

[0056] Optionally, the debugging and optimization module includes a verification submodule and an iterative debugging submodule;

[0057] The verification submodule is used to verify the correctness of the output results in the execution engine module, and to construct the large language model and tools into an intelligent agent when the correctness meets the requirements;

[0058] The iterative debugging submodule is used to send iterative instructions to the knowledge base module when the correctness does not meet the requirements. The knowledge base module reconfigures the label vector library and the knowledge base based on the iterative instructions, and / or sends iterative instructions to the prompt word module. The prompt word module regenerates the entity extraction instructions based on the iterative instructions, and / or sends iterative instructions to the model adaptation module. The model adaptation module reselects the adapted large language model based on the iterative instructions, and / or sends iterative instructions to the tool integration module. The tool integration module redevelops the tool based on the iterative instructions.

[0059] Optionally, the system is compatible with different large language model APIs and can expand the model library and tool library through a plug-in mechanism.

[0060] Beneficial effects:

[0061] The method for constructing an entity extraction intelligent agent based on a large language model provided by the present invention has no restrictions on the file types of files used in the knowledge base, and different types of data can be processed by the intelligent agent, which is highly practical. For writing prompt words, we have given the specific form of the paradigm, and it only needs to be rewritten according to the paradigm for different fields. The model used can be flexibly selected and does not require long-term fine-tuning, saving a lot of time. The constructed intelligent agent is easy to use for users. After the user enters the content to be extracted in the dialog box, the intelligent agent will directly return the result of calling the large language model and tool, eliminating the intermediate manual operation process and realizing the intelligent processing of downstream tasks. This intelligent agent can be embedded in the original systems of enterprises in different fields, which can save development time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 One of the flowcharts of constructing an entity extraction agent based on a large language model according to a preferred embodiment of the present invention;

[0063] Figure 2 This is the second flowchart of the entity extraction agent constructed based on the large language model in the preferred embodiment of the present invention;

[0064] Figure 3 This is a flowchart of the use of intelligent agents in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following is a clear and complete description of the technical solutions of the present invention. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0066] Unless otherwise defined, the technical or scientific terms used in the present invention shall have the usual meanings understood by persons of ordinary skill in the field to which the present invention belongs. The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "a" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship also changes accordingly.

[0067] Example 1

[0068] See Figure 1 The present application provides a method for constructing an entity extraction agent based on a large language model, including:

[0069] Obtain entity label files for a specific field, classify and slice them, convert label names and corresponding entity words into query vectors, organize them into a label vector library, and finally merge them into a knowledge base;

[0070] Write prompt words according to the preset paradigm, where prompt words include: role definition, task content, Chinese and English labels, extraction instructions and standard output format;

[0071] Select an adapted large language model from the model library, evaluate the accuracy and recall of the candidate models using a test sample set, and select the optimal model based on the evaluation results;

[0072] Develop tools based on downstream task requirements to complete service URL accessibility testing, tool usability testing, and automated call process verification;

[0073] Input the knowledge base and prompt words into the large language model for entity extraction, call the tool to process the extraction results, verify the correctness of the output, and iteratively optimize the configuration of each stage based on the debugging results.

[0074] like Figure 2 As shown, in this embodiment, the construction method may specifically include the following steps:

[0075] (1) Knowledge base acquisition phase: First, you need to obtain files related to entity extraction. These files must include both Chinese and English label information, as well as all entity words corresponding to each label. There is no restriction on the file type of these files, which can be text files, table files, etc. After obtaining these files, first classify the files belonging to different labels, slice the classified files, and then convert the label names and corresponding entity words into query vectors. , organize the query vectors of each label together The label vector library is composed of all label vector libraries. When combined, it forms the knowledge base. The knowledge base is the knowledge reserve for the large language model to process tasks in related fields. It helps the large language model learn the characteristics of each label. Only after the large language model acquires and learns the knowledge base can it extract entities from user questions in specific fields.

[0076] (2) Prompt word writing stage: Prompt words are small text instructions for the large language model to extract entities, which can be adjusted according to the requirements of downstream task processing. The writing of prompt words is not arbitrary and needs to follow a certain paradigm. First, the role played by the large language model needs to be given, such as "#role: You are a rigorous expert in the field of entity extraction". Then, the content of the entity extraction task needs to be defined. The task content should be adjusted according to the actual usage requirements. It starts with "#task:" and the specific extraction task is written after the colon. It should be noted that the task content should be summary and not described in detail. Next, the extracted Chinese label needs to be given (if there is an English label, it should also be given). This step is very critical and is related to the success rate of the large language model in performing entity extraction tasks. The paradigm content of this part starts with "#Chinese label and its English name:", and then the actual label information is added after the colon. Different labels are separated by commas, such as "call time corresponds to current_time, personnel code corresponds to user_id". After all the above information is given, the specific extraction instruction should be given directly. The content of the instruction can be flexibly changed according to the needs. The format requirement is to start with "#Instructions for entity extraction:", and add actual instructions after the colon. A specific example is "1. Work strictly according to the instructions and output strictly according to the standard format. 2. If no entity words are extracted, simply output empty. 3. After learning the knowledge base content, perform entity extraction." The last thing that needs to be stated is the standard output format, which is related to the user experience of the system, so it is best to design the final output format in advance. The prompt word in this part must start with "#Standard output:", and then write the content according to the actual required format.

[0077] (3) Model selection stage: The model used in this system can be flexibly selected according to the actual application field, and there are no major restrictions on the selection and use of models. The design architecture of this system is compatible with different large language models, that is, people in different fields can use different models to build intelligent agents. Models need to be selected from the model library, which is a collection of APIs that builders can call models. In order to find the model that is most suitable for the entity extraction task in this field, it is recommended to test all models in the model library. The test steps are as follows: First, the actual user of the system writes a test sample library. The writing rule of the test sample is to ensure that the questions with the test sample as input have correct (but not unique) answers in the application field. After the test sample library is written, random sampling can be performed in the test sample library according to a certain ratio. The selection of the ratio can also be determined according to actual needs. After the selection stage is completed, it is automatically merged into a test sample set. Then the questions in the test sample set are used as inputs of the large language model one by one, and the results of running different models on the same test sample set are recorded, that is, the final entity extraction results. After comparing with the correct answer and calculating, important indicators such as accuracy and recall rate are obtained, and finally the model with the best effect is selected for use.

[0078] (4) Tool development stage: The downstream tasks that personnel in different fields need to complete are different, which also brings certain challenges to the construction of intelligent agents. The difference in tasks directly determines the need to use different tools, so the intelligent agent must be compatible with different forms of tools. Tools are changed according to the downstream task requirements of a specific field. Based on this feature, the intelligent agent created by the present invention has weak restrictions on the use of tools. After the staff in a specific field independently develop the tool according to the needs of their own field, they can first perform a service URL accessibility test, that is, perform a accessibility test on the service URL specified in the configuration file. The standard for passing the test is that the URL can respond successfully, and the response status code is 200, and the returned data format meets the predetermined requirements. Then perform a tool usability test, that is, the tool can execute the specified function as expected, and no exceptions are thrown during the execution process, and the function execution result meets the expected effect. After passing the two tests, it is also necessary to verify its automated call process, which must be performed after completing all the above steps.

[0079] (5) Agent debugging phase: The agent will first read the provided knowledge base as a knowledge reserve for handling domain problems, and use the written prompt words as prompts for the large language model. Then, the selected large language model will be used to perform entity extraction tasks, obtain the entity extraction results, and store them in the agent's workflow. However, the entity extraction results will not be output on the front-end interface, but will be directly used as input to the downstream tool to directly call the tool and obtain the final result after the problem is solved. In other words, the final result output by the front-end interface is the answer to the question asked by the user. If the entire process runs correctly and the agent can return the correct answer to the corresponding question, the debugging phase ends. If the returned result is incorrect, return to step (1) and rebuild. If the returned result does not conform to the format, return to step (3) and rebuild. If the returned result is the entity extraction result, return to step (4) and rebuild.

[0080] Example 2

[0081] To use an agent on the Dify platform, first create a new, blank application on the Dify platform—Agent. Business personnel also need to segment ten text files, such as "operation type.txt" and "system platform.txt," to form query vectors {<operation type, initiated rollback>}, {<system platform, cloud video platform>}. Then, group query vectors for all label categories together to form a label vector library: {<operation type, initiated rollback>, <system platform, cloud video platform>}. Next, proceed to the second step—writing prompt words according to the paradigm. According to the actual needs of telecommunications business, the prompt words written according to the paradigm are "#Role: You are a little assistant in the field of rigorous entity extraction, but you are only allowed to extract the content in []. The other content is just instructions. Instructions do not need to be extracted. Do not be disturbed by irrelevant information. Strictly obey all instructions. One entity word belongs to only one label, and repeated extraction is not allowed. #Task: Extract the entity words in [] that belong to specific labels. #Chinese labels and their English names: The call start time corresponds to api_inst_create_date_start, the call end time corresponds to api_inst_create_date_end, the operation type corresponds to oper_type, the order status corresponds to order_state, the personnel code corresponds to user_id, the system platform corresponds to error_link_type_value, the large order number corresponds to cust_order_code, and the small order number corresponds to service_order_code. #Instructions for entity extraction: 1. Strictly follow the instructions and output in a standard format. 2. If no entity words are extracted, the output should be empty. 3. The business code entity word must contain uppercase English letters. 4. If time range information appears but is not specific to a specific day, it needs to be extracted into two parameters: call time and callback time. 5. If information about a specific day appears, extract it as "order creation time." A label can only have one entity word at most. If the customer does not mention the year or month, use the current system time to reasonably complete it. #Standard output: Only when a tag extracts an entity word will the English label and its entity word be added to the output. Tags without entity words are directly ignored in the output and are not allowed to appear in the output. After writing the prompt words, proceed to the third step - selecting a model. First, you need to query the telecom model library to see which models are available. After querying, you know that the available models are "Qwen, Qwen-2b, Chatglm4." The business personnel first write a test sample library, for example, "I need to query all business orders in March; I want to know all business status of employee No. 4031; give me all orders from the cloud video platform directly."Next, we can use the test sample library to test the three available models and obtain important indicators such as accuracy and recall rate for each model in this extraction task. After comparison, the Qwen large model was finally selected to handle this task. Since the downstream task that the telecommunications party wants to perform is report query, the intelligent agent needs to be equipped with a report query tool. This tool uses the entity words extracted by the large language model as parameter input, calls the report query capability interface, and then returns the queried URL information. Click the URL link to download the report and obtain the desired data. At this time, the prompt words can be adjusted according to the response speed of the intelligent agent. The prompt words can be deleted or added, and everything is subject to the final effect. After the final version of the prompt words is determined, the construction process of the intelligent agent is complete. Figure 3 The following is the process of using the agent of the present invention. The user only needs to enter the sentence they want to extract in the dialog box and click send. The agent will then return the result processed by the large language model and tools.

[0082] Example 3

[0083] To use the agent on a telecom platform, sales personnel must first shard five files, including the order status.xlsx and employee code.xlsx files, to create query vectors {<order status, CancelN>}, {<personnel code, 4031>}. They then group query vectors for all label categories together, forming a label vector library. Next, they proceed to the second step: creating prompt words according to the paradigm. According to the actual needs of telecommunications services, the prompt words written according to the paradigm are "Robot: Hello, this is the Hunan Big Data Analysis Center. We are analyzing data information generated in the Hunan system and need your help. #Role: You are a rigorous assistant in the field of entity extraction, but you are only allowed to extract specified content. Other content is just instructions. Instructions do not need to be extracted. Do not be disturbed by irrelevant information. Strictly obey all instructions. One entity word belongs to only one label, and repeated extraction is not allowed. #Task: Extract the entity words belonging to specific labels in the specified content. #Chinese labels and their English names: The call start time corresponds to api_inst_create_date_start, the call end time corresponds to api_inst_create_date_end, the operation type corresponds to oper_type, the order status corresponds to order_state, the personnel code corresponds to user_id, the system platform corresponds to error_link_type_value, the large order number corresponds to cust_order_code, and the small order number corresponds to service_order_code #Instructions for entity extraction:

[0084] 1. Divergent thinking is strictly prohibited. Work strictly according to instructions and output strictly according to the standard format. Do not organize your own language.

[0085] 2. If no entity words are extracted, the output can be empty and do not edit it.

[0086] 3. The cust_order_code corresponding to the large order number is a pure number starting with "8".

[0087] 4. The service_order_code corresponding to the small order number is a pure number starting with "7".

[0088] 5. If a time range appears but doesn't specify a specific day, extract it into two parameters: the call time and the callback time. For example, if "May" appears but doesn't specify a specific day, extract the "call time" as "2024-05-01" and the "callback time" as "2024-05-31."

[0089] 6. If a specific date appears, extract it as "Order Creation Time." A tag can only have one entity word. If the customer does not mention the year or month, use the current system time to fill in the gaps. For example, if the date "November 30th" is mentioned in the conversation, extract the order creation time as the current system year "2024-11-30."

[0090] 7. The entity word in the operation type label only needs to be perfectly matched with the two strings "initiate rollback" and "force return order". A perfect match means that every word is exactly the same, and only the corresponding code "10R" or "1FF" is retained.

[0091] 8. The entity word of the personnel code is the personnel's name.

[0092] 9. The entity words of order status do not contain Chinese characters. "Force return" and "Initiate return" are irrelevant to the order status and errors are not allowed.

[0093] It only needs to perfectly match "10C", "10CE", "10F", "10N", "CancelN", "CancelF", "FAILURE", "Rollback", "RollbackN", and "AwaitOrder". If the specified content is "Query all business orders in May", the output sample is:

[0094] "api_inst_create_date_start=2024-05-01|api_inst_create_date_end=2024-05-31""

[0095] After developing the prompt words, the third step—model selection—emerges. First, the available models in the telecom model library are searched. The query reveals the following three models: "Qwen, Qwen-2b, and Chatglm4." The salesperson first creates a test case library, such as "I need to query all business orders from December of last year; I want to know all the business status of employee number 1000; give me the work order records for the cloud video platform." The three available models are then tested using the test case library to determine their accuracy, recall, and other key metrics for the extraction task. After comparison, the Qwen large model was ultimately selected for this task. The telecom operator wants to use an intelligent agent to summarize work performance, so an intelligent summary tool is required. This tool takes the entity words extracted by the large language model as input, calls the intelligent summary capability interface, and returns the queried URL. Clicking the URL link downloads the report, obtaining the desired data. The large language model then uses the retrieved data as input to summarize the work performance. The prompt words can then be adjusted based on the agent's response speed until satisfactory results are achieved. After determining the final prompt words, package all capabilities and upload them to the telecommunications platform and successfully call the capability on the front-end page. At this point, the creation of the intelligent agent is completed.

[0096] The present application also provides an entity extraction agent construction system based on a large language model, the system comprising:

[0097] Knowledge base module, used to store the label vector library of a specific field;

[0098] The prompt word module is used to generate entity extraction instructions according to the preset paradigm;

[0099] Model adaptation module, used to select an adapted large language model from the model library;

[0100] Tool integration module, used to connect to downstream task processing tools and verify their availability;

[0101] The execution engine module is used to call the large language model to extract entities and input the results into the tool to complete downstream tasks;

[0102] The debugging and optimization module is used to iteratively optimize the knowledge base, prompt words, large language model, and tool configuration based on the output results.

[0103] Optionally, the tool integration module supports dynamic access to tools in multiple fields, and specifies the service URL and data format through a configuration file.

[0104] Optionally, the output result of the execution engine module is the final processing result of the downstream task, and the front-end interface directly displays the answer to the user's question, hiding the intermediate entity extraction process.

[0105] Optionally, the system is compatible with different large language model APIs and can expand the model library and tool library through a plug-in mechanism.

[0106] This embodiment provides an entity extraction intelligent agent construction system based on a large language model, which can implement the various embodiments of the above-mentioned entity extraction intelligent agent construction method based on a large language model and achieve the same beneficial effects. It will not be elaborated here.

[0107] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for constructing an entity extraction agent based on a large language model, characterized in that: include: Obtain entity label files for a specific field, classify and slice the entity label files, convert the label names and entity words corresponding to the label names in the entity label files into query vectors, organize the query vectors into a label vector library, and finally merge the label vector library into a knowledge base; Write prompt words according to the preset paradigm, where prompt words include: role definition, task content, Chinese and English labels, extraction instructions and standard output format; Selecting a large language model adapted to the specific domain from a model library, generating a test sample set based on the specific domain, evaluating the accuracy and recall of the selected large language model using the test sample set, and selecting the optimal model based on the evaluation results; Develop tools based on the downstream task requirements of the specific field, and conduct service URL testing, usability testing, and automated call process verification on the developed tools. Once all tests and verifications pass, the developed tools will be considered the optimal tools. Inputting the knowledge base and prompt words into the optimal model to extract entities, calling the optimal tool to process the results of the entity extraction, verifying the correctness of the output, and constructing the optimal model and the optimal tool into an intelligent agent when the correctness meets the requirements; Generate a test sample set based on the specific domain, including: Compile a test sample library containing domain questions and standard answers in the specific domain; Randomly sampling the test sample library according to a preset ratio to generate a test sample set; The selected large language model is evaluated for accuracy and recall using a test sample set, and the optimal model is selected based on the evaluation results, including: Input the domain questions in the test sample set into the selected large language model one by one and record the operation results of the large language model; The running results of the large language model are compared with the standard answers in the test sample set, the accuracy and recall of the running results and the standard answers are calculated, and the large language model with the highest accuracy and recall is regarded as the optimal model.

2. The entity extraction agent construction method based on a large language model according to claim 1 is characterized in that: The entity tag file includes: a text file or a table file.

3. The entity extraction agent construction method based on a large language model according to claim 1 is characterized in that: The preset paradigms include: The default pattern for defining a character's prompt is: a model character starting with the pound sign and the character name; The default prompt format for the task content is: an entity extraction task that begins with the pound sign and the task name; The default format for prompts for Chinese and English labels is: a mapping relationship between example labels starting with the pound sign plus the Chinese label and its English name; The default pattern of the prompt word for the extraction instruction is: the extraction rule starts with the pound character plus the instruction for entity extraction; The default format for the prompt word of the standard output format is: the output format starts with the pound sign and standard output.

4. The entity extraction agent construction method based on a large language model according to claim 1 is characterized in that: The automated call flow verification of the tool includes: Ensure that the service URL response status code is 200 and the returned data format meets the requirements; Verify that the tool performs its functions correctly and the results are as expected; Integrate tools into agents and validate end-to-end call flows.

5. The entity extraction agent construction method based on a large language model according to claim 1 is characterized in that: The process of verifying the correctness of the output includes: verifying the output result, verifying the output format and verifying the tool processing result; When a question in the specific field is input, if the output result is a correct answer, then the output result is verified to be correct; if the output correct answer conforms to the format, then the output format is verified to be correct; if the output correct answer is not an entity extraction result, then the processing result of the verification tool is correct; Correctness meets the requirements when the verification output is correct, the verification output format is correct, and the verification tool processes the result correctly; When there are inaccuracies in the verification output results, verification output format, and verification tool processing results, if the output result is wrong, reconfigure the knowledge base; if the output format does not match, adjust the model or prompt words; if the output result is an entity extraction result, redevelop the tool.

6. A system for constructing an entity extraction agent based on a large language model, for executing the method for constructing an entity extraction agent based on a large language model as claimed in any one of claims 1 to 5, characterized in that: The system comprises: The knowledge base module is used to store the label vector library and knowledge base of a specific field; The prompt word module is used to generate entity extraction instructions according to the preset paradigm; Model adaptation module, used to select an adapted large language model from the model library; Tool integration module, used to connect to downstream task tools and verify the tools; The execution engine module is used to call the large language model to extract entities based on the label vector library and entity extraction instructions, and input the entity extraction results into the tool to complete downstream tasks; The debugging and optimization module is used to iteratively optimize the knowledge base, prompt words, large language model, and tool configuration based on the output results.

7. The entity extraction agent construction system based on a large language model according to claim 6 is characterized in that: The tool integration module includes a tool development submodule and a tool verification submodule; The tool development submodule is used to develop tools according to the downstream task requirements of the specific field; The tool verification submodule is used to perform service URL testing, usability testing, and automated call process verification on the developed tools. When all tests and verifications pass, the developed tools will be used as tools for connecting to downstream tasks.

8. The entity extraction agent construction system based on a large language model according to claim 6 is characterized in that: The execution engine module includes a model processing submodule and a result output submodule; The model processing submodule is used to input the knowledge base and prompt words into the large language model to extract entities; The result output submodule is used to call the tool to process the result of entity extraction and output the result as an output result.

9. The entity extraction agent construction system based on a large language model according to claim 6 is characterized in that: The debugging and optimization module includes a verification submodule and an iterative debugging submodule; The verification submodule is used to verify the correctness of the output results in the execution engine module, and to construct the large language model and tools into an intelligent agent when the correctness meets the requirements; The iterative debugging submodule is used to send iterative instructions to the knowledge base module when the correctness does not meet the requirements. The knowledge base module reconfigures the label vector library and the knowledge base based on the iterative instructions, and / or sends iterative instructions to the prompt word module. The prompt word module regenerates the entity extraction instructions based on the iterative instructions, and / or sends iterative instructions to the model adaptation module. The model adaptation module reselects the adapted large language model based on the iterative instructions, and / or sends iterative instructions to the tool integration module. The tool integration module redevelops the tool based on the iterative instructions.

Images