Methods, apparatus, and devices for optimizing prompt words based on AI, RPA, LLM, and AI Agent.

Abstract

This application proposes a method, apparatus, and device for optimizing prompt words based on AI, RPA, LLM, and AI Agent, relating to the fields of AI, RPA, and AI Agent. It includes: acquiring structured prompt word fields and information extraction requirements corresponding to the document to be processed; generating first prompt words for LLM based on the prompt word fields and information extraction requirements; performing an information extraction task on the document to be processed using LLM according to the first prompt words to obtain the information extraction results; evaluating the information extraction results to obtain the evaluation results; and optimizing the first prompt words based on the evaluation results to obtain second prompt words for LLM. This application achieves global optimization of the first prompt words, improving the efficiency and accuracy of prompt word optimization, and obtaining prompt words with global consistency and high business relevance, which is beneficial for improving the accuracy and consistency of information extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of Artificial Intelligence (AI), Robotic Process Automation (RPA), Large Language Model (LLM) and Digital Employee Platform Artificial Intelligence Agent (AI Agent), in particular to a prompt word optimization method, device and equipment based on AI, RPA, LLM and AI Agent. BACKGROUND

[0002] Robotic Process Automation (RPA) is a specific "robotic software" that simulates human operations on a computer and automatically performs process tasks according to rules.

[0003] Artificial Intelligence (AI) is a technology science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0004] Large Language Model (LLM) is a model trained by massive text, which can recognize human language, perform language tasks and has a large number of parameters.

[0005] Artificial Intelligence Agent (AI Agent) can perceive the environment, make decisions and perform actions. The difference from traditional artificial intelligence is that it has the ability to think and act independently, and can call tools to complete the given target. AI Agent takes LLM as its core computing engine, so it can have conversations, perform tasks, reason and exhibit a certain degree of autonomy. It has the ability to understand, perceive, plan, remember and use tools, and can automatically execute complex tasks. Specifically, the LLM-driven AI Agent, which is a collection of multiple AI capabilities, can interact with employees in natural language, understand their instructions and needs, and provide feedback and replies; it can master domain knowledge related to business to complete complex professional work; it can break down complex tasks into several executable tasks and use data and tools to complete the tasks; it can also collaborate with employees, and AI Agents can also collaborate to complete complex work, enabling digital employees to make the leap from automation to intelligence, helping employees complete work more efficiently, and fully realizing human-machine collaboration.

[0006] In the related art, the content of the prompt word is usually optimized in a manual-led and serial processing manner. However, in the face of a large number of documents of a document processing platform, completely relying on manual and serial "trial and error" type optimization of the content of the prompt word makes the optimization period of the prompt word long and costly, and it is difficult to scale up to cope with complex business scenarios. SUMMARY

[0007] Embodiments of the present application provide a prompt word optimization method, device and equipment based on AI, RPA, LLM and AI Agent to solve one of the technical problems existing in the related art, and the technical solution is as follows:

[0008] In a first aspect, the embodiments of the present application provide a prompt word optimization method based on AI, RPA, LLM and AI Agent, comprising: obtaining a structured prompt word field and information extraction requirements corresponding to a to-be-processed document, generating a first prompt word of an LLM based on the prompt word field and the information extraction requirements; performing an information extraction task on the to-be-processed document according to the first prompt word by the LLM to obtain an information extraction result of the to-be-processed document; evaluating the information extraction result to obtain an evaluation result of the information extraction result; and optimizing the first prompt word according to the evaluation result to obtain a second prompt word of the LLM.

[0009] In an implementation, the evaluating the information extraction result to obtain an evaluation result of the information extraction result comprises: determining a target rule set matched with the to-be-processed document from a pre-constructed field relationship knowledge base based on a business type of the to-be-processed document; and evaluating the information extraction result based on the target rule set to obtain the evaluation result of the information extraction result.

[0010] In an implementation, the construction process of the field relationship knowledge base comprises: performing an information extraction task on a sample document according to a sample prompt word by the LLM to obtain a sample information extraction result of the sample document; obtaining standard information of the sample document; analyzing the standard information and the sample information extraction result, and constructing the field relationship knowledge base based on an analysis result.

[0011] In an implementation, the constructing the field relationship knowledge base based on the analysis result comprises: determining an abnormal extraction rule of a prompt word field in the sample prompt word and a logical constraint rule between prompt word fields based on the analysis result; and constructing the field relationship knowledge base based on the abnormal extraction rule and the logical constraint rule.

[0012] In an implementation, the evaluating the information extraction result based on the field relationship knowledge base comprises: extracting a first information extraction result corresponding to each prompt word field from the information extraction result; and evaluating the first information extraction result corresponding to each prompt word field based on the target rule set to obtain the evaluation result of the information extraction result.

[0013] In an implementation, the optimizing the first prompt word based on the evaluation result to obtain the second prompt word of the LLM comprises: in response to the evaluation result indicating that the information extraction result is abnormal, determining constraint information of the first prompt word based on the abnormal information extraction result; and optimizing the first prompt word based on the constraint information to obtain the second prompt word of the LLM.

[0014] In an implementation, the determining the constraint information of the first prompt word based on the abnormal information extraction result comprises: determining a target prompt word field corresponding to the abnormal information extraction result, obtaining first constraint information corresponding to the target prompt word field and second constraint information of the information extraction requirement; and determining the constraint information of the first prompt word based on the first constraint information and the second constraint information.

[0015] In an implementation, after obtaining the second prompt word of the LLM, the method further comprises: performing an information extraction task on a to-be-processed document based on the second prompt word by using the LLM to obtain a target information extraction result of the to-be-processed document.

[0016] In a second aspect, an embodiment of the present application provides a prompt word optimization device based on AI, RPA, LLM and AI Agent, comprising: an obtaining module configured to obtain a structured prompt word field corresponding to a to-be-processed document and an information extraction requirement, and generate a first prompt word of an LLM based on the prompt word field and the information extraction requirement; a processing module configured to perform an information extraction task on the to-be-processed document based on the first prompt word by using the LLM to obtain an information extraction result of the to-be-processed document; an evaluation module configured to evaluate the information extraction result to obtain an evaluation result of the information extraction result; and an optimization module configured to optimize the first prompt word based on the evaluation result to obtain a second prompt word of the LLM.

[0017] In an implementation, the evaluation module is further configured to: determine a target rule set matched with the to-be-processed document from a pre-constructed field relationship knowledge base based on a business type of the to-be-processed document; and evaluate the information extraction result based on the target rule set to obtain the evaluation result of the information extraction result.

[0018] In one implementation, the process of constructing the field relationship knowledge base includes: performing an information extraction task on a sample document using the LLM based on sample prompt words to obtain sample information extraction results of the sample document; obtaining standard information of the sample document; analyzing the standard information and the sample information extraction results; and constructing the field relationship knowledge base based on the analysis results.

[0019] In one implementation, the process of constructing the field relationship knowledge base includes: determining the anomaly extraction rules for the prompt word field in the sample prompt words and the logical constraint rules between the prompt word fields based on the analysis results; and constructing the field relationship knowledge base based on the anomaly extraction rules and the logical constraint rules.

[0020] In one implementation, the evaluation module is further configured to: extract a first information extraction result corresponding to each prompt word field from the information extraction result; evaluate the first information extraction result corresponding to each prompt word field based on the target rule set, and obtain an evaluation result of the information extraction result.

[0021] In one embodiment, the optimization module is further configured to: respond to the evaluation result indicating that the information extraction result is abnormal, determine the constraint information of the first prompt word based on the abnormal information extraction result; and optimize the first prompt word based on the constraint information to obtain the second prompt word of the LLM.

[0022] In one embodiment, the optimization module is further configured to: determine the target prompt word field corresponding to the abnormal information extraction result; obtain the first constraint information corresponding to the target prompt word field and the second constraint information of the information extraction requirement; and determine the constraint information of the first prompt word based on the first constraint information and the second constraint information.

[0023] In one embodiment, after obtaining the second prompt word of the LLM, the device is further configured to: perform an information extraction task on the document to be processed by the LLM according to the second prompt word, and obtain the target information extraction result of the document to be processed.

[0024] Thirdly, embodiments of this application provide an electronic device comprising a memory and a processor. The memory and the processor communicate with each other via an internal connection path. The memory stores instructions, and the processor executes the instructions stored in the memory. When the processor executes the instructions stored in the memory, it causes the processor to perform the method described in any of the above embodiments.

[0025] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium storing a computer program, when the computer program is executed on a computer, the method in any one of the embodiments of the above aspects is executed.

[0026] In a fifth aspect, the embodiments of the present application provide a computer program product comprising a computer program, when the computer program is executed by a processor, the method in any one of the embodiments of the above aspects is implemented.

[0027] The advantages or beneficial effects of the above technical solutions at least include:

[0028] By obtaining the structured prompt word field and information extraction requirement corresponding to the to-be-processed document, generating the first prompt word of the LLM based on the prompt word field and the information extraction requirement, executing the information extraction task on the to-be-processed document by the LLM according to the first prompt word to obtain the information extraction result of the to-be-processed document, evaluating the information extraction result to obtain the evaluation result of the information extraction result, and optimizing the first prompt word from a global perspective according to the evaluation result, the autonomous dynamic optimization of the prompt word is realized, the efficiency and accuracy of the prompt word optimization are improved, the prompt word with global consistency and high business fit is obtained, and the accuracy and consistency of information extraction are improved. In addition, the accuracy and efficiency of prompt word optimization can be improved based on AI, RPA, LLM and AI Agent technology.

[0029] The above summary is only for the purpose of the description and is not intended to limit in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent to those skilled in the art by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0030] In the drawings, like reference numerals refer to same or similar functionalities throughout the several views. The drawings are not necessarily to scale. It is to be understood that these drawings only depict several embodiments in accordance with the disclosure and should not be considered to be limiting.

[0031] Figure 1 is a flowchart of a prompt word optimization method based on AI, RPA, LLM and AI Agent provided by one embodiment of the present application;

[0032] Figure 2 is a flowchart of a prompt word optimization method based on AI, RPA, LLM and AI Agent provided by another embodiment of the present application;

[0033] Figure 3is a flowchart of a prompt word optimization method based on AI, RPA, LLM and AI Agent provided by another embodiment of the present application;

[0034] Figure 4 is a flowchart of a prompt word optimization method based on AI, RPA, LLM and AI Agent provided by another embodiment of the present application;

[0035] Figure 5 is a flowchart of a prompt word optimization method according to an embodiment of the present application;

[0036] Figure 6 is a structural diagram of a prompt word optimization device based on AI, RPA, LLM and AI Agent provided by an embodiment of the present application;

[0037] Figure 7 shows a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, in which the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are for the purpose of explanation only, and are not to be understood as limiting the present application.

[0039] With the deepening of large language models in enterprise-level applications, prompt word engineering has become a key link in information extraction tasks. In Agentic Document Processing (ADP), it is extremely dependent on high-quality prompt words to guide model behavior to accurately extract interrelated multi-field information from complex unstructured documents such as business contracts and trade invoices.

[0040] In related technologies, the optimization of prompt words has a fundamental defect: the optimization of prompt words is local and isolated, only aiming to improve the extraction accuracy of a single field, completely lacking global insight and modeling of the inherent business logic and constraint relationship between fields. This directly leads to frequent logical conflicts between fields in the output results of large models that violate business common sense, for example: correctly identifying “Party A country: China”, but incorrectly corresponding “Settlement Currency: USD”, which seriously restricts the consistency, credibility and business usability of the overall data. At the same time, completely relying on artificial, serial “trial and error” debugging makes the prompt word optimization cycle long and costly, making it difficult to scale to complex business scenarios.

[0041] In the related art, the optimization of prompt words mainly relies on the following methods: (1) optimization process based on series and isolation: after in-depth analysis of the requirements by prompt word engineers or business experts, the "one by one" method is used to write, test and optimize the prompt words for each field independently. This process is essentially an "trial and error" based on experience. When the overall extraction effect does not meet the standard, the team needs to repeatedly review and iterate the field prompt words with the lowest accuracy until all fields meet the preset standard. The whole process lacks systematic analysis of field association; (2) relying on expert experience: the entire optimization link, from problem diagnosis to the generation of solutions (prompt word modification), completely relies on human experience. This not only leads to low efficiency, but more seriously, it is difficult for humans to systematically find, define and solve complex global logical constraint problems across fields, making the optimization effect have an upper limit.

[0042] The above technical solutions have the following defects: (1) local optimization defects: the optimization strategy only focuses on improving the extraction accuracy of a single field, lacks a "global perspective", and cannot model and integrate the inter-field business constraint relationship in the prompt words. The output results of the large model may be accurate in a single field, but from the overall point of view, the logic is chaotic and contradictory, for example: the output "payment method: monthly settlement", but "account period: 0 days", which seriously affects the use of downstream business systems; (2) lack of conflict detection and processing mechanism: traditional evaluation modules only focus on comparing the extraction value of each field with the standard answer, without establishing an effective mechanism to check and ensure the logical consistency between the output results of multiple fields. For inter-field conflicts that violate business rules, the system cannot automatically identify them, let alone repair them; (3) high user (domain expert) participation threshold and cost: the system fails to provide a convenient way for users with core business knowledge to inject high-quality evaluation data and rules in bulk. This leads to extremely low efficiency of the system in learning complex business logic, long prompt word optimization cycle, difficulty in quickly adapting to business changes, and inability to form an efficient human-machine collaborative optimization closed loop.

[0043] To solve at least one problem existing in the related art, embodiments of the present application propose a prompt word optimization method, device and equipment based on artificial intelligence (AI), robotic process automation (RPA), a large language model (LLM) and a digital employee platform artificial intelligence agent (AI Agent). These and other aspects of embodiments of the present application will become clear from the following description and drawings. In these descriptions and drawings, some specific implementations of embodiments of the present application are specifically disclosed to represent some ways to implement the principles of embodiments of the present application, but it should be understood that the scope of embodiments of the present application is not limited thereto. On the contrary, embodiments of the present application include all variations, modifications and equivalents falling within the spirit and scope of the appended claims.

[0044] Before specifically describing the embodiments of the present application, first, the commonly used technical terms are introduced for easy understanding:

[0045] In the description of the present application, the term "LLM" is a kind of natural language processing model based on deep learning, which mainly has the characteristics of huge model parameters and complex neural network structure, has strong language understanding ability, context perception ability and language generation ability, which can automatically learn useful feature representation from input data and generate related text.

[0046] In the description of the present application, the term "first prompt word" refers to a structured and configurable instruction framework for guiding the information extraction task to be performed by the large language model, and the first prompt word is an initial prompt word that has not been optimized.

[0047] In the description of the present application, the constraint information of the first prompt word refers to a series of conditions and rules attached in the prompt word when using the large language model to perform the information extraction task, which is used to guide, limit and standardize the model output content.

[0048] Referring to the following description and drawings, these and other aspects of embodiments of the present application will become clear. In these descriptions and drawings, some specific implementations of embodiments of the present application are specifically disclosed to represent some ways to implement the principles of embodiments of the present application, but it should be understood that the scope of embodiments of the present application is not limited thereto. On the contrary, embodiments of the present application include all variations, modifications and equivalents falling within the spirit and scope of the appended claims.

[0049] The AI, RPA, LLM and AI Agent-based prompt word optimization method, device and equipment according to the embodiments of the present application are described below with reference to the accompanying drawings.

[0050] Figure 1 is a flowchart of the AI, RPA, LLM and AI Agent-based prompt word optimization method provided by an embodiment of the present application.

[0051] In a possible implementation manner provided by the embodiments of the present application, the AI, RPA and AI Agent-based prompt word optimization method is configured in an AI, RPA, LLM and AI Agent-based prompt word optimization device, which can be applied to any electronic device with computing capability.

[0052] The electronic device can be a personal computer, a mobile terminal, a server or the like, and the mobile terminal can be a mobile phone, a tablet computer, a personal digital assistant or the like, which is a hardware device with various operating systems.

[0053] In another possible implementation manner of the embodiments of the present application, the AI, RPA, LLM and AI Agent-based prompt word optimization method can be applied to an AI Agent, which can run in any electronic device with computing capability.

[0054] In still another possible implementation manner of the embodiments of the present application, the AI, RPA, LLM and AI Agent-based prompt word optimization method can be applied to a digital employee platform (WEP). The digital employee platform has undergone three stages. The first stage is the automation stage: for the RPA stage with low business complexity, the software automation technology is used to realize the automation of the rule-based and pre-definable process work. The second stage is the intelligent stage: the AI extends the boundary of RPA, such as processing unstructured documents and making judgments and decisions based on data. The third stage is the man-machine collaboration stage: the understanding, planning and execution capabilities of the large model are used to complete the automation of complex tasks end-to-end.

[0055] In the man-machine collaboration stage, the digital employee platform is a bridge connecting workers and systems, workers and data, and systems and data, which can: operate complex systems, process various types of data, and interact and collaborate with employees. The digital employee platform can help industries build large model-enabled digital employees (i.e., intelligent agents) to realize the automation, intelligence and man-machine collaboration of business processes.

[0056] The digital employee platform can realize seamless integration of multiple capabilities such as agentic process automation (APA), agentic document processing (ADP), and agentic business insights (ABI), and has five major functions: business understanding, process creation, running anywhere, centralized management and control, and human-machine collaboration. It enables end-to-end intelligent automation of business processes, replacing manual operations, further improving business efficiency, and accelerating digital transformation.

[0057] Among them, ADP is a new generation of platform based on LLM and Vision-Language Model (VLM) combined with AIAgent technology to realize end-to-end automatic processing of documents. It is a new generation of document processing solution based on LLM and AI Agent technology. It is no longer a "tool" that needs to be configured with templates or labeled samples, but an "intelligent agent" that can understand business requirements and plan and execute independently. Traditional document processing systems are "tools": users need to explicitly tell the system "how to do it" (How). ADP is an "agent" (Agent): users only need to tell the system "what to do" (What), and the system can understand, plan and execute independently.

[0058] As shown in Figure 1 The prompt word optimization method based on AI, RPA, LLM and AI Agent can include the following steps S101-S104:

[0059] Step S101: Obtain the structured prompt word field and information extraction requirements corresponding to the to-be-processed document, and generate a first prompt word for the LLM based on the prompt word field and information extraction requirements.

[0060] Among them, the to-be-processed document is any document that needs to be processed for information extraction.

[0061] Optionally, the original document can be accessed from the ADP platform, and the accessed original document can be used as the to-be-processed document.

[0062] Among them, the structured prompt word field is a clear disassembly and definition of key elements in the information extraction task, presented in an organized and standardized manner, providing a clear framework for subsequent prompt word generation.

[0063] It should be noted that each prompt word field includes but is not limited to field name, field type, and field description (specific information extraction rules).

[0064] It should be noted that the structured prompt word field and information extraction requirement corresponding to the document of different business types are often different, and the structured prompt word field and information extraction requirement corresponding to the to-be-processed document can be obtained according to the business type of the to-be-processed document.

[0065] For example, the business type of the to-be-processed document A is a sales contract type, and the structured prompt word field corresponding to the to-be-processed document A at least includes "Party A", "Party B", "total amount", "currency", "payment method", "account period", and for "payment method", the field name is "payment method", the field type is text, and the field description is to extract the payment settlement method agreed in the contract, such as wire transfer, letter of credit, monthly settlement, etc.

[0066] Among them, the information extraction requirement can be understood as the overall information extraction requirement of the to-be-processed document. For example: the information extraction requirement is "please accurately extract all fields"

[0067] Step S102: performing an information extraction task on the to-be-processed document according to the first prompt word through the LLM to obtain an information extraction result of the to-be-processed document.

[0068] In the embodiment of the present application, after the first prompt word is obtained, the first prompt word is combined into a natural instruction, the LLM is called to perform an information extraction task on the to-be-processed document, and information extraction is performed on the to-be-processed document through the LLM to obtain a structured information extraction result of the to-be-processed document. Among them, the LLM can be an information extraction model.

[0069] For example, for the to-be-processed document A, the information extraction result corresponding to the to-be-processed document at least includes the information extraction result corresponding to "Party A", "Party B", "total amount", "currency", "payment method", "account period" (i.e. the field value of the prompt word field).

[0070] Step S103: evaluating the information extraction result to obtain an evaluation result of the information extraction result.

[0071] It should be noted that after the information extraction result is obtained, the information extraction result can be evaluated according to multiple evaluation dimensions to obtain an evaluation result of the information extraction result.

[0072] In the embodiment of the present application, based on the business type of the to-be-processed document, a target rule set matched with the to-be-processed document can be determined from a pre-constructed field relationship knowledge base, and based on the target rule set, the information extraction result is evaluated to obtain an evaluation result of the information extraction result.

[0073] Optionally, based on the target rule set, the information extraction result can be evaluated in combination with multiple evaluation dimensions of the information extraction result to obtain an evaluation result of the information extraction result.

[0074] It should be noted that the evaluation dimensions include but are not limited to the accuracy evaluation dimension and the logical consistency evaluation dimension.

[0075] Step S104: optimizing the first prompt word according to the evaluation result to obtain a second prompt word of the LLM.

[0076] In the embodiments of the present application, after the evaluation result is obtained, it can be determined whether the information extraction result is abnormal according to the evaluation result, and it is determined whether to optimize the first prompt word according to the determination result to obtain the second prompt word of the LLM.

[0077] Optionally, if it is determined according to the evaluation result that the information extraction result is abnormal, which can be understood as that the information extraction result has errors and / or logical conflicts, it indicates that the first prompt word needs to be optimized, and it is determined to optimize the first prompt word to obtain the second prompt word of the LLM.

[0078] Optionally, if it is determined according to the evaluation result that the information extraction result is not abnormal, which can be understood as that the information extraction result has no errors and logical conflicts, it indicates that the first prompt word does not need to be optimized, and the first prompt word is directly used as the second prompt word of the LLM.

[0079] The prompt word optimization method based on AI, RPA, LLM and AI Agent provided in the embodiments of the present application obtains the structured prompt word field corresponding to the to-be-processed document and the information extraction requirement, generates the first prompt word of the LLM based on the prompt word field and the information extraction requirement, executes the information extraction task on the to-be-processed document according to the first prompt word by the LLM to obtain the information extraction result of the to-be-processed document, evaluates the information extraction result, obtains the evaluation result of the information extraction result, optimizes the first prompt word according to the evaluation result, and obtains the second prompt word of the LLM. Thus, the first prompt word is optimized from a global perspective according to the evaluation result, the autonomous dynamic optimization of the prompt word is realized, the efficiency and accuracy of the prompt word optimization are improved, the prompt word with global consistency and high business fit is obtained, and the accuracy and consistency of information extraction are improved. In addition, the accuracy and efficiency of the prompt word optimization can be improved based on AI, RPA, LLM and AI Agent technology.

[0080] Figure 2 is a flowchart of the prompt word optimization method based on AI, RPA, LLM and AI Agent provided in another embodiment of the present application.

[0081] It should be noted that the prompt word optimization method based on AI, RPA, LLM and AI Agent can be executed alone, or can be executed in combination with any embodiment or possible implementation in the present application, or can be executed in combination with any technical solution in the related art, and the present application embodiment does not limit this.

[0082] As shown in the prompt word optimization method based on AI, RPA, LLM and AI Agent can include the following steps S201 to S205: Figure 2

[0083] Step S201, obtaining the structured prompt word field and information extraction requirement corresponding to the to-be-processed document, generating the first prompt word of the LLM based on the prompt word field and the information extraction requirement.

[0084] Step S202, performing an information extraction task on the to-be-processed document according to the first prompt word by the LLM to obtain an information extraction result of the to-be-processed document.

[0085] The explanation and description of steps S201 to S202 can be referred to the related description in any embodiment of the present application, which will not be repeated here.

[0086] Step S203, determining a target rule set matched with the to-be-processed document from a pre-constructed field relationship knowledge base based on the business type of the to-be-processed document.

[0087] In the present application embodiment, the LLM performs an information extraction task on the sample document according to the sample prompt word to obtain a sample information extraction result of the sample document, obtains standard information of the sample document, analyzes the standard information and the sample information extraction result, and constructs the field relationship knowledge base based on the analysis result.

[0088] Optionally, the standard information and the sample information extraction result can be uploaded through a professional data interface, the true values of a plurality of fields pre-labeled in the standard information, the extraction values of the fields in the sample information extraction result and the context thereof are analyzed, and based on the analysis result, the abnormal extraction rules of the prompt word fields and the logical constraint rules between the prompt word fields are automatically mined to construct the field relationship knowledge base.

[0089] In the present application embodiment, based on the analysis result, the abnormal extraction rules of the prompt word fields in the sample prompt word and the logical constraint rules between the prompt word fields are determined, and based on the abnormal extraction rules and the logical constraint rules, the field relationship knowledge base is constructed.

[0090] ​Optionally, an abnormal extraction rule for the prompt word field: that is, to identify whether the field description corresponding to the single prompt word field causes the LLM to miss extraction, mis-extraction, or ambiguity, etc. For example, if the prompt word field "account period" corresponds to the field description "extract the agreed payment period in days", according to the analysis result, it is determined that the LLM misses "natural month settlement", indicating that the field description corresponding to the prompt word field "account period" needs to be expanded.

[0091] Optionally, a logical constraint rule between prompt word fields: that is, to identify whether there is a conflict or dependency between multiple prompt word fields, for example, the information extraction result corresponding to "payment method" is "monthly settlement" and the information extraction result corresponding to "account period" is "45" days, according to the analysis result, it is determined that "account period" is extracted incorrectly, indicating that there is a logical constraint between the prompt word fields "payment method" and "account period".

[0092] Step S204, based on the target rule set, evaluating the information extraction result to obtain an evaluation result of the information extraction result.

[0093] In the embodiments of the present application, the first information extraction result corresponding to each prompt word field is extracted from the information extraction result, and the first information extraction result corresponding to each prompt word field is evaluated based on the target rule set to obtain an evaluation result of the information extraction result.

[0094] Optionally, the first information extraction result corresponding to each prompt word field can be evaluated according to the abnormal extraction rule in the target rule set to evaluate whether the first information extraction result has errors, and according to the logical constraint rule in the target rule set, to evaluate whether the first information extraction result has a logical conflict, to obtain the evaluation result of the information extraction result.

[0095] Optionally, after obtaining the evaluation result of the information extraction result, an evaluation report of the information extraction result can be generated according to the evaluation result of the information extraction result, and the evaluation report can be visually displayed.

[0096] Step S205, according to the evaluation result, optimizing the first prompt word to obtain a second prompt word of the LLM.

[0097] The AI, RPA, LLM and AI Agent-based prompt word optimization method provided in the embodiment of the application obtains a structured prompt word field and information extraction requirement corresponding to a to-be-processed document, generates a first prompt word of an LLM based on the prompt word field and the information extraction requirement, performs an information extraction task on the to-be-processed document according to the first prompt word through the LLM, obtains an information extraction result of the to-be-processed document, determines a target rule set matched with the to-be-processed document from a pre-constructed field relationship knowledge base based on a business type of the to-be-processed document, evaluates the information extraction result based on the target rule set, obtains an evaluation result of the information extraction result, optimizes the first prompt word according to the evaluation result, and obtains a second prompt word of the LLM. In this way, the field relationship knowledge base is pre-constructed, the information extraction result is evaluated according to the target rule set in the field relationship knowledge base, the accuracy and flexibility of evaluating the information extraction result are improved, and data support is provided for subsequent accurate optimization of the prompt word.

[0098] Figure 3 is a flowchart of the AI, RPA, LLM and AI Agent-based prompt word optimization method provided in another embodiment of the application.

[0099] It should be noted that the AI, RPA, LLM and AI Agent-based prompt word optimization method can be executed alone, or can be executed in combination with any one of the embodiments or possible implementation manners in the embodiments, or can be executed in combination with any one of the related technical solutions, and the embodiments of the application do not limit this.

[0100] As shown in Figure 3 , the AI, RPA, LLM and AI Agent-based prompt word optimization method can include the following steps S301 to S306:

[0101] Step S301, a structured prompt word field and information extraction requirement corresponding to a to-be-processed document are obtained, and a first prompt word of an LLM is generated based on the prompt word field and the information extraction requirement.

[0102] Step S302, an information extraction task is performed on the to-be-processed document according to the first prompt word through the LLM, and an information extraction result of the to-be-processed document is obtained.

[0103] Step S303, the information extraction result is evaluated, and an evaluation result of the information extraction result is obtained.

[0104] The explanation and description of steps S301 to S303 can be referred to the related description in any one of the embodiments of the application, which will not be repeated here.

[0105] Step S304: In response to the abnormality in the extraction result of the evaluation result indication information, the constraint information of the first prompt word is determined based on the abnormal information extraction result.

[0106] In this embodiment of the application, the target prompt word field corresponding to the abnormal information extraction result is determined, the first constraint information and the second constraint information of the information extraction requirement corresponding to the target prompt word field are obtained, and the constraint information of the first prompt word is determined based on the first constraint information and the second constraint information.

[0107] It should be noted that in response to anomalies in the information extraction results of the evaluation results, they can be accurately mapped to the structured prompt word field, that is, the target prompt word field corresponding to the abnormal information extraction results can be located.

[0108] For example, if the information extraction result corresponding to the prompt word field "payment period" is missed, which meets the abnormal extraction rules, then "payment period" is the target prompt word field; if the information extraction result corresponding to the prompt word field "payment method" is "monthly settlement" and the information extraction result corresponding to the prompt word field "payment period" is 45 days, there is a logical conflict between the prompt word field "payment method" and the prompt word field "payment period", then "payment period" is the target prompt word field.

[0109] Step S305: Optimize the first prompt word based on the constraint information to obtain the second prompt word for LLM.

[0110] In this embodiment of the application, after obtaining the constraint information, the field description corresponding to the target prompt word field in the first prompt word is optimized according to the constraint information, while keeping the field description corresponding to the remaining prompt word fields in the first prompt word unchanged, so as to obtain the second prompt word of LLM.

[0111] For example, the following is an example of the first prompt words before optimization:

[0112] {

[0113] Field name: "Payment method"

[0114] Field Description: "Extracts the payment and settlement method stipulated in the contract, such as wire transfer, letter of credit, monthly settlement, etc."

[0115] Field name: "Payment period" / / Target prompt field

[0116] Field description: "The agreed payment cycle number of days"

[0117] Overall extraction requirement: "Please extract all fields accurately."

[0118] }

[0119] The optimized examples of some second suggestion words are explained below:

[0120] {

[0121] Field name: "Payment method"

[0122] Field description: "Extract the payment settlement method agreed in the contract, such as wire transfer, letter of credit, monthly settlement, etc."

[0123] Field name: "Account period"

[0124] Field description: "Extract the agreed payment period in days, with special attention that when the payment method is monthly settlement, this field should be 30 days."

[0125] Overall extraction requirements: "Please accurately extract all fields and strictly verify the logical relationship between the payment method and the account period."

[0126] }

[0127] The prompt word optimization method based on AI, RPA, LLM and AI Agent provided by the embodiment of the present application obtains the structured prompt word field and information extraction requirement corresponding to the to-be-processed document, generates a first prompt word of the LLM based on the prompt word field and the information extraction requirement, performs an information extraction task on the to-be-processed document according to the first prompt word through the LLM, obtains an information extraction result of the to-be-processed document, evaluates the information extraction result, obtains an evaluation result of the information extraction result, and in response to the evaluation result indicating that the information extraction result is abnormal, determines constraint information of the first prompt word according to the abnormal information extraction result, optimizes the first prompt word according to the constraint information, and obtains a second prompt word of the LLM. Thus, unlike the related art which rewrites or fuzzily optimizes the prompt word as a whole, the constraint information of the first prompt word is determined through the abnormal information extraction result, the specific prompt word field in the structured prompt word can be accurately located based on the abnormal information extraction result found in the information extraction result, and only this part is targeted for constraint enhancement, thereby realizing accurate optimization of the prompt word, maintaining the stability of the prompt word structure, greatly improving the optimization efficiency and effect of the prompt word, making the optimization process of the prompt word accurate, minimized and interpretable, improving the quality of the prompt word, and being conducive to improving the information extraction effect.

[0128] Figure 4 is a flowchart of the prompt word optimization method based on AI, RPA, LLM and AI Agent provided by another embodiment of the present application.

[0129] It should be noted that the prompt word optimization method based on AI, RPA, LLM and AI Agent can be executed alone, or can be executed in combination with any embodiment or possible implementation in the present application, or in combination with any technical solution in the related art, and the present application embodiment does not limit this.

[0130] As shown in Figure 4 The prompt word optimization method based on AI, RPA, LLM and AI Agent can include the following steps S401-S405:

[0131] Step S401, obtaining a structured prompt word field and information extraction requirement corresponding to the to-be-processed document, generating a first prompt word of the LLM based on the prompt word field and the information extraction requirement.

[0132] Step S402, performing an information extraction task on the to-be-processed document according to the first prompt word by the LLM to obtain an information extraction result of the to-be-processed document.

[0133] Step S403, based on the business type of the to-be-processed document, determining a target rule set matched with the to-be-processed document from a pre-constructed field relationship knowledge base.

[0134] Step S404, based on the target rule set, evaluating the information extraction result to obtain an evaluation result of the information extraction result.

[0135] Step S405, in response to the evaluation result indicating that the information extraction result is abnormal, determining constraint information of the first prompt word according to the abnormal information extraction result.

[0136] Step S406, optimizing the first prompt word according to the constraint information to obtain a second prompt word of the LLM.

[0137] Step S407, performing an information extraction task on the to-be-processed document according to the second prompt word by the LLM to obtain a target information extraction result of the to-be-processed document.

[0138] In the present application embodiment, the first prompt word is optimized to obtain the second prompt word of the LLM, and the LLM continues to perform the information extraction task on the to-be-processed document according to the second prompt word to obtain the target information extraction result of the to-be-processed document.

[0139] Among them, the accuracy of the target information extraction result of the to-be-processed document is higher than that of the information extraction result.

[0140] The prompt word optimization method based on AI, RPA, LLM and AI Agent in the embodiment of the application, by obtaining the structured prompt word field and information extraction requirement corresponding to the to-be-processed document, generating the first prompt word of the LLM based on the prompt word field and the information extraction requirement, performing the information extraction task on the to-be-processed document according to the first prompt word by the LLM to obtain the information extraction result of the to-be-processed document, determining the target rule set matched with the to-be-processed document from the pre-constructed field relationship knowledge base based on the business type of the to-be-processed document, evaluating the information extraction result based on the target rule set, obtaining the evaluation result of the information extraction result, in response to the evaluation result indicating that the information extraction result is abnormal, determining the constraint information of the first prompt word according to the abnormal information extraction result, optimizing the first prompt word according to the constraint information to obtain the second prompt word of the LLM, and performing the information extraction task on the to-be-processed document according to the second prompt word by the LLM to obtain the target information extraction result of the to-be-processed document. Thus, the application executes the information extraction task on the to-be-processed document according to the optimized second prompt word by the LLM to obtain the target information extraction result of the to-be-processed document, improves the accuracy and consistency of information extraction, and improves the user experience. In addition, the accuracy and efficiency of information extraction can be improved based on AI, RPA, LLM and AI Agent technology.

[0141] The specific process of the prompt word optimization method proposed in the application is explained below.

[0142] For example, as shown in Figure 5 (1) Structured prompt word: according to the input information of the user, obtain the structured prompt word field corresponding to the to-be-processed document, and generate the first prompt word of the LLM based on the prompt word field and the information extraction requirement; (2) LLM execution and output: perform the information extraction task on the to-be-processed document according to the first prompt word by the LLM, and output the information extraction result of the to-be-processed document; (3) Batch evaluation of information extraction result and analysis of prompt word field details: batch evaluate the information extraction result output by the large model, and analyze the details of each prompt word field; (4) Check the information extraction result corresponding to the prompt word field: use the field logic verifier to check the information extraction result corresponding to the prompt word field, judge whether it meets the abnormal extraction rule and the logic constraint rule, if it meets the abnormal extraction rule and / or the logic constraint rule, i.e. fails to pass the verification, locate the target prompt word field of the abnormality, if it does not meet the abnormal extraction rule and / or the logic constraint rule, i.e. passes the verification, obtain the first prompt word; (6) Generate field-level optimization suggestion to generate new structured prompt word: generate the field-level optimization suggestion, i.e. constraint information, according to the located target prompt word field, and generate the new structured second prompt word.

[0143] The following takes a sales contract as an example to explain the specific process of the prompt word optimization method proposed in the application.

[0144] (1) Initial setting: upload N sales contracts containing "Party A", "Party B", "total amount", "currency", "payment method", "account period", etc. fields, initialize structured prompt words, perform information extraction once by a large model, calibrate standard answers based on the field results extracted by the large model, and generate verifiable calibration text, i.e. evaluation results. (2) Constraint discovery: the system automatically analyzes the evaluation results to obtain abnormal extraction rules and logical constraint rules; (3) According to the evaluation results, the prompt word fields in the first prompt word are positioned abnormally, the problem is positioned to the field description and overall extraction requirements of the prompt word field, and the optimized prompt word text is directly generated, the constraint description (constraint information) and example are added in the field description, and the optimized prompt word is obtained. After 2-3 rounds of iteration, the optimized prompt word can correctly output the information extraction results that meet all business rules at 100%, and the recall rate of expressions such as "natural month settlement" is significantly improved.

[0145] In summary, the prompt word optimization method based on AI, RPA, LLM and AI Agent of the embodiments of the application fundamentally solves the problem of logical conflict of multiple field information extraction, and the internal consistency and business compliance of the output information extraction result are much higher than that of manual writing or general optimization method. Different from the related art of rewriting or fuzzy optimization of prompt words as a whole, the present application can accurately locate to a specific prompt word field (such as field description or overall extraction requirement) in the structured prompt word based on the problems found in the information extraction result, and only the part is constrained and enhanced, the stability of the prompt word structure is maintained, the optimization efficiency and effect are greatly improved, and the automatic and accurate mapping from problem discovery to solution (specific prompt word text) is realized. The field experts are liberated from the tedious and error-prone rule analysis and prompt word debugging work, and the automation and efficiency of the prompt word are improved. At the same time, the system can continuously learn with the change of business data, constantly discover new field constraints and optimize the prompt word, has strong self-adaptation ability, solves the common pain points in the field of information extraction, can be widely applied to real business scenarios such as intelligent contract review, financial bill processing and resume screening, and has strong landing property.

[0146] In order to realize the above-mentioned embodiments, the application further provides an AI, RPA, LLM and AI Agent-based prompt word optimization device.

[0147] Figure 6 is a structural diagram of an AI, RPA, LLM and AI Agent-based prompt word optimization device provided by an embodiment of the application.

[0148] As Figure 6 shown, the prompt word optimization apparatus 600 based on AI, RPA, LLM and AI Agent includes an acquisition module 610, a processing module 620, an evaluation module 630 and an optimization module 640.

[0149] The acquisition module 610 is configured to acquire a structured prompt word field and information extraction requirements corresponding to a to-be-processed document, and generate a first prompt word of an LLM based on the prompt word field and the information extraction requirements. The processing module 620 is configured to perform an information extraction task on the to-be-processed document according to the first prompt word by the LLM to obtain an information extraction result of the to-be-processed document. The evaluation module 630 is configured to evaluate the information extraction result to obtain an evaluation result of the information extraction result. The optimization module 640 is configured to optimize the first prompt word according to the evaluation result to obtain a second prompt word of the LLM.

[0150] In any one of the embodiments of the present application, the evaluation module 630 is further configured to: determine a target rule set matched with the to-be-processed document from a pre-constructed field relationship knowledge base based on a business type of the to-be-processed document; and evaluate the information extraction result based on the target rule set to obtain the evaluation result of the information extraction result.

[0151] In any one of the embodiments of the present application, the construction process of the field relationship knowledge base includes: performing an information extraction task on a sample document according to a sample prompt word by the LLM to obtain a sample information extraction result of the sample document; acquiring standard information of the sample document; and analyzing the standard information and the sample information extraction result, and constructing the field relationship knowledge base based on an analysis result.

[0152] In any one of the embodiments of the present application, the construction process of the field relationship knowledge base includes: determining an abnormal extraction rule of a prompt word field in the sample prompt word and a logical constraint rule between prompt word fields based on the analysis result; and constructing the field relationship knowledge base based on the abnormal extraction rule and the logical constraint rule.

[0153] In one implementation, the evaluation module 630 is further configured to: extract a first information extraction result corresponding to each prompt word field from the information extraction result; and evaluate the first information extraction result corresponding to each prompt word field based on the target rule set to obtain the evaluation result of the information extraction result.

[0154] In one implementation, the optimization module 640 is further configured to: in response to the evaluation result indicating that the information extraction result is abnormal, determine constraint information of the first prompt word according to the abnormal information extraction result; and optimize the first prompt word according to the constraint information to obtain the second prompt word of the LLM.

[0155] In an implementation, the optimization module 640 is further configured to: determine a target prompt word field corresponding to the abnormal information extraction result, obtain first constraint information corresponding to the target prompt word field and second constraint information of the information extraction requirement, and determine constraint information of the first prompt word according to the first constraint information and the second constraint information.

[0156] In an implementation, after obtaining the second prompt word of the LLM, the device 600 is further configured to: perform, by the LLM, an information extraction task on the to-be-processed document according to the second prompt word, to obtain a target information extraction result of the to-be-processed document.

[0157] It should be noted that the prompt word optimization device based on AI, RPA, LLM and AI Agent provided by the embodiments of the present application can implement all the method steps achieved by any of the method embodiments described above, and achieve the same technical effects. Therefore, the same parts and beneficial effects of the method embodiments in the embodiments will not be described in detail here.

[0158] The prompt word optimization device based on AI, RPA, LLM and AI Agent provided by the embodiments of the present application obtains the structured prompt word field corresponding to the to-be-processed document and the information extraction requirement, generates the first prompt word of the LLM based on the prompt word field and the information extraction requirement, performs, by the LLM, an information extraction task on the to-be-processed document according to the first prompt word, to obtain an information extraction result of the to-be-processed document, evaluates the information extraction result, obtains an evaluation result of the information extraction result, optimizes the first prompt word according to the evaluation result, and obtains the second prompt word of the LLM. Therefore, the embodiments of the present application optimize the first prompt word from a global perspective according to the evaluation result, realize autonomous and dynamic optimization of the prompt word, improve the efficiency and accuracy of the prompt word optimization, obtain a prompt word with global consistency and high business fit, and help to improve the accuracy and consistency of information extraction. In addition, the accuracy and efficiency of the prompt word optimization can be improved based on AI, RPA, LLM and AI Agent technology.

[0159] Figure 7 A structural block diagram of an electronic device according to an embodiment of the present application is shown. As shown in the figure, the electronic device includes a memory 710 and a processor 720, and the memory 710 stores a computer program that can run on the processor 720. The processor 720 implements the prompt word optimization method based on AI, RPA, LLM and AI Agent in the above embodiments when executing the computer program. The number of the memory 710 and the processor 720 can be one or more. Figure 7

[0160] The electronic device further includes:

[0161] ​The communication interface 730 is configured to communicate with external devices and perform data interaction transmission.

[0162] If the memory 710, the processor 720 and the communication interface 730 are independently implemented, the memory 710, the processor 720 and the communication interface 730 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0163] Optionally, in a specific implementation, if the memory 710, the processor 720 and the communication interface 730 are integrated on a chip, the memory 710, the processor 720 and the communication interface 730 can complete communication between each other through an internal interface.

[0164] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the prompt word optimization method based on AI, RPA, LLM and AI Agent provided in any embodiment of the present application.

[0165] The embodiment of the present application also provides a chip, which includes a processor, is used for calling and running instructions stored in a memory from the memory, so that a communication device installed with the chip executes the prompt word optimization method based on AI, RPA, LLM and AI Agent provided in any embodiment of the present application.

[0166] The embodiment of the present application also provides a chip, which includes an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used for executing code in the memory, and when the code is executed, the processor is used for executing the prompt word optimization method based on AI, RPA, LLM and AI Agent provided in any embodiment of the present application.

[0167] It is to be understood that the above-described processor can be a central processing unit (CPU), but can also be other general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general purpose processor can be a microprocessor or any conventional processor, etc. It is to be noted that the processor can be an advanced RISC machine (ARM) architecture processor.

[0168] Further, the above-described memory can include read-only memory and random access memory, and can also include non-volatile random access memory. The memory can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can include random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM can be used. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM) can be used.

[0169] In the above-described embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded on a computer, all or part generates a flow or function according to the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0170] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0171] In addition, the terms "first", "second", etc. are used only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0172] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions or other processes. Also, the preferred embodiments of the present application can include additional implementation with respect to the order in which steps are performed, including the use of simultaneous processes, or the order of steps can be reversed between successive process steps depending on the functionality involved.

[0173] The logic and / or steps represented in flow charts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be specifically embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions execution systems, apparatus or devices.

[0174] It should be understood that each part of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-mentioned embodiment method can be completed by a program instructing the relevant hardware, which can be stored in a computer readable storage medium, and the program includes one or a combination of the steps of the method embodiment when executed.

[0175] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The above-mentioned integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0176] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An artificial intelligence (AI), robotic process automation (RPA), large language model (LLM) and digital employee platform agent (AI Agent) based prompt word optimization method, characterized in that, The method comprises: obtaining a structured prompt field and information extraction requirements corresponding to a to-be-processed document, generating a first prompt for an LLM based on the prompt field and the information extraction requirements; performing an information extraction task on the to-be-processed document by the LLM according to the first prompt to obtain an information extraction result of the to-be-processed document; evaluating the information extraction result to obtain an evaluation result of the information extraction result; optimizing the first prompt according to the evaluation result to obtain a second prompt for the LLM; The evaluation of the information extraction result and the obtaining of the evaluation result of the information extraction result comprise: based on the business type of the to-be-processed document, determining a target rule set matched with the to-be-processed document from a pre-constructed field relationship knowledge base; based on the target rule set, evaluating the information extraction result to obtain an evaluation result of the information extraction result; The construction process of the field relationship knowledge base comprises: performing an information extraction task on a sample document by the LLM according to a sample prompt to obtain a sample information extraction result of the sample document; obtaining standard information of the sample document; analyzing the standard information and the sample information extraction result, and constructing the field relationship knowledge base based on the analysis result; The construction of the field relationship knowledge base based on the analysis result comprises: based on the analysis result, determining an abnormal extraction rule of a prompt field in the sample prompt and a logical constraint rule between prompt fields; based on the abnormal extraction rule and the logical constraint rule, constructing the field relationship knowledge base.

2. The method of claim 1, wherein, The evaluation of the information extraction result based on the field relationship knowledge base and the obtaining of the evaluation result of the information extraction result comprise: extracting a first information extraction result corresponding to each prompt field from the information extraction result; based on the target rule set, evaluating the first information extraction result corresponding to each prompt field to obtain an evaluation result of the information extraction result.

3. The method of claim 1, wherein, The optimization of the first prompt according to the evaluation result to obtain a second prompt for the LLM comprises: in response to the evaluation result indicating that the information extraction result is abnormal, determining constraint information of the first prompt according to the abnormal information extraction result; optimizing the first prompt according to the constraint information to obtain a second prompt for the LLM.

4. The method of claim 3, wherein, The determination of the constraint information of the first prompt according to the abnormal information extraction result comprises: determining a target prompt field corresponding to the abnormal information extraction result, obtaining first constraint information corresponding to the target prompt field and second constraint information of the information extraction requirement; determining the constraint information of the first prompt according to the first constraint information and the second constraint information.

5. The method of any one of claims 1-4, wherein, After obtaining the second prompt for the LLM, the method further comprises: performing an information extraction task on the to-be-processed document by the LLM according to the second prompt to obtain a target information extraction result of the to-be-processed document. 6.A device for prompt word optimization based on AI, RPA, LLM, and AI Agent, characterized by, comprise: An acquisition module is configured to acquire a structured prompt word field and information extraction requirements corresponding to a to-be-processed document, and generate a first prompt word of an LLM based on the prompt word field and the information extraction requirements. A processing module is configured to perform an information extraction task on the to-be-processed document by the LLM according to the first prompt word, and obtain an information extraction result of the to-be-processed document. An evaluation module is configured to evaluate the information extraction result and obtain an evaluation result of the information extraction result. An optimization module is configured to optimize the first prompt word according to the evaluation result and obtain a second prompt word of the LLM. The evaluation module is configured to: determine a target rule set matched with the to-be-processed document from a pre-constructed field relationship knowledge base based on a business type of the to-be-processed document; evaluate the information extraction result based on the target rule set and obtain an evaluation result of the information extraction result. The construction process of the field relationship knowledge base includes: performing an information extraction task on a sample document by the LLM according to a sample prompt word and obtaining a sample information extraction result of the sample document; obtaining standard information of the sample document; analyzing the standard information and the sample information extraction result, and constructing the field relationship knowledge base based on an analysis result; The construction process of the field relationship knowledge base includes: determining an abnormal extraction rule of a prompt word field and a logical constraint rule between prompt word fields in the sample prompt word based on the analysis result; constructing the field relationship knowledge base based on the abnormal extraction rule and the logical constraint rule.

7. An electronic device, comprising: It includes: a processor and a memory, the memory stores instructions, the instructions are loaded and executed by the processor to implement the method in any one of claims 1-5.

8. A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to implement the method in any one of claims 1-5.

Images