LLM-based entity content extraction method and device, electronic equipment and medium

Abstract

The invention provides an LLM-based entity content extraction method and device and a medium, and the method comprises the steps: creating a text embellishing data set, continuously updating a first prompt word through an original text and an embellishing text, and determining a first text embellishing prompt word; based on the text embellishing data set and the first text embellishing prompt, respectively determining a first evaluation data set and a second evaluation data set, and respectively generating first entity content and second entity content through the target LLM and the second prompt; determining a second comprehensive evaluation index of the target LLM, respectively iterating the first evaluation data set and the second evaluation data set, and determining a first entity extraction cue word and a second entity extraction cue word; and determining a target prompt word from the first entity extraction result and the second entity extraction result through the target LLM based on the target data set and the third prompt word. Through the technical scheme provided by the invention, the problem of insufficient information extraction reliability and safety is solved, and the information extraction reliability and safety are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of natural language processing, and in particular to an entity content extraction method, device, electronic device and medium based on LLM. Background Art

[0002] In text-based entity recognition technology, large language models are used for general information extraction. Although they can perform well in specific scenarios such as named entity recognition, their general capabilities will be reduced. In particular, after fine-tuning, the original instruction-following ability of the large language model will be reduced, which makes the scope of use narrower and there are problems with the reliability and security of the information extraction results. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] The present disclosure provides an entity content extraction method, device, electronic device and medium based on LLM, aiming to optimize the existing entity recognition technology to avoid resource consumption caused by fine-tuning of a large model under an entity content extraction task, and to ensure the instruction-following capability of the large language model and the reliability and security of the output results.

[0005] The first aspect of the present disclosure provides an entity content extraction method based on LLM, the method comprising: creating a text polishing dataset, and continuously updating a first prompt using the original text and the polished text in the text polishing dataset, and determining a first text polishing prompt, wherein the polished text is generated based on the original text and the first prompt through a target LLM, and the first text polishing prompt is the first prompt with the highest first comprehensive evaluation index;

[0006] Based on the text polishing dataset and the first text polishing prompt, respectively determine a first evaluation dataset corresponding to the original text and a second evaluation dataset corresponding to the polished text, and respectively generate a first entity content corresponding to the first evaluation dataset and a second entity content corresponding to the second evaluation dataset through the target LLM and the second prompt;

[0007] Determine a second comprehensive evaluation index of the LLM according to the first entity content, the second entity content and the entity label, iterate the first evaluation data set and the second evaluation data set respectively based on the second comprehensive evaluation index, determine a first entity extraction prompt word corresponding to the first evaluation data set and a second entity extraction prompt word corresponding to the second evaluation data set, and the entity label is a preset label corresponding to the first entity content and the second entity content;

[0008] According to the first entity extraction prompt word and the first evaluation data set, the second entity extraction prompt word and the second evaluation data set, the first entity extraction result and the second entity extraction result are obtained respectively through the target LLM, the first entity extraction result, the second entity extraction result and the first evaluation data set are constructed as the target data set, and based on the target data set and the third prompt word, the target prompt word is determined from the first entity extraction result and the second entity extraction result through the target LLM.

[0009] In one embodiment of the present disclosure, a text polishing dataset is created, and a first prompt is continuously updated using original text and polished text in the text polishing dataset to determine a first text polishing prompt, including:

[0010] Obtain at least two preset semantically clear texts to be recognized as entities;

[0011] Performing semantic interference processing on the preset semantically clear text to obtain the semantically ambiguous text, and using the preset semantically clear text and the semantically ambiguous text as the original text;

[0012] Based on the first prompt and the original text, a polished text is generated through the target LLM;

[0013] Construct a text polishing dataset based on the original text and the polished text;

[0014] Calculate a first comprehensive evaluation index between the original text and the polished text;

[0015] Based on the text polishing dataset, continuously updating the first prompt according to the first comprehensive evaluation indicator;

[0016] The first prompt corresponding to the highest value of the first comprehensive evaluation index is used as the first text polishing prompt.

[0017] In one embodiment of the present disclosure, calculating a first comprehensive evaluation index between the original text and the polished text includes:

[0018] Calculate the fluency index of the polished text;

[0019] Calculate the semantic similarity index between the polished text and the original text;

[0020] A first comprehensive evaluation index is determined based on the fluency index and the semantic similarity index.

[0021] In one embodiment of the present disclosure, determining a second comprehensive evaluation index of a target LLM according to the first entity content, the second entity content and the entity tag includes:

[0022] Determine a string similarity index and a set similarity index between the first entity content or the second entity content and the entity tag;

[0023] A second comprehensive evaluation index is determined according to the string similarity index and the set similarity index.

[0024] In one embodiment of the present disclosure, based on the target data set and the third prompt, determining the target prompt from the first entity extraction result and the second entity extraction result by using the target LLM includes:

[0025] Based on the first entity extraction result and the entity label, calculating a first entity evaluation index of the first entity extraction result according to a second comprehensive evaluation index;

[0026] Based on the second entity extraction result and the entity label, calculating a second entity evaluation index of the second entity extraction result according to a second comprehensive evaluation index;

[0027] The larger of the first entity evaluation index and the second entity evaluation index is used as the target entity label, and the target entity label is added to the target data set;

[0028] A target prompt is determined according to the first entity extraction result and the second entity extraction result.

[0029] In one embodiment of the present disclosure, determining a target prompt according to the first entity extraction result and the second entity extraction result includes:

[0030] Determine a first prediction probability corresponding to the first entity extraction result and a second prediction probability corresponding to the second entity extraction result;

[0031] The larger of the first predicted probability and the second predicted probability is used as the target predicted probability;

[0032] If the target prediction probability is equal to the target entity label, the third comprehensive evaluation index of the first entity extraction result or the second entity extraction result is set to 1, otherwise, it is set to 0;

[0033] The average value of the third comprehensive evaluation index is used as the target evaluation score of the target data set;

[0034] Based on the target evaluation score and the third prompt, the target LLM is used to generate an intermediate entity extraction result, and the target evaluation score of the intermediate entity extraction result is calculated, and the third prompt is continuously updated according to the target evaluation score;

[0035] The third prompt corresponding to the highest target evaluation score is used as the target prompt.

[0036] The second aspect of the present disclosure provides an entity content extraction device based on LLM, the device comprising:

[0037] A text polishing module, used to create a text polishing data set, and continuously update the first prompt using the original text and the polished text in the text polishing data set, and determine the first text polishing prompt, wherein the polished text is generated based on the original text and the first prompt through the target LLM, and the first text polishing prompt is the one with the highest first comprehensive evaluation index among the first prompts;

[0038] An entity extraction module is used to determine a first evaluation data set corresponding to the original text and a second evaluation data set corresponding to the polished text based on the text polishing data set and the first text polishing prompt, and generate a first entity content corresponding to the first evaluation data set and a second entity content corresponding to the second evaluation data set through a target LLM and a second prompt;

[0039] An evaluation module is used to determine a second comprehensive evaluation index of the target LLM according to the first entity content, the second entity content and the entity label, and iterate the first evaluation data set and the second evaluation data set respectively based on the second comprehensive evaluation index to determine a first entity extraction prompt word corresponding to the first evaluation data set and a second entity extraction prompt word corresponding to the second evaluation data set, where the entity label is a preset label corresponding to the first entity content and the second entity content;

[0040] A prompt determination module obtains a first entity extraction result and a second entity extraction result respectively through a target LLM based on a first entity extraction prompt word and a first evaluation data set, a second entity extraction prompt word and a second evaluation data set, constructs the first entity extraction result, the second entity extraction result and the first evaluation data set into a target data set, and determines a target prompt from the first entity extraction result and the second entity extraction result through a target LLM based on the target data set and a third prompt.

[0041] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods in the first aspect embodiment of the present disclosure.

[0042] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the method in the first aspect embodiment of the present disclosure.

[0043] The fifth aspect embodiment of the present disclosure proposes a computer program product, characterized in that it includes a computer program, and when the computer program is executed by a processor, it implements any method in the first aspect embodiment of the present disclosure.

[0044] In summary, according to the entity content extraction method based on LLM proposed in the present disclosure, a text polishing dataset is created, and the original text and the polished text in the text polishing dataset are used to continuously update the first prompt to determine the first text polishing prompt, wherein the polished text is generated through the target LLM based on the original text and the first prompt, and the first text polishing prompt is the one with the highest first comprehensive evaluation index among the first prompts, so that LLM can improve the expression quality and readability of the original text, thereby replacing the traditional manual or semi-automatic text preprocessing technology; based on the text polishing dataset and the first text polishing prompt, a first evaluation dataset corresponding to the original text and a second evaluation dataset corresponding to the polished text are determined respectively, and the first entity content corresponding to the first evaluation dataset and the second entity content corresponding to the second evaluation dataset are generated respectively through the target LLM and the second prompt, and the entity content of the original text and the polished text are obtained; according to the first The second comprehensive evaluation index of the target LLM is determined based on the entity content, the second entity content and the entity label. The first evaluation data set and the second evaluation data set are iterated based on the second comprehensive evaluation index respectively to determine the first entity extraction prompt word corresponding to the first evaluation data set and the second entity extraction prompt word corresponding to the second evaluation data set. The entity label is the preset label corresponding to the first entity content and the second entity content. The performance of the target LLM in the entity recognition task is comprehensively evaluated; according to the first entity extraction prompt word and the first evaluation data set, the second entity extraction prompt word and the second evaluation data set, the first entity extraction result and the second entity extraction result are obtained respectively through the target LLM, and the first entity extraction result, the second entity extraction result and the first evaluation data set are constructed as the target data set. Based on the target data set and the third prompt, the target prompt word is determined from the first entity extraction result and the second entity extraction result through the target LLM. A better prompt word is obtained, which provides a reliable and secure prompt word for the large language model to process the entity extraction task, and improves the reliability and security of information extraction.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0047] Figure 1 It is a flow chart of an entity content extraction method based on LLM according to an embodiment of the present disclosure;

[0048] Figure 2 A flowchart of determining a first text polishing prompt according to an embodiment of the present disclosure;

[0049] Figure 3 A flowchart of text polishing according to an embodiment of the present disclosure is provided;

[0050] Figure 4 A flowchart of calculating a first comprehensive evaluation index between an original text and a polished text according to an embodiment of the present disclosure;

[0051] Figure 5 It is a flowchart of determining a second comprehensive evaluation index of a target LLM according to a first entity content, a second entity content and an entity tag according to an embodiment of the present disclosure;

[0052] Figure 6 A flowchart of an embodiment of the present disclosure for determining a target prompt from a first entity extraction result and a second entity extraction result through a target LLM based on a target data set and a third prompt;

[0053] Figure 7 A schematic diagram of a process of entity extraction according to an embodiment of the present disclosure;

[0054] Figure 8 A flowchart of determining a target prompt according to a first entity extraction result and a second entity extraction result according to an embodiment of the present disclosure;

[0055] Fig. 9 A schematic diagram of a process of determining a target prompt in an embodiment of the present disclosure;

[0056] Fig.10 It is a structural schematic diagram of an entity content extraction device based on LLM according to an embodiment of the present disclosure;

[0057] Fig.11 It is a schematic diagram of the module composition of the entity content extraction device based on LLM;

[0058] Fig.12 is a block diagram of an electronic device for implementing the entity content extraction method based on LLM disclosed in the present invention according to an exemplary embodiment;

[0059] Fig.13 It is a schematic diagram of the structure of the chip of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout identify the same or similar originals or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0061] First, a brief introduction to the relevant terms in this disclosure is given:

[0062] Large Language Model (LLM): In this disclosure, it refers to a deep learning model that can understand and generate natural language text in the field of natural language processing (NLP) through deep learning technology, especially a neural network-based architecture. It is often called a large model. It usually includes hundreds of millions to hundreds of billions of parameters. After training with a large amount of text data, it shows strong language understanding and generation capabilities. It is widely used in tasks such as dialogue systems, content creation, and machine translation. For example, commonly used large models include ChatGLM, QWEN, DeepSeek, GPT, etc.

[0063] Named Entity Recognition (NER): In this disclosure, it refers to a key task in natural language processing, which involves identifying entities with specific meanings in text, such as names of people, places, institutions, proper nouns, etc. These entities play an important role in text and are the basis for fields such as information extraction, question-answering systems, syntactic analysis, and machine translation. The goal of entity recognition is to determine the location of these entities in the text and assign them the correct semantic category so that computers can understand and process these entities.

[0064] Prompt: In this disclosure, it refers to a method of input that guides the model to perform specific outputs or tasks through specific instructions or questions. In the field of artificial intelligence, especially in natural language processing, Prompt is used to stimulate specific functions of the model, such as answering questions, generating text, translating languages, etc. It can be a question, a description, a task description, or even part of the conversation history, etc., designed to guide the model to generate a specific type of response. Prompt not only helps the model better understand the intention of the input, but also improves the interpretability and accessibility of the model. In prompt word engineering, the definition of Prompt covers three main elements: tasks, instructions, and roles to ensure that the text generated by the model meets user needs. Through clear and specific guidance language models, Prompt can guide the model to obtain the desired output and thus better complete the task.

[0065] In the field of named entity recognition, a variety of technical solutions have been proposed to improve the accuracy and robustness of recognition, including non-generative deep learning methods, such as BERT in a feature-based manner; UIE combined with general information extraction of prompts, and LLM_UIE, an extension of UIE on the language generation model, with prompts as the main paradigm.

[0066] Neural Network (NN) combined with Conditional Random Function (CRF) solution: This solution combines a non-generative neural network model with CRF for named entity recognition. The specific process is to first use the prepared data set to train the neural network model to obtain a better feature model to extract features from the input text and generate a high-dimensional feature representation. Then, on this basis, the CRF model is used for sequence labeling to improve the accuracy of named entity recognition.

[0067] Universal Information Extraction (UIE) solution: This technology is based on the ERNIE 3.0 knowledge-enhanced pre-trained model and is more suitable for handling the rapid cold start problem of zero-sample or small-sample data. UIE enhances the model's adaptability and generalization to new entity types through deep learning and big data advantages.

[0068] Large Language Model-based Communication Information Extraction (LLM_UIE) solution: fine-tune the instructions of a large-scale language model through high-quality information extraction data, and unify the training of information extraction tasks. This solution can cover tasks such as named entity recognition, relation extraction (RE), and event extraction (EE), aiming to achieve general structured extraction, thereby improving the overall effect and efficiency of information extraction tasks. Through instruction fine-tuning, LLM_UIE can perform well in different types of information extraction tasks and has strong adaptability.

[0069] The named entity recognition method based on the combination of non-generative neural network model and CRF has certain limitations. First, this method cannot cope with a variety of complex text situations, especially when faced with diverse and complex semantic scenarios, and its performance is relatively weak. In addition, the NN combined with CRF model lacks the ability to fine-tune with a few samples, and cannot be effectively adjusted and optimized when data is limited, thus affecting its flexibility and adaptability in practical applications.

[0070] Although UIE performs well under few-sample conditions and can achieve high extraction effects with a small amount of labeled data, the fine-tuning process still requires a certain amount of cost. The ability to understand different text contexts is particularly important for UIE. For example, for situations like "Jinnan District, Tianjin" and "Jinnan District" that should be identified as the same address, UIE may mistakenly identify them as two different addresses, which may lead to accuracy issues in information extraction in practical applications.

[0071] After LLM_UIE is fine-tuned on high-quality information extraction data, it can perform well in specific scenarios such as named entity recognition, but its general ability will be reduced. In particular, the original instruction-following ability of the fine-tuned LLM will be reduced, which makes its scope of application narrower. In addition, since LLM is prone to hallucinations, that is, generating content unrelated to the text, for example, when only the text "Yao Ming" is input, information such as the address "Shanghai" that exists in the training data may be generated. This situation will not only affect the accuracy of information extraction, but may also lead to security issues such as information leakage.

[0072] The present disclosure aims to optimize the existing entity recognition technology to avoid resource consumption caused by fine-tuning the model under this task, and to ensure the instruction compliance ability of the large language model and the reliability and security of the output results. The present disclosure plans to use a common LLM or all the same large language models at different stages to ensure that the large language model has compliance when executing specific instructions, and use the semantic understanding ability of the LLM to use different prompt texts for recognition and verification, so that the model can effectively summarize and extract key information in the text, while avoiding information leakage and generating irrelevant content, ensuring the high reliability and security of the output results.

[0073] The entity content extraction method based on LLM disclosed in the present invention has a wide range of application fields and scenarios. The following are some possible applications:

[0074] 1. Content creation and editing

[0075] 1. Automated Writing Assistant

[0076] Application scenarios: journalists writing press releases, writers creating novels, marketers writing advertising copy, etc.

[0077] Function description: Use LLM to generate a first draft, and use the polishing module to optimize semantic fluency and correctness, and finally output high-quality text content.

[0078] 2. Academic writing assistance

[0079] Application scenarios: students writing papers, scholars writing scientific research reports, etc.

[0080] Function description: Extract key information of relevant documents through the recognition and extraction module, and generate text that meets academic standards in combination with the polishing module.

[0081] 2. Data Analysis and Information Extraction

[0082] 1. Business intelligence analysis

[0083] Application scenarios: enterprise market research, competitor analysis, etc.

[0084] Function description: Extract key information from large amounts of unstructured data and determine the best business decision through the decision-making module.

[0085] 2. Financial Data Analysis

[0086] Application scenarios: stock analysts predict market trends, risk management, etc.

[0087] Functional description: Extract key data from financial news and reports, and assist decision making through market analysis reports generated by LLM.

[0088] 3. Customer Service and Support

[0089] 1. Intelligent customer service system

[0090] Application scenarios: customer services of e-commerce platforms, banks, telecom operators, etc.

[0091] Function description: Understand user questions through recognition and extraction modules, automatically generate accurate answers, and improve customer service efficiency and satisfaction.

[0092] 2. Technical support and troubleshooting

[0093] Application scenarios: IT company technical support, smart home equipment after-sales service, etc.

[0094] Function description: Extract key points of problems from user feedback and generate detailed troubleshooting guides through LLM.

[0095] IV. Education and Training

[0096] 1. Personalized learning guidance

[0097] Application scenarios: online education platform, after-school tutoring in schools, etc.

[0098] Function description: Generate personalized learning materials and exercises based on students' learning situation.

[0099] 2. Language learning assistance

[0100] Application scenarios: language training institutions, self-taught foreign language learners, etc.

[0101] Functional description: The polishing module corrects grammatical errors, and the recognition and extraction module helps learners extract key expressions in articles or conversations.

[0102] V. Law and Compliance

[0103] 1.Contract review and drafting

[0104] Application scenarios: law firms, corporate legal departments, etc.

[0105] Functional description: Quickly generate a contract draft through LLM, and ensure the accuracy and legality of the contract terms through the polishing module.

[0106] 2. Regulatory compliance and risk assessment

[0107] Application scenarios: compliance inspection of financial institutions, enterprise risk control, etc.

[0108] Functional description: Extract key requirements from regulatory texts and evaluate the compliance of enterprise operations through decision-making modules.

[0109] 6. Media and Entertainment

[0110] 1.Script creation and revision

[0111] Application scenarios: film studios, TV drama production companies, etc.

[0112] Functional description: Use LLM to generate a first draft of the script, and use the polishing module to optimize the dialogue and plot development.

[0113] 2. Game content development

[0114] Application scenarios: video game design, interactive plot development, etc.

[0115] Function description: The game background story is enriched through the recognition and extraction module, and the polishing module enhances the naturalness of the character dialogue.

[0116] 7. Healthcare

[0117] 1. Electronic medical record recording and organization

[0118] Application scenarios: hospitals, clinics and other medical institutions.

[0119] Function description: Automatically generate electronic medical records and ensure the accuracy of medical terminology through the polishing module.

[0120] 2. Analysis of medical research literature

[0121] Application scenarios: medical researchers, clinicians, etc.

[0122] Function description: Extract key research results from a large amount of medical literature to assist medical research and clinical decision-making.

[0123] 8. Government and Public Services

[0124] 1. Policy formulation and analysis

[0125] Application scenarios: government departments, think tanks, etc.

[0126] Functional description: Analyze public opinions and expert suggestions through the identification and extraction module to assist in policy making.

[0127] 2. Optimization of public services

[0128] Application scenarios: urban management, traffic planning, etc.

[0129] Functional description: Extract key issues from citizen feedback and generate solutions and improvement measures through LLM.

[0130] In summary, this technical solution has broad application prospects in many fields such as content creation, data analysis, customer service, education and training, legal compliance, media entertainment, medical health, and government public services. With the continuous advancement and optimization of technology, its application scope and depth will be further expanded. The application scenarios are not limited in the embodiments of this disclosure.

[0131] The LLM-based entity content extraction method provided by the present disclosure is described in detail below with reference to the accompanying drawings.

[0132] Figure 1 FIG. 1 is a flow chart of an entity content extraction method based on LLM according to an embodiment of the present disclosure. Figure 1 In the embodiment shown, the entity content extraction method based on LLM includes:

[0133] Step 101, creating a text polishing dataset, and continuously updating a first prompt using the original text and the polished text in the text polishing dataset, and determining a first text polishing prompt, wherein the polished text is generated based on the original text and the first prompt through a target LLM, and the first text polishing prompt is the one with the highest first comprehensive evaluation index among the first prompts.

[0134] In this embodiment, the original text refers to the text input by the user to the large language model, including semantically clear text and semantically unclear text. The polished text refers to the text obtained by improving the text expression quality and readability of the original text. The text polishing data set refers to a data set constructed by the original text and the polished text, which is used to improve the expression quality of the output text. The first prompt refers to the instruction statement for performing text enhancement processing on the original text in the input text provided to the large language model. The first text polishing prompt refers to the prompt corresponding to the maximum value of the first comprehensive evaluation index in the first prompt. The target LLM refers to the large language model used in the present disclosure. In each step of the present disclosure, the large language model can use the same one, or different large language models can be used in different processing stages. The first comprehensive evaluation index refers to the total score of the comprehensive confusion and semantic similarity between the original text and the corresponding polished text, which is used to evaluate the improvement of the polished text relative to the original text in terms of semantic clarity, expression quality, and readability. The larger the first comprehensive evaluation index, the better the improvement effect of the original text to the polished text.

[0135] After creating a text polishing dataset based on the original text and the polished text, since there is a one-to-one correspondence between the original text and the polished text, a polished text is obtained after polishing an original text. The original text and the first prompt are used as the input of the target LLM to generate the corresponding polished text. Therefore, the first prompt can be scored based on the original text and the polished text in the text polishing dataset, thereby reflecting the accuracy of the rule setting in the instruction of the first prompt. The score for the first prompt is used as the first comprehensive evaluation index. For each first comprehensive evaluation index obtained through iteration in the text polishing dataset, the first prompt corresponding to the maximum value is taken as the first text polishing prompt.

[0136] Step 102, based on the text polishing dataset and the first text polishing prompt, respectively determine a first evaluation dataset corresponding to the original text and a second evaluation dataset corresponding to the polished text, and respectively generate a first entity content corresponding to the first evaluation dataset and a second entity content corresponding to the second evaluation dataset through the target LLM and the second prompt.

[0137] In this embodiment, the first evaluation data set refers to the polished text set generated by taking the original text and the first text polishing prompt in the text polishing data set as the input of the target LLM, and the text in the text set is annotated automatically or manually. The second evaluation data set refers to the polished text set generated by taking the polished text and the first text polishing prompt in the text polishing data set as the input of the target LLM, and the polished text set is annotated similar to the first evaluation data set. The first entity content refers to an entity with a specific meaning in the text identified from the first evaluation data set. The second entity content refers to an entity with a specific meaning in the text identified from the second evaluation data set. Optionally, the first entity content and the second entity content include names of people, places, names of organizations, proper nouns, etc. The second prompt refers to a prompt that controls the large model to extract entity content from the input text.

[0138] The original text in the text polishing dataset and the first text polishing prompt are used as the input of the target LLM, and the output text obtained constitutes the first evaluation dataset. The polished text in the text polishing dataset and the first text polishing prompt are used as the input of the target LLM, and the output text obtained constitutes the second evaluation dataset. Based on the text of the first evaluation dataset and the second prompt as the input of the target LLM, the first entity content can be obtained by performing entity recognition through the target LLM. Similarly, based on the text of the second evaluation dataset and the second prompt as the input of the target LLM, the second entity content can be obtained by performing entity recognition through the target LLM.

[0139] Step 103, determine the second comprehensive evaluation index of the target LLM according to the first entity content, the second entity content and the entity label, iterate the first evaluation data set and the second evaluation data set respectively based on the second comprehensive evaluation index, determine the first entity extraction prompt corresponding to the first evaluation data set and the second entity extraction prompt corresponding to the second evaluation data set, and the entity label is a preset label corresponding to the first entity content and the second entity content.

[0140] In this embodiment, the entity tag refers to a preset tag corresponding to the first entity content and the second entity content, and the first entity content and the second entity content jointly correspond to the entity tag. The second comprehensive evaluation index refers to an index obtained based on the set similarity index and the string similarity index for reflecting the similarity between two text contents. The first entity extraction prompt refers to a prompt for entity extraction obtained by continuous iteration based on the second comprehensive evaluation index and the first evaluation data set. The second entity extraction prompt refers to a prompt for entity extraction obtained by continuous iteration based on the second comprehensive evaluation index and the second evaluation data set.

[0141] According to the first entity content, the second entity content and the entity labels corresponding to the two, the performance of the target LLM can be evaluated by constructing a second comprehensive evaluation index. The set similarity and text similarity of the first entity content and the second evaluation data set are calculated, and the second comprehensive evaluation index is obtained by integrating the set similarity and text similarity. Based on the second comprehensive evaluation index, the first evaluation data set and the second evaluation data set are iterated respectively to obtain the first entity extraction prompt and the second entity extraction prompt.

[0142] Step 104, according to the first entity extraction prompt and the first evaluation data set, the second entity extraction prompt and the second evaluation data set, respectively obtain the first entity extraction result and the second entity extraction result through the target LLM, construct the first entity extraction result, the second entity extraction result and the first evaluation data set into a target data set, and based on the target data set and the third prompt, determine the target prompt from the first entity extraction result and the second entity extraction result through the target LLM.

[0143] In this embodiment, the target data set refers to a data set constructed by the first entity extraction result, the second entity extraction result and the first evaluation data set, which is used to determine the final prompt for entity recognition. The third prompt refers to the instruction sentence for entity recognition in the input text provided to the large language model. The target prompt refers to the prompt that the large language model finally provides to the user with clear semantics, high readability and reliable expression quality text output.

[0144] According to the first entity prompt and the first evaluation data set, a first entity extraction result is obtained through a target LLM, and according to the second entity prompt and the second evaluation data set, a second entity extraction result is obtained through a target LLM. The first entity extraction result, the second entity extraction result and the first evaluation data set are constructed into a target data set. According to the target data set and the third prompt, one of the first entity extraction result and the second entity extraction result is selected as a target prompt through the target LLM.

[0145] In summary, according to the entity content extraction method based on LLM proposed in the present disclosure, a text polishing dataset is created, and the original text and the polished text in the text polishing dataset are used to continuously update the first prompt to determine the first text polishing prompt, wherein the polished text is generated through the target LLM based on the original text and the first prompt, and the first text polishing prompt is the one with the highest first comprehensive evaluation index among the first prompts, so that LLM can improve the expression quality and readability of the original text, thereby replacing the traditional manual or semi-automatic text preprocessing technology; based on the text polishing dataset and the first text polishing prompt, a first evaluation dataset corresponding to the original text and a second evaluation dataset corresponding to the polished text are determined respectively, and the first entity content corresponding to the first evaluation dataset and the second entity content corresponding to the second evaluation dataset are generated respectively through the target LLM and the second prompt, and the entity content of the original text and the polished text are obtained; according to the first The second comprehensive evaluation index of the target LLM is determined based on the entity content, the second entity content and the entity label. The first evaluation data set and the second evaluation data set are iterated based on the second comprehensive evaluation index to determine the first entity extraction prompt corresponding to the first evaluation data set and the second entity extraction prompt corresponding to the second evaluation data set. The entity label is the preset label corresponding to the first entity content and the second entity content. The performance of the target LLM in the entity recognition task is comprehensively evaluated; according to the first entity extraction prompt and the first evaluation data set, the second entity extraction prompt and the second evaluation data set, the first entity extraction result and the second entity extraction result are obtained respectively through the target LLM, and the first entity extraction result, the second entity extraction result and the first evaluation data set are constructed as the target data set. Based on the target data set and the third prompt, the target prompt is determined from the first entity extraction result and the second entity extraction result through the target LLM. A better prompt is obtained, which provides a reliable and secure prompt for the large language model to process the entity extraction task, and improves the reliability and security of information extraction.

[0146] Figure 2 The present invention is a flowchart of determining a first text polishing prompt according to an embodiment of the present invention. Figure 2 Yes Figure 1 Further explanation of step 101 is based on Figure 2 The embodiment shown comprises the following steps:

[0147] Step 201, obtaining at least two preset semantically clear texts to be recognized as entities.

[0148] In this embodiment, the preset semantically clear text refers to a text with clear meaning, high readability, and reliable expression quality. At least two preset semantically clear texts for entity recognition are obtained.

[0149] Step 202: semantic interference processing is performed on the preset semantically clear text to obtain semantically ambiguous text, and the preset semantically clear text and the semantically ambiguous text are used as original text.

[0150] In this embodiment, semantically ambiguous text refers to a text with unclear meaning, poor readability, and ambiguous expression corresponding to a preset semantically clear text. The preset semantically ambiguous text is subjected to semantic interference processing to obtain the semantically ambiguous text. Optionally, the semantic interference processing can be achieved by adding noise, repeating the text, adding useless words, and disordering the text. The obtained semantically ambiguous text and semantically clear text are used as the original text.

[0151] In one implementation of this embodiment,

[0152] Through the assistance of the preset interference pattern, several complete and fluent semantic texts to be recognized are manually screened, and rules are used for each data to generate semantically incoherent texts containing noise, repetition, useless words, and disorder, etc., which are jointly constructed with the original text to form a data and annotate a data.

[0153] The specific data generation algorithm is as follows:

[0154] 1. Initialize the data set: D = {}

[0155] Where D is the currently generated semantically unclear data set, including the original text of the preset semantically clear text and the semantically unclear text, which is initially empty.

[0156] 2. Introduce interference, calculate the number of characters n introduced to interfere and the selected position set P:

[0157] n i =max(1,round(ε*len(t i ))),P={p1,p2,...,pn}

[0158] where t i ∈T, T represents the original text set, P is a randomly selected position, len(t i ) is the length of the original text, and ε is the error rate hyperparameter ε∈[0,1].

[0159] Modify iFor the characters at P in the text, apply the preset interference patterns (such as noise, repetition, useless words, and disorder):

[0160] t′ i =modify(t i , P)

[0161] where t i ′ is a text with ambiguous semantics.

[0162] Save the semantically ambiguous text and the original text as the original text:

[0163] D=D∪{(t′ i , t i )}

[0164] Step 203: Generate a polished text through the target LLM based on the first prompt and the original text.

[0165] In this embodiment, the first prompt and the original text are used as inputs of the target LLM, and the polished text is generated by the target LLM.

[0166] Step 204 , using the original text as sample data and the polished text as sample labels, to construct a text polishing dataset.

[0167] In this embodiment, the original text and the polished text are constructed into a text polishing dataset.

[0168] Step 205: Calculate a first comprehensive evaluation index between the original text and the polished text.

[0169] In this embodiment, a first comprehensive evaluation index between the original text and the polished text is calculated to determine the improvement strength and effect of the first prompt word in processing the original text into the polished text.

[0170] Step 206: Based on the text polishing data set, continuously update the first prompt according to the first comprehensive evaluation index.

[0171] In this embodiment, each sample in the text polishing data set is traversed, and a first comprehensive evaluation index of the sample is calculated to continuously update the first prompt.

[0172] Step 207: Use the first prompt corresponding to the highest value of the first comprehensive evaluation index as the first text polishing prompt.

[0173] In this embodiment, the first prompt corresponding to the maximum value of all first comprehensive evaluation indicators obtained by traversing the text polishing data set is used as the first text polishing prompt, and the polishing prompt required from the original text to the polished text is obtained.

[0174] Figure 3FIG. 1 is a flowchart of a text polishing process according to an embodiment of the present disclosure. Figure 3 As shown, the polished text template (first prompt) and the polished text dataset are used as inputs of the target LLM, the polished text is generated through the target LLM, the first comprehensive evaluation index of the current polished text and the last polished text is calculated, and the current evaluation value is evaluated according to the first comprehensive evaluation index. The iterative first prompt and the original text of the polished text dataset are continuously used as inputs of the target LLM until the preset upper limit of the number of iterations is reached, and the polished prompt text of the first stage is obtained. The polished prompt text of the first stage and the original text of the information to be extracted are used as inputs of the large language model (target LLM) to generate the polished text, thereby achieving the polishing of the original text and improving the semantic quality, readability and reliability of the original text.

[0175] Figure 4 The present invention is a flowchart of calculating a first comprehensive evaluation index between an original text and a polished text according to an embodiment of the present invention. Figure 4 Yes Figure 3 Further description of step 205 is made based on Figure 4 The embodiment shown comprises the following steps:

[0176] Step 401, calculating the fluency index of the polished text.

[0177] In this embodiment, the fluency index refers to an index for evaluating whether the text generated by the target LLM is fluent, easy to understand and read. Optionally, it is expressed by the perplexity of the text generation quality index. The lower the perplexity, the higher the certainty and better performance of the text. The perplexity index is:

[0178]

[0179] Where R is the polished text sequence generated by LLM, containing N words. is the conditional probability of the i-th word appearing given the previous i-1 words, provided by LLM.

[0180] In one implementation of this embodiment, the perplexity index may be calculated by using statistics generated by an open source n-gram model or ChatGLM.

[0181] Step 402, calculating the semantic similarity index between the polished text and the original text.

[0182] In this embodiment, the semantic similarity index refers to the similarity of the meaning between two texts. The semantic similarity between the original text and the polished text is calculated using an NLP model that can encode text. Optionally, the NLP model can use Bidirectional Encoder Representations from Transformers (BERT). The specific method is as follows:

[0183] E_R=Embedding(R)

[0184] E_T=Embedding(T)

[0185] S(R,T)=similarity(E_R,E_T)

[0186] Among them, E_R and E_T respectively represent the encoding results of the polished text R and the original text T after the BERT model, indicating their similarity scores.

[0187] In one implementation of this embodiment, the semantic similarity index can be calculated using an open source Robustly Optimized BERT Pretraining Approach (RoBERTa).

[0188] Step 403: Determine a first comprehensive evaluation index according to the fluency index and the semantic similarity index.

[0189] In this embodiment, the comprehensive score of the polished text and the original text is comprehensively judged based on the fluency index and the semantic similarity index, wherein the perplexity is converted into a standardized percentage. The larger the value, the better the instruction manual effect. The comprehensive score is:

[0190]

[0191] Among them, a and b are weighting parameters, and the weights can be adjusted according to the evaluation requirements to reflect the importance of error correction and semantic preservation.

[0192] In one implementation of this embodiment, the score indicator of a piece of data can use the above-mentioned Score, and the total score can be obtained by averaging the scores of all instances, that is, Where N is the total number of instances in the evaluation dataset.

[0193] In one embodiment of the present disclosure, the first prompt and the first comprehensive evaluation index Score are initialized; for each iteration, the prompt is automatically adjusted based on the previous output and the current score. A fixed template is formed for the currently generated prompt to be evaluated to obtain a new score, and iterate again. Among them, the iteration method is to let LLM iteratively update the prompt template according to the current prompt and the corresponding score. The constructed prompt contains multiple historical template data and corresponding scores, and adds the prompt of the task that needs to be iteratively updated. The construction method is as follows:

[0194]

[0195] The iteration upper limit hyperparameter is set. When the upper limit is reached, the template with the highest comprehensive score (the first comprehensive evaluation index) among the historical prompt templates is selected as the polishing text prompt.

[0196] In this embodiment, the first comprehensive evaluation index is determined by the perplexity and similarity between the original text and the polished text, which provides a data index for improving semantic reliability and polishing effect.

[0197] Figure 5 The present invention is a flowchart of an embodiment of the present invention for determining a second comprehensive evaluation index of a target LLM according to first entity content, second entity content and entity tag. Figure 5 Yes Figure 1 The specific description of step 103 is based on Figure 5 The embodiment shown comprises the following steps:

[0198] Step 501: Determine a string similarity index and a set similarity index between the first entity content or the second entity content and the entity tag.

[0199] In this embodiment, the string similarity index is used to quantify the difference between two strings by calculating the minimum number of operations (such as inserting, deleting, and replacing characters) required to transform one string into another string, such as the edit distance (Levenshtein distance) algorithm, in which the Levenshtein distance Lev between the result R generated by LLM and the annotated recognition extraction result L is R,L , the formula is as follows:

[0200]

[0201] Where i and j represent the positions of the string in R and L respectively, and cost is a conditional cost defined as:

[0202]

[0203] Among them, the maximum distance value is expressed as follows:

[0204] lev R，L (max)=|R|+|L|

[0205] The final string similarity metric is given by Lev R,L express.

[0206] The set similarity index is mainly used to evaluate and compare the similarity between two sets. Compared with the string similarity index, it focuses on the existence of elements rather than the position or order of elements in the structure. For example, Jaccard similarity, where Jaccard similarity is calculated as follows:

[0207]

[0208] Among them, |R∩L| represents the number of characters in the intersection of R output by LLM and L marked, that is, the number of correct characters. |R∩L| represents the number of characters in the union of R and L, that is, the total number of all characters.

[0209] Step 502: Determine a second comprehensive evaluation index according to the string similarity index and the set similarity index.

[0210] In this embodiment, according to the string similarity index Lev R,L And the set similarity index J(R, L), construct the second comprehensive evaluation index as follows:

[0211]

[0212] Among them, a and b are weight factors, which are adjusted according to their importance in practical applications. The larger the score, the better the effect.

[0213] In one implementation of this embodiment, the score index for a piece of data can be represented by the second comprehensive evaluation index Score, and the total score for each piece of data can be obtained by averaging the scores of all instances, that is:

[0214]

[0215] Where N is the total number of instances in the evaluation dataset.

[0216] In this embodiment, a second comprehensive evaluation index is obtained through string similarity and set similarity indicators. By combining the string similarity and set similarity indicators, the performance of LLM in the entity recognition task can be comprehensively evaluated. The string similarity index provides an intuitive measure of the degree of character-level modification required, while the set similarity index evaluates the completeness and accuracy of recognition from the perspective of the set.

[0217] Figure 6The present invention is a flowchart of an embodiment of the present invention for determining a target prompt from a first entity extraction result and a second entity extraction result through a target LLM based on a target data set and a third prompt. Figure 6 Yes Figure 5 and Figure 1 The specific description of step 104 is based on Figure 6 The embodiment shown comprises the following steps:

[0218] Step 601: Based on the first entity extraction result and the entity label, a first entity evaluation index of the first entity extraction result is calculated according to a second comprehensive evaluation index.

[0219] In this embodiment, the first entity evaluation index refers to the similarity between the first entity extraction result and the text content of the entity tag. The second comprehensive evaluation index can be used to calculate the similarity between the first entity extraction result and the entity tag.

[0220] Step 602: Based on the second entity extraction result and the entity label, a second entity evaluation index of the second entity extraction result is calculated according to a second comprehensive evaluation index.

[0221] In this embodiment, the second entity evaluation index refers to the similarity between the second entity extraction result and the text content of the entity tag. The second comprehensive evaluation index can be used to calculate the similarity between the second entity extraction result and the entity tag.

[0222] Step 603: Use the larger one of the first entity evaluation index and the second entity evaluation index as the target entity label, and add the target entity label to the target data set.

[0223] In this embodiment, the target entity label refers to the real label of the entity content of the text. After obtaining the first entity evaluation index and the second entity evaluation index, the larger one of the two is used as the target entity label.

[0224] Step 604: determine a target prompt according to the first entity extraction result and the second entity extraction result.

[0225] In this embodiment, the target prompt can be obtained by combining the first entity extraction result and the second entity extraction result with the third prompt as the input of the target LLM.

[0226] For example, for each piece of data Di in D and the recognition and extraction result Li annotated in the second stage, the comprehensive evaluation index of the second stage (the comprehensive index of the string similarity index and the set similarity index) is used to calculate the scores of R1i and R2i. The highest value is L of the data set D, and its formula is as follows:

[0227] L(i)=max(R1i,R2i)

[0228] Where i represents the i-th data in the data set D, R1i represents the first entity evaluation index of the first entity extraction result of the i-th data, and R2i represents the second entity evaluation index of the second entity extraction result of the i-th data.

[0229] Finally, through the union of D and L, that is, D=DU{L}, the obtained L is put into the optimal data set in the data set D.

[0230] In one implementation of this embodiment,

[0231] Figure 7 FIG. 1 is a flow chart of entity extraction according to an embodiment of the present disclosure. Figure 7 As shown, for the original text, the recognition and extraction prompt template (first prompt word) and the recognition and extraction data set of the original text (first evaluation data set) are evaluated according to the second comprehensive evaluation index through the target LLM, and the original text of the information to be extracted is obtained by continuous iteration. For the polished text, the recognition and extraction prompt board and the recognition and extraction data set containing the polished text (second evaluation data set) are evaluated according to the second comprehensive evaluation index through the target LLM, and the polished text is obtained by continuous iteration. According to the original text of the information to be extracted and the recognition and extraction prompt text of the second stage (the second prompt word), the entity recognition and extraction result (first entity extraction result) of the original text is generated through the large language model. According to the polished text and the second prompt word, the recognition and extraction result (second entity extraction result) of the polished text is generated through the large language model. Through this embodiment, the entity extraction results of the original text and the polished text with the same entity tag are obtained, which provides a selectable result set for entity content extraction.

[0232] Figure 8 The present invention is a flowchart of determining a target prompt according to a first entity extraction result and a second entity extraction result according to an embodiment of the present invention. Figure 8 Yes Figure 6 The specific description of step 604 is based on Figure 8 The embodiment shown comprises the following steps:

[0233] Step 801 : Determine a first prediction probability corresponding to a first entity extraction result and a second prediction probability corresponding to a second entity extraction result.

[0234] In this embodiment, the first prediction probability and the second prediction probability respectively refer to the probability that the first entity extraction result or the second entity extraction result is the correct answer when the large model predicts the i-th data.

[0235] Step 802: Taking the larger of the first prediction probability and the second prediction probability as the target prediction probability.

[0236] In this embodiment, the target prediction probability refers to the prediction probability that can be used as the corresponding final output result. The larger of the first prediction probability and the second prediction probability is used as the target prediction probability.

[0237] Step 803: If the target prediction probability is equal to the target entity label, the third comprehensive evaluation index of the first entity extraction result or the second entity extraction result is set to 1, otherwise, it is set to 0.

[0238] In this embodiment, the third comprehensive evaluation index refers to an evaluation index indicating whether the prediction of the entity extraction result is correct. If the target prediction probability is equal to the value of the target entity label, the third comprehensive evaluation index of the first entity extraction result or the second entity extraction result is set to 1, otherwise, it is set to 0.

[0239] Step 804: Use the average value of the third comprehensive evaluation index as the target evaluation score of the target data set.

[0240] In this embodiment, the target evaluation score refers to the ratio of entity extraction results to actual labels. In order to obtain the target evaluation score of all entity extraction results, the average value of the third comprehensive evaluation index of each text in all target data sets is used as the target evaluation score of the target data set.

[0241] Step 805: Based on the target evaluation score and the third prompt, use the target LLM to generate an intermediate entity extraction result, calculate the target evaluation score of the intermediate entity extraction result, and continuously update the third prompt according to the target evaluation score.

[0242] In this embodiment, the intermediate entity extraction result refers to the further optimized extraction result generated by using the target LLM based on the first entity extraction result or the second entity extraction result during the updating and iterating process of the third prompt. According to the target evaluation score and the third prompt, the target LLM is used to continuously update and iterate the generation of the intermediate entity extraction result, and the target evaluation score of the intermediate entity extraction result is calculated, and then the third prompt is continuously updated by the target evaluation score. Thus, the third prompt is continuously optimized.

[0243] Step 806: Use the third prompt corresponding to the highest target evaluation score as the target prompt.

[0244] In this embodiment, the third prompt corresponding to the highest target evaluation score of each extracted result text in the target data set is used as the target prompt, so as to provide the user with a final reliable and safe prompt.

[0245] In one implementation of this embodiment,

[0246] The first step is to construct a selected dataset.

[0247] The first entity extraction result P1 and the second entity extraction result P2 of the recognition and extraction task obtained in the second stage are respectively used for the semantically unclear text dataset (first evaluation dataset) D1 and the semantically unclear text dataset (second evaluation dataset) D2 to obtain the corresponding first entity extraction result R1 and second entity extraction result R2, and construct the dataset D = {D1, R1, R2}.

[0248] For each piece of data Di in D and the recognition and extraction result Li annotated in the second stage, the comprehensive evaluation index of the second stage (the comprehensive index of the string similarity index and the set similarity index) is used to calculate the scores of R1i and R2i. The highest value is L of the data set D, and its formula is as follows:

[0249] L(i)=max(R1 i , R2 i )

[0250] The second step is to evaluate the LLM.

[0251] The dataset D constructed in the first step uses the initial fixed prompt template to evaluate LLM. The initial fixed prompt template reference is as follows:

[0252]

[0253] Among them, the part in quotation marks is the initial fixed prompt template to be iterated, and the entity recognition is modified according to the specific task, such as location recognition.

[0254] The answer to the large model using the fixed prompt template is calculated using the logits probability indicator to output the first corresponding logits probability score ["A", "B"].

[0255] The specific formula is as follows:

[0256]

[0257] Where P A (i) and P B (i) are the probabilities that option A and option B are the correct answers when the large model predicts the i-th data. A (i) is the first predicted probability, P B (i) is the second prediction probability. The larger of the two is taken as the target prediction probability by the following formula. m = max(P A (i), P B (i)).

[0258] The total score can be obtained by averaging the scores of all instances, that is,

[0259]

[0260] Where N is the total number of instances in the evaluation dataset.

[0261] The third step is to use LLM for iteration and output.

[0262] 1. Initialization:

[0263] Determine the dataset D, the initial prompt and the corresponding Score.

[0264] 2. Iterative optimization:

[0265] Perform the following operations on data set D:

[0266] 1) For each iteration, the prompt is automatically adjusted based on the previous output and the current score.

[0267] 2) A fixed template is formed for the currently generated prompt to be evaluated to obtain a new score and iterate again.

[0268] The iterative method is to let LLM iteratively update the prompt template according to the current prompt and the corresponding score. The constructed prompt contains multiple historical template data and the corresponding scores, and adds the prompt of the task that needs to be iteratively updated. The construction method is as follows:

[0269]

[0270] 3. Output the final result.

[0271] The iteration upper limit hyperparameter is set. When the upper limit is reached, the template with the highest comprehensive score among the historical prompt templates in the data set D1 is used as the corresponding final selected prompt.

[0272] In another implementation of this embodiment,

[0273] Fig. 9 FIG. 1 is a flow chart of determining a target prompt in an embodiment of the present disclosure. Fig. 9As shown, the decision prompt template (third prompt) and the decision data set (target data set) are used as the input of the target LLM, and the third comprehensive scoring index of the first entity extraction result or the second entity extraction result in the third prompt and the target data set is used for evaluation, and the decision prompt text (intermediate entity extraction result) of the third stage is obtained through continuous iteration. The original text (original text) of the information to be extracted, the recognition and extraction result of the original text (first entity extraction result), the recognition and extraction result of the polished text (second entity extraction result) and the intermediate entity extraction result are used as the input of the large language model, and the result obtained by the large language model is used as the target prompt.

[0274] In the embodiment of the present disclosure, a recognition and extraction method based on a large language model is used. Taking location information recognition as an example, the text to be extracted by location information recognition is first used as the original text to be input. The original text is composed of natural language and can be input by the user, written by the customer service, etc. The LLM is selected as ChatGLM.

[0275] Step 1: Text polishing.

[0276] If this is the first time to execute step 1, the text polishing prompt is the initial template and the following steps are required. Otherwise, the first stage polishing prompt template is used directly.

[0277] Before obtaining the first-stage polishing prompt template, a batch of dozens of polished texts that meet the task specifications are first produced. In the current embodiment, taking the polished text containing location information as an example, each piece of data needs to be semantically clear and concise, without obvious sentence errors, etc., such as ["The Five Avenues Scenic Spot in Heping District, Tianjin, is located near the Yingkou Road subway station"], and the polished text is randomly combined through functions such as noise, repetition, useless words, and disorder to obtain a semantically unclear text containing interference, such as ["The Five Avenues Scenic Spot in Heqi District, Tianjin, oh, it is near the Yingkou Road subway station, really, it is over there, next to the subway station, that place, you know, the Five Avenues, that scenic spot, near the subway station."].

[0278] The semantically unclear text is used as the intended user input text, and the original polished text is used as the label data. By replying to the initial text polishing prompt template through ChatGLM, each new polished text corresponding to the intended user input text will be obtained. The prompt is as follows:

[0279]

[0280] The perplexity index is calculated by using the open source n-gram model or ChatGLM generation statistics, and the semantic similarity index is calculated by using the open source RoBERTa as a comprehensive index to determine the average score Score of the original polished text set and the new polished text set. The Score and the polishing prompt template used in the previous step are used as examples. The iteratively optimized prompt pairs are used to require ChatGLM to optimize the current template and obtain a new optimized polishing prompt template. This is iterated until the upper limit of the iteration hyperparameter is reached. The first-stage polishing prompt template with the best score obtained through the data set is selected.

[0281] Iteration prompts such as,

[0282]

[0283] Step 2: Identification and extraction.

[0284] If step 2 is being executed for the first time, the position recognition and extraction prompt is the initial template and the following steps are required. Otherwise, the second-stage position recognition and extraction prompt template of data set 1 is directly used.

[0285] Use the semantically unclear text dataset created in step 1 and the newly polished text dataset generated by aligning the first-stage polishing prompt template, and annotate the recognition and extraction results one by one. Note that the labels of the text annotations in the same batch are the same. Two datasets will be constructed, namely, dataset 1 {semantically unclear text, Label} and dataset 2 {newly polished text, Label}.

[0286] The unclear text and the newly polished text are respectively replied to the data by using the initial position recognition and extraction prompt template through ChatGLM, and the respective recognition and extraction results will be obtained. The prompts are as follows:

[0287]

[0288] For the initial fixed template of dataset 1, the content generated by ChatGLM is:

[0289]

[0290] According to the above two generated contents, it can be seen that the initial fixed templates of dataset 1 and dataset 2 can obtain texts with very high semantic similarity.

[0291] By using Levenshtein distance and Jaccard similarity as evaluation indicators, the average score of dataset 1 and dataset 2 in the location recognition and extraction task is determined. The Score and the location recognition and extraction prompt template used in the previous step are used as examples. The iteratively optimized prompt pairs are used to require ChatGLM to optimize the current template and obtain new location recognition and extraction prompt templates optimized for different datasets. This is iterated until the upper limit of the iterative hyperparameter is reached. The second-stage location recognition and extraction prompt template with the best score obtained from each dataset is selected.

[0292] Iteration prompts such as,

[0293]

[0294] Step 3: Decision making.

[0295] If this is the first time to execute step 3, the decision prompt will be the initial template and the following steps need to be performed. Otherwise, the third stage decision prompt will be used directly.

[0296] In step 2, the position recognition and extraction prompt templates of the second stage are used for the semantically unclear text and the newly polished text to obtain the position recognition and extraction results, and the position extraction results of the semantically unclear text, the semantically unclear text, and the position extraction results of the newly polished text are constructed as a data set. The ChatGLM is used to reply to the data using the initial decision prompt template. The prompts are as follows:

[0297]

[0298] By using the logits probability score to calculate the score of ['A', 'B'], the average score Score is obtained. Take the Score and the decision prompt template used in the previous step as an example, use the iterative optimization prompt pair to require ChatGLM to optimize the current template, and obtain the optimized decision prompt template in the new data set, and iterate until the upper limit of the iteration hyperparameter is reached. Select the third-stage decision prompt template with the best score obtained through each data set. For example, the iterative prompt,

[0299]

[0300] Step 4: Output the results.

[0301] After the prompt template is selected in the third stage, the location recognition result output by ChatGLM will be output as the final result.

[0302] The disclosed embodiment proposes a method for extracting entity content based on LLM, which creates a text polishing dataset, and continuously updates a first prompt using the original text and the polished text in the text polishing dataset to determine the first text polishing prompt, wherein the polished text is generated through a target LLM based on the original text and the first prompt, and the first text polishing prompt is the first prompt with the highest first comprehensive evaluation index, so that LLM can improve the expression quality and readability of the original text, thereby replacing traditional manual or semi-automatic text preprocessing technology; based on the text polishing dataset and the first text polishing prompt, a first evaluation dataset corresponding to the original text and a second evaluation dataset corresponding to the polished text are determined respectively, and a first entity content corresponding to the first evaluation dataset and a second entity content corresponding to the second evaluation dataset are generated respectively through the target LLM and the second prompt, thereby obtaining the entity content of the original text and the polished text; according to the first The second comprehensive evaluation index of the target LLM is determined based on the entity content, the second entity content and the entity label. The first evaluation data set and the second evaluation data set are iterated based on the second comprehensive evaluation index to determine the first entity extraction prompt corresponding to the first evaluation data set and the second entity extraction prompt corresponding to the second evaluation data set. The entity label is the preset label corresponding to the first entity content and the second entity content. The performance of the target LLM in the entity recognition task is comprehensively evaluated; according to the first entity extraction prompt and the first evaluation data set, the second entity extraction prompt and the second evaluation data set, the first entity extraction result and the second entity extraction result are obtained respectively through the target LLM, and the first entity extraction result, the second entity extraction result and the first evaluation data set are constructed as the target data set. Based on the target data set and the third prompt, the target prompt is determined from the first entity extraction result and the second entity extraction result through the target LLM. A better prompt is obtained, which provides a reliable and secure prompt for the large language model to process the entity extraction task, and improves the reliability and security of information extraction.

[0303] Corresponding to the methods provided in the above-mentioned embodiments, the present disclosure also provides an entity content extraction device based on LLM. Since the device provided in the embodiments of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation method of the method is also applicable to the device provided in this embodiment and will not be described in detail in this embodiment.

[0304] Fig.10 FIG. 1 is a schematic diagram of the structure of an entity content extraction device 1000 based on LLM according to an embodiment of the present disclosure. Fig.10 As shown, the LLM-based entity content extraction device includes:

[0305] The text polishing module 1010 is used to create a text polishing data set, and continuously update the first prompt using the original text T and the polished text in the text polishing data set, and determine the first text polishing prompt, wherein the polished text is generated based on the original text and the first prompt through the target LLM, and the first text polishing prompt is the one with the highest first comprehensive evaluation index among the first prompts;

[0306] An entity extraction module 1020 is used to determine a first evaluation data set corresponding to the original text and a second evaluation data set corresponding to the polished text based on the text polishing data set and the first text polishing prompt, and generate first entity content corresponding to the first evaluation data set and second entity content corresponding to the second evaluation data set through the target LLM and the second prompt;

[0307] Evaluation module 1030, used to determine a second comprehensive evaluation index of the target LLM according to the first entity content, the second entity content and the entity label, iterate the first evaluation data set and the second evaluation data set respectively based on the second comprehensive evaluation index, determine a first entity extraction prompt corresponding to the first evaluation data set and a second entity extraction prompt corresponding to the second evaluation data set, and the entity label is a preset label corresponding to the first entity content and the second entity content;

[0308] The prompt determination module 1040 obtains the first entity extraction result and the second entity extraction result respectively through the target LLM according to the first entity extraction prompt and the first evaluation data set, the second entity extraction prompt and the second evaluation data set, constructs the first entity extraction result, the second entity extraction result and the first evaluation data set into a target data set, and determines the target prompt from the first entity extraction result and the second entity extraction result through the target LLM based on the target data set and the third prompt.

[0309] In some embodiments, the text polishing module 1010 is used to:

[0310] Obtain at least two preset semantically clear texts to be recognized as entities;

[0311] Performing semantic interference processing on the preset semantically clear text to obtain the semantically ambiguous text, and using the preset semantically clear text and the semantically ambiguous text as the original text;

[0312] Based on the first prompt and the original text, a polished text is generated through the target LLM;

[0313] Construct a text polishing dataset based on the original text and the polished text;

[0314] Calculate a first comprehensive evaluation index between the original text and the polished text;

[0315] Based on the text polishing dataset, continuously updating the first prompt according to the first comprehensive evaluation indicator;

[0316] The first prompt corresponding to the highest value of the first comprehensive evaluation index is used as the first text polishing prompt.

[0317] In some embodiments, the text polishing module 1010 calculates the first comprehensive evaluation index between the original text and the polished text in the following manner:

[0318] Calculate the fluency index of the polished text;

[0319] Calculate the semantic similarity index between the polished text and the original text;

[0320] A first comprehensive evaluation index is determined based on the fluency index and the semantic similarity index.

[0321] In some embodiments, the evaluation module 1030 determines the second comprehensive evaluation index of the target LLM according to the first entity content, the second entity content and the entity tag in the following manner:

[0322] Determine a string similarity index and a set similarity index between the first entity content or the second entity content and the entity tag;

[0323] A second comprehensive evaluation index is determined according to the string similarity index and the set similarity index.

[0324] In some embodiments, the prompt determination module 1040 determines the target prompt from the first entity extraction result and the second entity extraction result through the target LLM based on the target data set and the third prompt in the following manner:

[0325] Based on the first entity extraction result and the entity label, calculating a first entity evaluation index of the first entity extraction result according to a second comprehensive evaluation index;

[0326] Based on the second entity extraction result and the entity label, calculating a second entity evaluation index of the second entity extraction result according to a second comprehensive evaluation index;

[0327] The larger of the first entity evaluation index and the second entity evaluation index is used as the target entity label, and the target entity label is added to the target data set;

[0328] A target prompt is determined according to the first entity extraction result and the second entity extraction result.

[0329] In some embodiments, the prompt determination module 1040 determines the target prompt according to the first entity extraction result and the second entity extraction result in the following manner:

[0330] Determine a first prediction probability corresponding to the first entity extraction result and a second prediction probability corresponding to the second entity extraction result;

[0331] The larger of the first predicted probability and the second predicted probability is used as the target predicted probability;

[0332] If the target prediction probability is equal to the target entity label, the third comprehensive evaluation index of the first entity extraction result or the second entity extraction result is set to 1, otherwise, it is set to 0;

[0333] The average value of the third comprehensive evaluation index is used as the target evaluation score of the target data set;

[0334] Based on the target evaluation score and the third prompt, the target LLM is used to generate an intermediate entity extraction result, and the target evaluation score of the intermediate entity extraction result is calculated, and the third prompt is continuously updated according to the target evaluation score;

[0335] The third prompt corresponding to the highest target evaluation score is used as the target prompt.

[0336] In one implementation of this embodiment,

[0337] Fig.11 This is a schematic diagram of the module structure of the entity content extraction device based on LLM. Fig.11 As shown, the device includes an input module, a text polishing module, a recognition module, a verification module and an output module. The functions of each module are as follows:

[0338] 1. Input module

[0339] This module is responsible for receiving and processing the original text to be recognized and extracted as the starting point of system processing.

[0340] 2. Text polishing module

[0341] This module involves a text polishing method that optimizes the composition of LLM prompts, aiming to improve the semantic fluency and correctness of text polished by LLM. First, this module creates a text polishing dataset, which contains a series of text pairs. Each pair of texts includes a semantically unclear text (simulating the actual input text) and its corresponding correct semantic text, which is used as a label. In the processing flow, this module receives semantically unclear text and an initial polishing prompt template as input.

[0342] The polished text is generated using LLM. The quality of the output is evaluated by two main indicators: semantic fluency and similarity to the correct semantic text. This module uses these indicators as a comprehensive evaluation criterion to automatically optimize and iterate the prompt text during the polishing process. In addition, through the set number of iterations, the module continuously optimizes the input text and selects the one with the highest comprehensive evaluation index as the first stage text polishing prompt of LLM.

[0343] This module only needs the first process, and the first stage text polishing prompts will be directly adopted in the future without the need for further optimization and iteration.

[0344] 3. Identification module

[0345] This module involves a recognition and extraction method that optimizes the composition of LLM prompts, aiming to improve the accuracy and completeness of LLM recognition and extraction. First, this module creates two recognition and extraction data sets, which contain a series of text pairs. For the first data set, the semantically unclear text and the correct recognition and extraction results of the first stage will be used. For the second data set, the first stage text polishing prompts and the semantically unclear text of the first stage will be used as the input LLM to obtain the polished text and the correct recognition and extraction results. The correct recognition and extraction results in the above two data sets are the same data used as labels. In the processing flow, this module takes the text of the two data sets and an initialization recognition and extraction prompt template as input.

[0346] LLM is used to generate the text after recognition and extraction for the two datasets. The quality of the output is evaluated by two main indicators: one is the string similarity indicator, and the other is the set similarity indicator. This module uses these indicators as a comprehensive evaluation standard to automatically optimize and iterate the prompt text in the recognition and extraction process. In addition, through the set number of iterations, the module continuously optimizes the input text and selects the highest comprehensive evaluation index for dataset one and dataset two as the second stage dataset one recognition and extraction prompt of LLM and the second stage dataset two recognition and extraction prompt.

[0347] This module only needs the first process, and will directly take the second stage to identify and extract prompts, without the need for further optimization and iteration.

[0348] 4. Verification module

[0349] This module involves a decision method for optimizing the composition of LLM prompts, aiming to improve the accuracy of LLM in selecting the best option. First, this module creates a multiple-choice question dataset, which contains a question, two options and label data. For the dataset, the recognition and extraction prompts of the second-stage dataset one are used for the dataset one in the recognition and extraction module as the LLM input to obtain the recognition and extraction results of the dataset one, and the scores of the recognition and extraction results of the dataset one are calculated using the comprehensive indicators of the corresponding string similarity and set similarity. Similarly, the recognition and extraction prompts of the second-stage dataset two and the dataset two are used as LLM inputs in the same way to obtain the recognition and extraction results of the dataset two, and the scores are calculated. In the dataset, the text with unclear semantics in the text polishing module, the recognition and extraction results of the dataset one, the recognition and extraction results of the dataset two, and the highest scores of the recognition and extraction results of the dataset one and the dataset two are used as labels. In the processing flow, this module takes the text of the dataset and an initialization decision prompt template as input.

[0350] LLM generates the logits of the first token in the dataset for option 1 and option 2. The quality of the output is measured by the normalized logits score corresponding to the correct option. This indicator is used to automatically optimize and iterate the prompt text in the decision process. In addition, through the set number of iterations, the module continuously optimizes the input text and selects the one with the highest comprehensive evaluation index as the third stage decision prompt of LLM.

[0351] This module only needs the process for the first time, and will directly adopt the third stage decision prompt afterwards without the need for further optimization iteration.

[0352] 5. Output module

[0353] By extracting the original text to be recognized in the input module and passing through the above modules, the recognition and extraction result finally determined by LLM will be obtained.

[0354] In summary, a text polishing data set is created through an entity content extraction device based on LLM, and the original text and the polished text in the text polishing data set are used to continuously update the first prompt to determine the first text polishing prompt, wherein the polished text is generated through the target LLM based on the original text and the first prompt, and the first text polishing prompt is the one with the highest first comprehensive evaluation index among the first prompts; based on the text polishing data set and the first text polishing prompt, a first evaluation data set corresponding to the original text and a second evaluation data set corresponding to the polished text are determined respectively, and the first entity content corresponding to the first evaluation data set and the second entity content corresponding to the second evaluation data set are generated respectively through the target LLM and the second prompt; according to the first entity content, the second entity content and the entity label, the target is determined. The second comprehensive evaluation index of the target LLM is used to iterate the first evaluation data set and the second evaluation data set based on the second comprehensive evaluation index, determine the first entity extraction prompt corresponding to the first evaluation data set and the second entity extraction prompt corresponding to the second evaluation data set, and the entity label is the preset label corresponding to the first entity content and the second entity content; according to the first entity extraction prompt and the first evaluation data set, the second entity extraction prompt and the second evaluation data set, the first entity extraction result and the second entity extraction result are obtained respectively through the target LLM, the first entity extraction result, the second entity extraction result and the first evaluation data set are constructed as the target data set, and based on the target data set and the third prompt, the target prompt is determined from the first entity extraction result and the second entity extraction result through the target LLM. The device solves the problem of insufficient reliability and security of information extraction and improves the reliability and security of information extraction.

[0355] In the above embodiments provided by the present disclosure, the methods and devices provided by the embodiments of the present disclosure are introduced. In order to implement the functions in the methods provided by the above embodiments of the present disclosure, the electronic device may include a hardware structure and a software module, and implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. One of the above functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0356] Fig.12 It is a block diagram of an electronic device 1200 for implementing the above-mentioned entity content extraction method based on LLM according to an exemplary embodiment.

[0357] For example, the electronic device 1200 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0358] Reference Fig.12, the electronic device 1200 may include one or more of the following components: a processing component 1202 , a memory 1204 , a power component 1206 , a multimedia component 1208 , an audio component 1210 , an input / output (I / O) interface 1212 , a sensor component 1214 , and a communication component 1216 .

[0359] The processing component 1202 generally controls the overall operation of the electronic device 1200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1202 may include one or more processors 1220 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1202 may include one or more modules to facilitate the interaction between the processing component 1202 and other components. For example, the processing component 1202 may include a multimedia module to facilitate the interaction between the multimedia component 1208 and the processing component 1202.

[0360] The memory 1204 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 1200, contact data, phone book data, messages, pictures, videos, etc. The memory 1204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0361] The power supply component 1206 provides power to the various components of the electronic device 1200. The power supply component 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1200.

[0362] The multimedia component 1208 includes a screen that provides an output interface between the electronic device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1208 includes a front camera and / or a rear camera. When the electronic device 1200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0363] The audio component 1210 is configured to output and / or input audio signals. For example, the audio component 1210 includes a microphone (MIC), and when the electronic device 1200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1204 or sent via the communication component 1216. In some embodiments, the audio component 1210 also includes a speaker for outputting audio signals.

[0364] I / O interface 1212 provides an interface between processing component 1202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0365] The sensor assembly 1214 includes one or more sensors for providing various aspects of status assessment for the electronic device 1200. For example, the sensor assembly 1214 can detect the open / closed state of the electronic device 1200, the relative positioning of components, such as the display and keypad of the electronic device 1200, and the sensor assembly 1214 can also detect the position change of the electronic device 1200 or a component of the electronic device 1200, the presence or absence of user contact with the electronic device 1200, the orientation or acceleration / deceleration of the electronic device 1200, and the temperature change of the electronic device 1200. The sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0366] The communication component 1216 is configured to facilitate wired or wireless communication between the electronic device 1200 and other devices. The electronic device 1200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 1216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0367] In an exemplary embodiment, the electronic device 1200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0368] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1204 including instructions, and the instructions can be executed by the processor 1220 of the electronic device 1200 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0369] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the LLM-based entity content extraction method described in the above embodiments of the present disclosure.

[0370] The embodiments of the present disclosure further provide a computer program product, including a computer program, which, when executed by a processor, performs the entity content extraction method based on LLM described in the above embodiments of the present disclosure.

[0371] Fig.13 is a schematic structural diagram of a chip 1300 for implementing the above-mentioned LLM-based entity content extraction method according to an exemplary embodiment.

[0372] Reference Fig.13The chip 1300 includes at least one communication interface 1301 and a processor 1302; the communication interface 1301 is used to receive a signal input to the chip 1300 or a signal output from the chip 1300, and the processor 1302 communicates with the communication interface 1301 and implements the LLM-based entity content extraction method described in the above embodiment through a logic circuit or executing code instructions.

[0373] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0374] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0375] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0376] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0377] It should be understood that the various parts of the embodiments of the present disclosure can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0378] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0379] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0380] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for extracting entity content based on LLM, characterized in that: The method comprises: Creating a text polishing data set, and continuously updating a first prompt using the original text and the polished text in the text polishing data set, and determining a first text polishing prompt, wherein the polished text is generated based on the original text and the first prompt through a target LLM, and the first text polishing prompt is the one with the highest first comprehensive evaluation index among the first prompts; Based on the text polishing dataset and the first text polishing prompt, respectively determine a first evaluation dataset corresponding to the original text and a second evaluation dataset corresponding to the polished text, and respectively generate a first entity content corresponding to the first evaluation dataset and a second entity content corresponding to the second evaluation dataset through the target LLM and the second prompt; Determine a second comprehensive evaluation index of the target LLM according to the first entity content, the second entity content and the entity label, iterate the first evaluation data set and the second evaluation data set respectively based on the second comprehensive evaluation index, determine a first entity extraction prompt word corresponding to the first evaluation data set and a second entity extraction prompt word corresponding to the second evaluation data set, and the entity label is a preset label corresponding to the first entity content and the second entity content; According to the first entity extraction prompt word and the first evaluation data set, the second entity extraction prompt word and the second evaluation data set, the first entity extraction result and the second entity extraction result are respectively obtained through the target LLM, the first entity extraction result, the second entity extraction result and the first evaluation data set are constructed as a target data set, and based on the target data set and the third prompt word, the target prompt word is determined from the first entity extraction result and the second entity extraction result through the target LLM.

2. The method according to claim 1, characterized in that The step of creating a text polishing dataset, and continuously updating a first prompt using the original text and the polished text in the text polishing dataset to determine a first text polishing prompt includes: Obtain at least two preset semantically clear texts to be recognized as entities; Performing semantic interference processing on the preset semantically clear text to obtain semantically ambiguous text, and using the preset semantically clear text and the semantically ambiguous text as the original text; Based on the first prompt and the original text, generating the polished text through the target LLM; Constructing the text polishing dataset according to the original text and the polished text; Calculating the first comprehensive evaluation index between the original text and the polished text; Based on the text polishing dataset, continuously updating the first prompt according to the first comprehensive evaluation index; The first prompt corresponding to the highest value of the first comprehensive evaluation index is used as the first text polishing prompt.

3. The method according to claim 2, characterized in that The calculating the first comprehensive evaluation index between the original text and the polished text includes: Calculating a fluency index of the polished text; Calculating a semantic similarity index between the polished text and the original text; The first comprehensive evaluation index is determined according to the fluency index and the semantic similarity index.

4. The method according to claim 1, characterized in that: The determining, according to the first entity content, the second entity content and the entity tag, a second comprehensive evaluation index of the target LLM includes: Determine a string similarity index and a set similarity index between the first entity content or the second entity content and the entity tag; The second comprehensive evaluation index is determined according to the string similarity index and the set similarity index.

5. The method according to claim 4, characterized in that The determining a target prompt from the first entity extraction result and the second entity extraction result by the target LLM based on the target data set and the third prompt includes: Based on the first entity extraction result and the entity label, calculating a first entity evaluation index of the first entity extraction result according to the second comprehensive evaluation index; Based on the second entity extraction result and the entity label, calculating a second entity evaluation index of the second entity extraction result according to the second comprehensive evaluation index; Using the larger of the first entity evaluation index and the second entity evaluation index as a target entity label, and adding the target entity label to the target data set; The target prompt is determined according to the first entity extraction result and the second entity extraction result.

6. The method according to claim 5, characterized in that The step of determining the target prompt according to the first entity extraction result and the second entity extraction result includes: Determine a first prediction probability corresponding to the first entity extraction result and a second prediction probability corresponding to the second entity extraction result; taking the larger of the first predicted probability and the second predicted probability as the target predicted probability; If the target prediction probability is equal to the target entity label, the third comprehensive evaluation index of the first entity extraction result or the second entity extraction result is set to 1, otherwise, it is set to 0; Taking the average value of the third comprehensive evaluation index as the target evaluation score of the target data set; Based on the target evaluation score and the third prompt, using the target LLM to generate an intermediate entity extraction result, and calculating the target evaluation score of the intermediate entity extraction result, and continuously updating the third prompt according to the target evaluation score; The third prompt corresponding to the highest target evaluation score is used as the target prompt.

7. An entity content extraction device based on LLM, characterized in that: The device comprises: A text polishing module, used to create a text polishing data set, and continuously update a first prompt using the original text T and the polished text in the text polishing data set, and determine a first text polishing prompt, wherein the polished text is generated based on the original text and the first prompt through a target LLM, and the first text polishing prompt is the one with the highest first comprehensive evaluation index among the first prompts; An entity extraction module, configured to determine, based on the text polishing dataset and the first text polishing prompt, a first evaluation dataset corresponding to the original text and a second evaluation dataset corresponding to the polished text, respectively, and to generate, through the target LLM and the second prompt, first entity content corresponding to the first evaluation dataset and second entity content corresponding to the second evaluation dataset, respectively; An evaluation module, configured to determine a second comprehensive evaluation index of the target LLM according to the first entity content, the second entity content and the entity label, and iterate the first evaluation data set and the second evaluation data set respectively based on the second comprehensive evaluation index to determine a first entity extraction prompt word corresponding to the first evaluation data set and a second entity extraction prompt word corresponding to the second evaluation data set, wherein the entity label is a preset label corresponding to the first entity content and the second entity content; A prompt determination module obtains a first entity extraction result and a second entity extraction result respectively through the target LLM according to the first entity extraction prompt word and the first evaluation data set, the second entity extraction prompt word and the second evaluation data set, constructs the first entity extraction result, the second entity extraction result and the first evaluation data set as a target data set, and determines a target prompt from the first entity extraction result and the second entity extraction result through the target LLM based on the target data set and the third prompt.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images