Information extraction method and device based on large language model, equipment and storage medium

Abstract

The invention discloses an information extraction method and device based on a large language model, equipment and a storage medium, and relates to the technical field of information extraction, and the method comprises the following steps: converting an original text into a to-be-extracted character string; extracting entities and relationships from the to-be-extracted character string according to the pre-labeled sample; retrieving from a professional knowledge database according to the original text, the entity and the relationship to obtain a related professional knowledge text; according to the original text, the entity, the relationship and the related professional knowledge text, constructing cue words; and inputting the original text, the entity, the relationship and the cue word into a large language model, so that the large language model outputs extraction information of the original text. According to the method, entities and relationships can be accurately extracted from character strings by using a small amount of annotation data through pre-annotation samples, the extracted entities and relationships are retrieved in a professional knowledge database according to an original text, and cue words are generated in combination with retrieval contents, so that a large language model is helped to understand information needing to be extracted, and the extraction efficiency is improved. And then proper information is extracted.

Description

Technical Field

[0001] The present application relates to the technical field of information extraction, and in particular to an information extraction method, apparatus, device and storage medium based on a large language model. Background Art

[0002] With the development of artificial intelligence technology in the field of text, information extraction technology has become more mature; however, the scope of use of rule-based and statistical information extraction methods is too limited and cannot adapt to changing needs; information extraction methods based on traditional deep learning rely on high-quality annotated text, and the extraction effect is poor in complex semantics and scenarios, and it is easy to make mistakes or omissions. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an information extraction method, apparatus, device and storage medium based on a large language model to improve the accuracy of information extraction.

[0004] To achieve the above purpose, an embodiment of the present application provides an information extraction method based on a large language model, the method comprising the following steps:

[0005] Convert the original text into the string to be extracted;

[0006] Extracting entities and relations from the character string to be extracted according to the pre-labeled samples;

[0007] Retrieving relevant professional knowledge text from a professional knowledge database according to the original text, the entity and the relationship;

[0008] Constructing prompt words according to the original text, the entity, the relationship and the relevant professional knowledge text;

[0009] The original text, the entity, the relationship and the prompt word are input into a large language model, so that the large language model outputs extraction information of the original text.

[0010] In some embodiments, the converting of the original text into the character string to be extracted comprises the following steps:

[0011] Reading text from an original document to obtain the original text; or obtaining data in a text format as the original text;

[0012] The original text is divided according to the length of the text blocks to obtain a plurality of text blocks as the character strings to be extracted; wherein a set number of overlapping characters exist between adjacent text blocks.

[0013] In some embodiments, the step of extracting entities and relationships from the character string to be extracted according to the pre-labeled samples comprises the following steps:

[0014] Defining input data of a decision model according to the pre-labeled sample and the character string to be extracted;

[0015] Constructing extraction expressions of entities and relations and reward mechanism expressions in the decision model;

[0016] Inputting the input data into the decision model, outputting a target content set using the input data according to the extraction expression, and calculating an evaluation score of the target content set according to the reward mechanism expression;

[0017] Return the extraction expressions and reward mechanism expressions of the entities and relationships in the decision model to update the extraction expressions and the reward mechanism expressions until the evaluation score meets the preset conditions, and determine the target content set obtained by the latest extraction as the entities and the relationships.

[0018] In some embodiments, after inputting the original text, the entity, the relationship, and the prompt word into a large language model so that the large language model outputs the extracted information of the original text, the method further includes the following steps:

[0019] Return the entities and relations extracted from the character string to be extracted according to the pre-labeled samples, so that the large language model outputs new extracted information, until the evaluation score is greater than a preset threshold, and determine the latest extracted information as the target extracted information.

[0020] In some embodiments, the step of retrieving relevant professional knowledge text from a professional knowledge database according to the original text, the entity, and the relationship comprises the following steps:

[0021] Concatenating the original text, the entity, and the relationship, and converting the concatenated result into a combined text vector;

[0022] Calculating a similarity score between the combined text vector and the professional knowledge text vector in the professional knowledge database;

[0023] The relevant professional knowledge text is obtained by screening from the professional knowledge database according to the similarity score.

[0024] In some embodiments, constructing prompt words according to the original text, the entity, the relationship, and the related professional knowledge text includes the following steps:

[0025] Constructing a first prompt word according to the original text and the relevant professional knowledge text;

[0026] Constructing a second prompt word according to the original text, the entity and the relevant professional knowledge text;

[0027] A third prompt word is constructed according to the original text, the entity, the relationship and the relevant professional knowledge text.

[0028] In some embodiments, inputting the original text, the entity, the relationship, and the prompt word into a large language model so that the large language model outputs the extraction information of the original text includes the following steps:

[0029] The original text, the entity, the relationship, the first prompt word, the second prompt word, and the third prompt word are input into the large language model so that the large language model outputs extraction information of the original text.

[0030] To achieve the above purpose, another aspect of the embodiment of the present application provides an information extraction device based on a large language model, the device comprising:

[0031] A string conversion unit, used to convert the original text into a string to be extracted;

[0032] An entity and relationship extraction unit, used to extract entities and relationships from the character string to be extracted according to pre-labeled samples;

[0033] A knowledge retrieval unit, configured to retrieve relevant professional knowledge text from a professional knowledge database according to the original text, the entity and the relationship;

[0034] A prompt word construction unit, used to construct prompt words according to the original text, the entity, the relationship and the relevant professional knowledge text;

[0035] The information extraction unit is used to input the original text, the entity, the relationship and the prompt word into a large language model so that the large language model outputs the extracted information of the original text.

[0036] In some embodiments, the string conversion unit includes:

[0037] A text acquisition unit, used to read text from an original document to obtain the original text; or to acquire data in a text format as the original text;

[0038] The text segmentation unit is used to segment the original text according to the length of the text blocks to obtain multiple text blocks as the character strings to be extracted; wherein a set number of overlapping characters exist between each adjacent text block.

[0039] In some embodiments, the entity and relationship extraction unit includes:

[0040] An input definition unit, used to define input data of a decision model according to the pre-labeled sample and the character string to be extracted;

[0041] A model determination unit, used to construct an extraction expression of entities and relations and an expression of a reward mechanism in the decision model;

[0042] A content extraction unit, configured to input the input data into the decision model, output a target content set using the input data according to the extraction expression, and calculate an evaluation score of the target content set according to the reward mechanism expression;

[0043] A model updating unit is used to return the extraction expression and the reward mechanism expression of the entities and relationships in the decision model to update the extraction expression and the reward mechanism expression until the evaluation score meets the preset conditions and the target content set obtained by the latest extraction is determined as the entity and the relationship.

[0044] In some embodiments, the apparatus further comprises:

[0045] The repeated extraction unit is used for returning the entities and relationships extracted from the character string to be extracted according to the pre-labeled samples after the original text, the entities, the relationships and the prompt words are input into the large language model so that the large language model outputs the extracted information of the original text, so that the large language model outputs the new extracted information, until the evaluation score is greater than a preset threshold, and the latest extracted information is determined as the target extracted information.

[0046] In some embodiments, the knowledge retrieval unit includes:

[0047] A text concatenation and conversion unit, used for concatenating the original text, the entity and the relationship, and converting the concatenation result into a combined text vector;

[0048] A similarity calculation unit, used to calculate a similarity score between the combined text vector and the professional knowledge text vector in the professional knowledge database;

[0049] A knowledge screening unit is used to screen the relevant professional knowledge text from the professional knowledge database according to the similarity score.

[0050] In some embodiments, the prompt word construction unit includes:

[0051] A first prompt word construction subunit, used to construct a first prompt word according to the original text and the relevant professional knowledge text;

[0052] A second prompt word construction subunit, used to construct a second prompt word according to the original text, the entity and the relevant professional knowledge text;

[0053] The third prompt word construction subunit is used to construct a third prompt word according to the original text, the entity, the relationship and the relevant professional knowledge text.

[0054] In some embodiments, the information extraction unit includes:

[0055] The information extraction subunit is used to input the original text, the entity, the relationship, the first prompt word, the second prompt word and the third prompt word into the large language model, so that the large language model outputs the extracted information of the original text.

[0056] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0057] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0058] The embodiments of the present application include at least the following beneficial effects:

[0059] This application can convert the original text into a string to be extracted; extract entities and relationships from the string to be extracted based on pre-labeled samples; retrieve relevant professional knowledge text from the professional knowledge database based on the original text, entities and relationships; construct prompt words based on the original text, entities, relationships and relevant professional knowledge text; input the original text, entities, relationships and prompt words into the large language model so that the large language model outputs the extraction information of the original text. Pre-labeled samples can be used to accurately extract entities and relationships from the string using a small amount of labeled data, and the extracted entities and relationships are retrieved in the professional knowledge database based on the original text, and then prompt words are generated based on the search content, thereby helping the large language model understand the information that needs to be extracted, and then extract appropriate information. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 A schematic diagram of the steps of an information extraction method based on a large language model provided in an embodiment of the present application;

[0062] Figure 2 Provided in the embodiments of this application Figure 1 Example step diagram of S100 in;

[0063] Figure 3 Provided in the embodiments of this application Figure 1 Example step diagram of S120 in;

[0064] Figure 4 Provided in the embodiments of this application Figure 1 Example step diagram of S130 in;

[0065] Figure 5 An example flow chart of an information extraction method based on a large language model provided in an embodiment of the present application;

[0066] Figure 6 An example step diagram of an information extraction method based on a large language model provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of the structure of an information extraction device based on a large language model provided in an embodiment of the present application;

[0068] Figure 8 A schematic diagram of the structure of a string conversion unit provided in an embodiment of the present application;

[0069] Fig. 9 A schematic diagram of the structure of a knowledge retrieval unit provided in an embodiment of the present application;

[0070] Fig.10 A schematic diagram of the structure of a prompt word construction unit provided in an embodiment of the present application;

[0071] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0073] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0074] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0076] Before describing the embodiments of the present application in detail, some terms and related technologies involved in the embodiments of the present application are first described as follows:

[0077] LLM (Large Language Model): refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, dialogue, etc., and are an important path to artificial intelligence.

[0078] IE (Information extraction): Extracting specific events or factual information from natural language texts helps us automatically classify, extract, and reconstruct massive amounts of content. This information usually includes entities, relations, and events.

[0079] MDP (Markov decision process) is a mathematical model of sequential decision-making, which is used to simulate the stochastic strategies and rewards that can be achieved by intelligent agents in an environment where the system state has Markov properties.

[0080] With the development of artificial intelligence technology in the field of text, information extraction technology has become more mature; however, the scope of use of information extraction methods based on rules and statistics is too limited and cannot adapt to changing needs; information extraction methods based on traditional deep learning rely on high-quality annotated text, and the extraction effect is poor for complex semantics and scenes, and it is easy to make mistakes or miss. At present, applying LLM to information extraction tasks and improving information extraction efficiency with semantic understanding ability is a major application direction, but there are still problems such as complex task data fine-tuning and poor extraction quality. At the same time, the directly extracted content often needs to be integrated and output in combination with external knowledge. There is currently no mature solution or framework.

[0081] In the process of business intelligence management, some businesses need to extract specialized terms such as hardware, software, cloud network, and services from the demand documents provided by the account manager and the solution documents provided by the solution manager, and correspond them with related materials / services inside and outside the business system to obtain more specific and professional extracted / derived content. The above work process currently relies mainly on experience and manual labor, with low efficiency and a certain threshold for the work.

[0082] This application aims to provide an information extraction solution based on a large language model, which is suitable for the information extraction of business opportunities, business information and professional knowledge, and at the same time improves the robustness of LLM information extraction through an easy-to-train decision model.

[0083] Therefore, the embodiments of the present application provide an information extraction method, device, equipment and storage medium based on a large language model. The technical solution of the present application includes: converting the original text into a string to be extracted; extracting entities and relationships from the string to be extracted according to pre-labeled samples; retrieving relevant professional knowledge text from a professional knowledge database according to the original text, entities and relationships; constructing prompt words according to the original text, entities, relationships and relevant professional knowledge text; inputting the original text, entities, relationships and prompt words into a large language model, so that the large language model outputs the extracted information of the original text. Pre-labeled samples can be used to accurately extract entities and relationships from the string using a small amount of labeled data, and the extracted entities and relationships are retrieved in the professional knowledge database according to the original text, and then prompt words are generated in combination with the search content, thereby helping the large language model understand the information that needs to be extracted, and then extract appropriate information.

[0084] The embodiment of the present application provides an information extraction method based on a large language model, and relates to the field of information extraction technology. The information extraction method based on a large language model provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements an information extraction method based on a large language model, etc., but is not limited to the above forms.

[0085] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0086] Reference Figure 1 The present application embodiment provides an information extraction method based on a large language model, which may include but is not limited to S100 to S140, as follows:

[0087] S100: Convert the original text into a character string to be extracted.

[0088] Specifically, the original text is the text from which information needs to be extracted. The text may exist in any form of document. To facilitate subsequent extraction, the original text can be converted into a string.

[0089] Reference Figure 2 , further, S100 may include S101-S102:

[0090] S101: reading text from an original document to obtain the original text; or obtaining data in a text format as the original text.

[0091] It can be understood that this embodiment can extract text from documents in different formats such as word, pdf, excel, etc. as original text, or directly obtain data in text format as original text.

[0092] S102: Divide the original text according to the length of the text blocks to obtain a plurality of text blocks as the character strings to be extracted; wherein a set number of overlapping characters exist between adjacent text blocks.

[0093] Specifically, the text block slides in the original text to divide the original text into various text blocks, and each text block is used as a character string to be mentioned. Compared with directly processing a long original text, dividing the original text into text blocks can improve processing efficiency.

[0094] S110: Extracting entities and relations from the character string to be extracted according to the pre-labeled samples.

[0095] Specifically, the pre-annotated sample includes the form of the extracted information and the content to be extracted. In this embodiment, entities and relationships can be extracted from the character string to be extracted with reference to the pre-annotated sample.

[0096] Furthermore, S110 may include:

[0097] S111: Defining input data of a decision model according to the pre-labeled samples and the character string to be extracted.

[0098] The input data of the decision model needs to define a corresponding format, and this embodiment can define the input data according to the pre-labeled samples and the character string to be extracted with reference to the format of the decision input data, that is, convert the above data into the input format required by the decision model.

[0099] S112: Constructing extraction expressions of entities and relationships and reward mechanism expressions in the decision model.

[0100] It can be understood that this embodiment can define the association between entity pre-relationships and construct corresponding extraction expressions, and a reward mechanism is also provided in the extraction expression. The reward mechanism can be used to evaluate whether the content extracted by the extraction expression is appropriate. If not, the extraction expression can be adjusted according to the evaluation results.

[0101] S113: Input the input data into the decision model, output a target content set using the input data according to the extraction expression, and calculate an evaluation score of the target content set according to the reward mechanism expression.

[0102] S114: Return the extraction expressions and reward mechanism expressions of the entities and relationships in the decision model to update the extraction expressions and the reward mechanism expressions until the evaluation score meets the preset conditions, and determine the target content set obtained by the latest extraction as the entities and the relationships.

[0103] Specifically, this embodiment may return to S112 and repeatedly execute S112 to S113. If the directly evaluated score meets the preset conditions, it may indicate that the reward mechanism tends to be stable. Then, the newly extracted target content set may be determined as the entity and the relationship.

[0104] To further improve the accuracy of information extraction, the embodiment of the present application may further include the following steps:

[0105] Return the entities and relations extracted from the character string to be extracted according to the pre-labeled samples, so that the large language model outputs new extracted information, until the evaluation score is greater than a preset threshold, and determine the latest extracted information as the target extracted information.

[0106] It is understandable that this embodiment can repeatedly execute S110 to S140 according to the reward mechanism until the evaluation score is greater than the preset threshold, at which time higher accuracy extraction information can be obtained, that is, the latest extraction information can be determined as the target extraction information.

[0107] S120: Retrieve relevant professional knowledge text from a professional knowledge database according to the original text, the entity and the relationship.

[0108] This embodiment can retrieve relevant professional knowledge text from the professional knowledge database for use in large language model extraction.

[0109] Reference Figure 3 Further, S120 may include:

[0110] S121: concatenate the original text, the entity, and the relationship, and convert the concatenation result into a combined text vector;

[0111] S122: Calculating a similarity score between the combined text vector and the professional knowledge text vector in the professional knowledge database;

[0112] S123: Filtering the relevant professional knowledge text from the professional knowledge database according to the similarity score.

[0113] Specifically, after converting the concatenated text into a vector, this embodiment can calculate the similarity score of the professional knowledge text vectors in the professional knowledge database, and select several professional knowledge text vectors with top similarity scores, and determine their corresponding professional knowledge texts as relevant professional knowledge texts.

[0114] S130: Construct prompt words according to the original text, the entity, the relationship and the relevant professional knowledge text.

[0115] It can be understood that the prompt words constructed by combining the original text, entities, relations, and related professional knowledge text can enable the large language model to accurately understand the extraction requirements and thus extract information.

[0116] Reference Figure 4 Further, S130 may include:

[0117] S131: constructing a first prompt word according to the original text and the relevant professional knowledge text;

[0118] S132: constructing a second prompt word according to the original text, the entity and the relevant professional knowledge text;

[0119] S133: Construct a third prompt word according to the original text, the entity, the relationship and the relevant professional knowledge text.

[0120] It is understandable that this embodiment can use different data to construct different prompt words, so that the large language model can extract information from multiple angles.

[0121] S140: Inputting the original text, the entity, the relationship, and the prompt word into a large language model, so that the large language model outputs extraction information of the original text.

[0122] Specifically, this embodiment can use a large language model to extract information from the original text with reference to the prompt words, thereby obtaining extracted information.

[0123] Further, S130 may include:

[0124] The original text, the entity, the relationship, the first prompt word, the second prompt word, and the third prompt word are input into the large language model so that the large language model outputs extraction information of the original text.

[0125] It can be understood that in this embodiment, the three prompt words can be used together to extract information, and more abundant extracted information that is closer to actual needs can be obtained.

[0126] Next, the solution of the embodiment of the present application will be described in detail and explained in conjunction with specific application examples:

[0127] This embodiment proposes an information extraction method based on a large language model. After extracting entities and relationships in the requirement document through a decision model and a reward mechanism, professional knowledge is integrated into the LLM to achieve rapid realization of extracted requirements under specific tasks. Ultimately, it achieves the solution of information extraction problems under business opportunity and business information tasks with a small amount of annotated example data and professional knowledge data. Figure 5 This is a flowchart of LLM general information extraction that integrates professional knowledge provided in this embodiment.

[0128] Next, Figure 5 To elaborate:

[0129] 1. Preprocessing module:

[0130] This module mainly converts various unstructured documents that need to be extracted, such as word and pdf, into text strings through the text conversion engine, providing data preprocessing functions for the adaptive learning stage. At the same time, after processing professional knowledge in the form of word, pdf, and execl into text strings, text vectorization operations are performed to provide a data basis for the retrieval of relevant knowledge content in the LLM extraction module.

[0131] 2. Adaptive learning module:

[0132] The purpose of this module is to obtain the extracted entities and relations under specific task requirements by learning from a small number of pre-labeled samples. Pre-extraction is performed through the decision model to mark the internal relationship between entities and highlight the logical relationship of the content to be extracted; the decision model is rewarded or punished for extracting content through a reward mechanism to achieve adaptive adjustment of the accuracy of content extraction to improve the generalization of the model.

[0133] 3. Professional knowledge module:

[0134] This module is mainly used to retrieve relevant candidate knowledge from the professional knowledge base based on the entities and relationships output by the adaptive learning module for use by the LLM extraction module. Professional knowledge includes equipment components, brands, technical specifications, cost unit prices, implementation methods, manufacturers, etc.

[0135] 4.LLM extraction module:

[0136] This module integrates the entities and relationships output by the adaptive module and the relevant professional knowledge retrieved from the professional knowledge base into the prompt template, and uses the professional large model capabilities to extract specific content and requirements.

[0137] The specific steps of this embodiment can be referred to Figure 6 , Figure 6 There is a loop optimization logic in S2-S4, which randomly combines the entity and relationship pairs output by S2, the corresponding professional candidate knowledge retrieved in S3 and the prompt template in S4 to output the candidate results, evaluates the output results according to the reward mechanism, and iterates repeatedly to improve the accuracy of the output results. At the same time, the model parameters and prompt templates in this case are saved for production application use. The following are the detailed steps of the technical solution of this embodiment:

[0138] Step S1: Based on the input attachment, output the original d text to be extracted and initialize the professional knowledge vector library to be matched.

[0139] Step S1-1: For attachments in word, pdf, and excel input formats, use professional libraries (docx, pdfplumber, pandas) to read the text and obtain the original text string.

[0140] Step S1-1-1: If the input content is data in text format, the string of the original text output is the string to be extracted S, S is divided into text blocks of length k characters, and the number of overlapping characters between text blocks is c, so the string list to be extracted S = (S1, S2, ..., S n ), where n is the number of divided text blocks.

[0141] Step S1-1-2: If the input content is professional knowledge, the original text string is output as the professional knowledge string Z, and Z is divided into text blocks of length k characters. The number of overlapping characters between text blocks is c, so the professional knowledge string list Z = (Z1, Z2, ..., Z n ), where n is the number of divided text blocks.

[0142] Step S1-2: Perform text vectorization operation on the professional knowledge string list Z.

[0143] Step S1-2-1: Use the GTE text vector-Qwen1.5 model to vectorize the professional knowledge string list Z to obtain the professional knowledge text vector

[0144] Step S1-2-2: Transform professional knowledge text into vector Stored in the faiss vector database.

[0145] Step S2: Based on the pre-labeled samples, pre-extract the corresponding entities and relationships, and save the optimized decision model.

[0146] Step S2-1: Provide pre-labeled sample data in the following format:

[0147]

[0148] Among them, S i For a single sample text, R i is the relationship in the sample text, E i is the entity in the sample text, N i is the expertise in the sample text, and j is the number of entity-relation pairs actually existing in the sample text.

[0149] Step S2-2: According to the pre-labeled sample data, the BERT method based on the decision model (BERT-MDP) performs continuous entity-relationship extraction on the text.

[0150] Step S2-2-1: Define the input of bert-MDP (the original text of the concatenated sample data and the output result) as:

[0151] x i =[[CLS],S i ,[SEP],R i ,[SEP],E i ,[SEP]];

[0152] Among them, CLS is the start mark of text extraction, and SEP is the interval mark of text extraction.

[0153] Step S2-2-2: Define the entity-relationship modeling formula as:

[0154]

[0155] Among them, Q(S,R,E) is the target content set, r(S,R,E) is the reward mechanism, S (S,R,E) is the state set for obtaining new extracted content based on the current extracted content state, τ is the influence coefficient, max R′∈R,E′∈E Q(S′, R′, E′) is the content set extracted at that time.

[0156] Step S2-2-3: The reward mechanism is defined as follows:

[0157] Set s =Set r ∩Set e ;

[0158]

[0159] Among them, EE_F1 is the evaluation score based on words, EE_R and EE_E are the correct proportions of relations and entities respectively, and Set s , Set r , Set eThey are the sets of correctly extracted entity-relationship, relationship, and entity respectively; TextP is the precision of entity-relationship pairs, and TextR is the recall of entity-relationship pairs.

[0160] Step S2-2-4: According to step S2-2-2 and step S2-2-3, the first round of iterative model is performed until r(S, R, E) tends to be stable.

[0161] Step S2-2-5: Output the entity-relationship pair result set Result (S, R, E).

[0162] Step S3: Calculate the entity and relationship set Result (S, R, E) output from step S2 from the professional knowledge vector library Match and output relevant professional knowledge text N.

[0163] Step S3-1: Combine the text S, entity E, and relation R, and use the GTE text vector-Qwen1.5 model to convert them into a combined text vector Query, where:

[0164]

[0165] Step S3-2: Calculate the query and professional knowledge text vector Perform related content retrieval calculation, the calculation formula is as follows:

[0166]

[0167] Among them, Query j Indicates the text to be matched, Z i Represents the professional knowledge vector library.

[0168] Step S3-3: Obtain matching professional knowledge data N before the similarity score Cnt according to the formula S3-2 Cnt =(Z′1,Z′2,...,Z′ Cnt ).

[0169] Step S4: Apply LLM+prompt+entity relationship to Result(S,R,E)+external knowledge to construct corpus N Cnt , complete the extraction of specific task information.

[0170] Step S4-1: Construct a prompt template based on external knowledge.

[0171] Step S4-1-1: original text S+prompt template definition P s :

[0172] P s = "You are an expert in the field of business opportunities and information. Please combine your professional knowledge Z'i , extract information from S in the form of {R,E,N}".

[0173] Step S4-1-2: original text S+E+prompt template definition P (s,e) :

[0174] P (s,e) = "You are an expert in the field of business opportunities and information. Please combine your professional knowledge Z' i With the reference entity E, information is extracted from S in the form of {R,E,N}".

[0175] Step S4-1-3: original text S+E+R+prompt template definition P (s,e,r) :

[0176] P (s,e,r) = "You are an expert in the field of business opportunities and information. Please combine your professional knowledge Z' i , refer to the entity and relationship pair (E,R), and extract information from S in the form of {R,E,N}".

[0177] Step S4-2: Combine the entity-relationship pair Result(S, R, E) output from step S2 and the external knowledge construction corpus N output from step S3 Cnt , fill in the prompt template of step S4-1.

[0178] Step S4-3: Use LLM to extract content in professional fields.

[0179]

[0180] where X = [x1, x2, ..., x n ] is the text sequence to be extracted Result(S,R,E), Y=[y1,y2,...,y n ] is the target sequence (R, E, N), P is the prompt word template after filling in the content (P s , P (s,e) , P (s,e,r) ), θ is the parameter of LLMs for training or freezing; p θ (Y|X,P) represents the probability of completely extracting the correct answer in the prompt word template after filling in the content, p θ (y i |X,P,y ＜i ) represents the probability of correctly extracting a single content in the prompt word template after filling in the content.

[0181] Step S4-4: Loop through steps S2-S4 until EE_F1>Target score , stop the loop and output the optimal extraction result (R, E, N).

[0182] The following is a more specific implementation example, namely an information extraction task based on hardware, software, and cloud network.

[0183] Step T1: The user inputs the required document as a traffic brain.doc; it is expected to parse the hardware, software, and cloud network content in the document, and output the corresponding content entities and relationships in combination with the business-related management methods. The construction of the professional knowledge vector library is detailed in the above step S1.

[0184] Step T2: Construct pre-labeled samples.

[0185] 1. Hardware:

[0186] Original text S: The traffic AI video audit platform includes a private network firewall, core network switching equipment, and network switching equipment (including a pair of optical modules); the corresponding specifications are as follows:

[0187] Output information (R, E, N): (Hardware-Traffic AI Video Audit Platform-Device Name), (Traffic AI Video Audit Platform, Private Network Firewall, Device Components), (Traffic AI Video Audit Platform, Core Network Switching Device, Device Components), (Traffic AI Video Audit Platform, Network Switching Device, Device Components),

[0188] (Private network firewall, equipment model), (Private network firewall, general centralized procurement, implementation method).

[0189] 2. Software:

[0190] Original text S: It is planned to build a transportation big data middle platform,.... The construction content includes a resource library, including a basic database, a business database,... used to build...; a unified identity authentication system includes user management, unified organization management,... to complete the system...; a unified integrated communication dispatching system, including voice integration / traditional PBX / IP-PBX integration docking, voice integration / wireless intercom system docking,... to support...

[0191] Output information (R, E, N): (software, transportation big data middle platform, system / platform), (transportation big data middle platform, basic database, subsystem / submodule), (basic database, self-developed, implementation method).

[0192] 3. Cloud network resources:

[0193] Original text S: By building a Web application firewall resource pool and pulling user traffic to the protection pool, ..., it provides fine-grained application layer DDoS attack protection functions to ensure its security and legitimacy, and blocks illegal requests in real time, thereby effectively protecting various websites, reducing the impact of attacks, and ensuring the continuity and availability of business systems.

[0194] Output information (R, E, N): (cloud network resources, Web application firewall, product name), (Web application firewall, manufacturer), (Web application firewall, general centralized procurement, implementation method).

[0195] Step T3: Initialize relevant parameters for training. i [[CLS], communications, A.I., video, audit, platform,...,[SEP], hardware, [SEP], communications, A.I., video, audit, platform, [SEP]].

[0196] Step T4: Retrieve professional knowledge text based on the output entity-relationship pairs of T3, such as Result(S, R, E) = ("The traffic AI video audit platform includes a private network firewall, core network switching equipment, network switching equipment (including a pair of optical modules); the corresponding specifications are as follows:....", hardware, traffic AI video audit platform) retrieve the corresponding knowledge candidate set N = ("The video audit platform includes...", ...).

[0197] Step T5: Based on the results Result (S, R, E), N output by T3 and T4, use the prompt template to generate a large model. Some examples are as follows: (s,e,r) = "You are an expert in the field of business opportunities and information. Please combine your professional knowledge "The video audit platform includes..." and refer to the entity and relationship pairs (hardware, traffic AI video audit platform) to extract information from S in the form of {hardware, traffic AI video audit platform, implementation requirements}."

[0198] Step T6: Finally, the LLM module loop outputs the text extraction result set: (hardware-traffic AI video audit platform-device name), (traffic AI video audit platform, private network firewall, device components), (traffic AI video audit platform, core network switching equipment, device components), (traffic AI video audit platform, network switching equipment, device components), (private network firewall, equipment model), (private network firewall, general centralized procurement, implementation method).

[0199] Beneficial effects:

[0200] 1. Combining information extraction with professional knowledge (business opportunity and business information knowledge). Different from the traditional extraction task of directly extracting text, this embodiment derives the extracted content based on external knowledge and can directly obtain the expected results.

[0201] 2. Unlike traditional extraction tasks that directly apply pre-trained models, this embodiment combines pre-trained model fine-tuning, professional knowledge, and large model extraction to complete professional task extraction.

[0202] 3. This embodiment transforms the decision-making and reward mechanism of the pre-trained model so that its output is more consistent with the original input extracted by the large model, thereby improving the extraction effect.

[0203] Reference Figure 7 The embodiment of the present application further provides an information extraction device based on a large language model, which can implement the above-mentioned information extraction method based on a large language model, and the device includes:

[0204] A string conversion unit, used to convert the original text into a string to be extracted;

[0205] An entity and relationship extraction unit, used to extract entities and relationships from the character string to be extracted according to pre-labeled samples;

[0206] A knowledge retrieval unit, configured to retrieve relevant professional knowledge text from a professional knowledge database according to the original text, the entity and the relationship;

[0207] A prompt word construction unit, used to construct prompt words according to the original text, the entity, the relationship and the relevant professional knowledge text;

[0208] The information extraction unit is used to input the original text, the entity, the relationship and the prompt word into a large language model so that the large language model outputs the extracted information of the original text.

[0209] Reference Figure 8 In some embodiments, the string conversion unit includes:

[0210] A text acquisition unit, used to read text from an original document to obtain the original text; or to acquire data in a text format as the original text;

[0211] The text segmentation unit is used to segment the original text according to the length of the text blocks to obtain multiple text blocks as the character strings to be extracted; wherein a set number of overlapping characters exist between each adjacent text block.

[0212] In some embodiments, the entity and relationship extraction unit includes:

[0213] An input definition unit, used to define input data of a decision model according to the pre-labeled sample and the character string to be extracted;

[0214] A model determination unit, used to construct an extraction expression of entities and relations and an expression of a reward mechanism in the decision model;

[0215] A content extraction unit, configured to input the input data into the decision model, output a target content set using the input data according to the extraction expression, and calculate an evaluation score of the target content set according to the reward mechanism expression;

[0216] A model updating unit is used to return the extraction expression and the reward mechanism expression of the entities and relationships in the decision model to update the extraction expression and the reward mechanism expression until the evaluation score meets the preset conditions and the target content set obtained by the latest extraction is determined as the entity and the relationship.

[0217] In some embodiments, the apparatus further comprises:

[0218] The repeated extraction unit is used for returning the entities and relationships extracted from the character string to be extracted according to the pre-labeled samples after the original text, the entities, the relationships and the prompt words are input into the large language model so that the large language model outputs the extracted information of the original text, so that the large language model outputs the new extracted information, until the evaluation score is greater than a preset threshold, and the latest extracted information is determined as the target extracted information.

[0219] Reference Fig. 9 In some embodiments, the knowledge retrieval unit includes:

[0220] A text concatenation and conversion unit, used for concatenating the original text, the entity and the relationship, and converting the concatenation result into a combined text vector;

[0221] A similarity calculation unit, used to calculate a similarity score between the combined text vector and the professional knowledge text vector in the professional knowledge database;

[0222] A knowledge screening unit is used to screen the relevant professional knowledge text from the professional knowledge database according to the similarity score.

[0223] Reference Fig.10 In some embodiments, the prompt word construction unit includes:

[0224] A first prompt word construction subunit, used to construct a first prompt word according to the original text and the relevant professional knowledge text;

[0225] A second prompt word construction subunit, used to construct a second prompt word according to the original text, the entity and the relevant professional knowledge text;

[0226] The third prompt word construction subunit is used to construct a third prompt word according to the original text, the entity, the relationship and the relevant professional knowledge text.

[0227] In some embodiments, the information extraction unit includes:

[0228] The information extraction subunit is used to input the original text, the entity, the relationship, the first prompt word, the second prompt word and the third prompt word into the large language model, so that the large language model outputs the extracted information of the original text.

[0229] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0230] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned information extraction method based on a large language model when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0231] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0232] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0233] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0234] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the information extraction method based on a large language model in the embodiment of this application;

[0235] Input / output interface 1103, used to implement information input and output;

[0236] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0237] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0238] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0239] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned information extraction method based on a large language model.

[0240] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0241] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0242] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0243] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0244] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0245] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0246] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0247] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0248] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0249] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0250] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0251] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0252] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. An information extraction method based on a large language model, characterized in that: The method comprises the following steps: Convert the original text into the string to be extracted; Extracting entities and relations from the character string to be extracted according to the pre-labeled samples; Retrieving relevant professional knowledge text from a professional knowledge database according to the original text, the entity and the relationship; Constructing prompt words according to the original text, the entity, the relationship and the relevant professional knowledge text; The original text, the entity, the relationship and the prompt word are input into a large language model, so that the large language model outputs extraction information of the original text.

2. The information extraction method based on a large language model according to claim 1, characterized in that: The step of converting the original text into a character string to be extracted comprises the following steps: Reading text from an original document to obtain the original text; or obtaining data in a text format as the original text; The original text is divided according to the length of the text blocks to obtain a plurality of text blocks as the character strings to be extracted; wherein a set number of overlapping characters exist between adjacent text blocks.

3. The information extraction method based on a large language model according to claim 1, characterized in that: The step of extracting entities and relationships from the character string to be extracted according to the pre-labeled samples comprises the following steps: Defining input data of a decision model according to the pre-labeled sample and the character string to be extracted; Constructing extraction expressions of entities and relations and reward mechanism expressions in the decision model; Inputting the input data into the decision model, outputting a target content set using the input data according to the extraction expression, and calculating an evaluation score of the target content set according to the reward mechanism expression; Return the extraction expressions and reward mechanism expressions of the entities and relationships in the decision model to update the extraction expressions and the reward mechanism expressions until the evaluation score meets the preset conditions, and determine the target content set obtained by the latest extraction as the entities and the relationships.

4. The information extraction method based on a large language model according to claim 3, characterized in that: After inputting the original text, the entity, the relationship and the prompt word into the large language model so that the large language model outputs the extraction information of the original text, the method further includes the following steps: Return the entities and relations extracted from the character string to be extracted according to the pre-labeled samples, so that the large language model outputs new extracted information, until the evaluation score is greater than a preset threshold, and determine the latest extracted information as the target extracted information.

5. The information extraction method based on a large language model according to claim 1, characterized in that: The step of retrieving relevant professional knowledge text from a professional knowledge database according to the original text, the entity and the relationship comprises the following steps: Concatenating the original text, the entity, and the relationship, and converting the concatenated result into a combined text vector; Calculating a similarity score between the combined text vector and the professional knowledge text vector in the professional knowledge database; The relevant professional knowledge text is obtained by screening from the professional knowledge database according to the similarity score.

6. The information extraction method based on a large language model according to claim 1, characterized in that: The step of constructing prompt words according to the original text, the entity, the relationship, and the related professional knowledge text includes the following steps: Constructing a first prompt word according to the original text and the relevant professional knowledge text; Constructing a second prompt word according to the original text, the entity and the relevant professional knowledge text; A third prompt word is constructed according to the original text, the entity, the relationship and the relevant professional knowledge text.

7. The information extraction method based on a large language model according to claim 6, characterized in that: The step of inputting the original text, the entity, the relationship and the prompt word into a large language model so that the large language model outputs the extraction information of the original text comprises the following steps: The original text, the entity, the relationship, the first prompt word, the second prompt word, and the third prompt word are input into the large language model so that the large language model outputs extraction information of the original text.

8. An information extraction device based on a large language model, characterized in that: The device comprises: A string conversion unit, used to convert the original text into a string to be extracted; An entity and relationship extraction unit, used to extract entities and relationships from the character string to be extracted according to pre-labeled samples; A knowledge retrieval unit, configured to retrieve relevant professional knowledge text from a professional knowledge database according to the original text, the entity and the relationship; A prompt word construction unit, used to construct prompt words according to the original text, the entity, the relationship and the relevant professional knowledge text; The information extraction unit is used to input the original text, the entity, the relationship and the prompt word into a large language model so that the large language model outputs the extracted information of the original text.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the information extraction method based on a large language model as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, an information extraction method based on a large language model as described in any one of claims 1 to 7 is implemented.

Images