Information extraction method and related equipment

Abstract

The present application provides an information extraction method and related equipment, the information extraction method comprising: extracting information from a text to be extracted based on elements to be extracted, obtaining M first extraction results, where M is an integer greater than 1; performing inconsistency detection on the M first extraction results to obtain a detection result, the detection result being used to reflect the inconsistency between the M first extraction results; and determining a second extraction result of the element to be extracted based on the detection result, so as to improve the accuracy of element extraction.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an information extraction method and related equipment. Background Art

[0002] In daily life or work, there are more and more scenarios where texts are used, so it is becoming increasingly important to check texts. For example, in normal business exchanges and activities, in order to protect the interests of the parties involved and provide legal protection, the parties involved need to sign a large number of contract texts. In order to ensure the legality of the contract texts and avoid risks, the contract signatories need to review / check the contract before it is officially signed.

[0003] The commonly used text review method is the text intelligent inspection method. In text intelligent inspection, element extraction is the key. Therefore, how to obtain the content corresponding to the element to be extracted from the text has become one of the hot issues in this field. Summary of the invention

[0004] The embodiments of the present application provide an information extraction method and related equipment, which are conducive to improving the accuracy of factor extraction.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an information extraction method, comprising:

[0007] Extract information from the text to be extracted based on the elements to be extracted, and obtain M first extraction results, where M is an integer greater than 1;

[0008] Performing inconsistency detection on the M first extraction results to obtain a detection result; the detection result is used to reflect the inconsistency between the M first extraction results;

[0009] A second extraction result of the element to be extracted is determined according to the detection result.

[0010] In a second aspect, an embodiment of the present application further provides an information extraction device, comprising:

[0011] An extraction module, used for extracting information from the text to be extracted based on the elements to be extracted, and obtaining M first extraction results, where M is an integer greater than 1;

[0012] A determination module, configured to perform an inconsistency detection on the M first extraction results to obtain a detection result, wherein the detection result is used to reflect the inconsistency between the M first extraction results;

[0013] The determination module is further used to determine a second extraction result of the element to be extracted according to the detection result.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned information extraction method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned information extraction method are implemented.

[0016] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which implements the steps in the above-mentioned information extraction method when executed by a processor.

[0017] It can be seen that in this embodiment, based on the elements to be extracted, information extraction is performed on the text to be extracted, and M first extraction results of the elements to be extracted can be obtained. The M first extraction results can be statistically analyzed to detect the inconsistency of the M first extraction results to obtain the detection result. The detection result reflects the inconsistency of the M first extraction results, or can characterize the degree of inconsistency of the M first extraction results. It can also be understood that it characterizes the differences of the M first extraction results. For the same element to be extracted, the smaller the difference between the M first extraction results, that is, the smaller the inconsistency, it can be said that the extraction effect for the same element to be extracted is better, and vice versa, the worse the extraction effect is. The inconsistency parameter reflects the extraction effect of the M first extraction results to a certain extent. Therefore, in the information extraction method of this embodiment, after performing information extraction to obtain the M first extraction results of the element to be extracted, the M first extraction results can be tested for inconsistency, that is, the extraction effect of the M first extraction results is tested, and the detection results of the M first extraction results are obtained, and the detection results of multiple first extraction results are further used to determine the second extraction result. The differences between the M extraction results are used to select the second extraction result, so as to achieve secondary screening of the extraction results, which can improve the accuracy of the extraction results of the elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 This is one of the flow charts of an information extraction method provided in an embodiment of the present application;

[0020] Figure 2 This is the second flowchart of an information extraction method provided in an embodiment of the present application;

[0021] Figure 3 is a structural schematic diagram of an information extraction device provided in an embodiment of the present application;

[0022] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] At present, in order to realize text detection, etc., elements of the text can be extracted. However, in the process of information extraction, the commonly used method is to extract information through entity recognition models such as Long Short-Term Memory (LSTM)-Conditional Random Field (CRF) model, Bidirectional Encoder Representations from Transformers (BERT)-CRF and other models, and directly use the results obtained by the model as the final extraction results of the elements to be extracted, which easily leads to poor accuracy of the extraction results of the elements to be extracted.

[0025] Based on this, an embodiment of the present application provides an information extraction method, which can further detect the inconsistency of the M first extraction results on the basis of extracting information from the text to be extracted to obtain M first extraction results of the element to be extracted, and obtain a detection result. The detection result can characterize the inconsistency of the M first extraction results, and can also be understood as characterizing the degree of inconsistency / difference of the M first extraction results. For the same element to be extracted, the smaller the difference between the M first extraction results, the smaller the inconsistency, which can indicate that the extraction effect for the same element to be extracted is better, and vice versa, the worse the extraction effect is. The inconsistency parameter reflects the extraction effect of the M first extraction results to a certain extent. Therefore, in the information extraction method of this embodiment, after performing information extraction to obtain M first extraction results of the element to be extracted, the M first extraction results can be detected for inconsistency, that is, the extraction effect of the M first extraction results is detected. After obtaining the detection results of the M first extraction results, the detection results of the multiple first extraction results are further used to determine the second extraction result of the element to be extracted. By utilizing the differences among the M extraction results, the second extraction result is selected therefrom, thereby realizing a secondary screening of the extraction results, and the accuracy of the extraction results of the elements can be improved.

[0026] See also Figure 1 , Figure 1 1 is a flow chart of an information extraction method provided by an embodiment of the present application. The information extraction method of this embodiment can be applied to an electronic device, which can be a mobile device or a non-mobile device. Figure 1 As shown, the following steps are included:

[0027] Step 101: extract information from the text to be extracted based on the elements to be extracted, and obtain M first extraction results, where M is an integer greater than 1.

[0028] The text to be extracted can be understood as any text from which elements need to be extracted, and the elements to be extracted can be pre-set elements to be extracted. In different application scenarios, the text to be extracted can be the corresponding text in the scenario, and the elements to be extracted are the corresponding elements in the scenario, for example, the pre-set elements to be extracted corresponding to the scenario, etc. For example, the method can be applied to scenarios including but not limited to contract review scenarios, ruling scenarios, case query scenarios, work report scenarios, etc., and correspondingly, the elements to be extracted can include but are not limited to elements corresponding to scenarios such as contract review scenarios, ruling scenarios, case query scenarios, and work report scenarios. In addition, correspondingly, the text to be extracted can include but are not limited to contract texts, ruling documents, case texts, work report texts, etc. For example, for the contract review scenario, the text to be extracted can be the contract text, which can also be called the contract text to be extracted or the contract text to be reviewed, etc., and the corresponding elements to be extracted can include but are not limited to at least one of the elements such as the name of Party A, the latest delivery time, the total contract price, and the invoice type.

[0029] Exemplarily, the number of elements to be extracted may be one or more (e.g., K, where K is an integer greater than 1). In the case of K elements to be extracted, for each of the K elements to be extracted, a corresponding result is extracted from the text to be extracted. For each element to be extracted, information extraction is performed separately to obtain corresponding M first extraction results. It should be understood that the first extraction result is the content corresponding to the element to be extracted obtained by performing information extraction on the text to be extracted. Exemplarily, the elements to be extracted and the text to be extracted may be input into a large language model for information extraction, and M first extraction results for each element to be extracted are generated through M reasoning paths of the large language model. Alternatively, the elements to be extracted and the text to be extracted may be input into M language models for information extraction to obtain the first extraction results output by the M language models, that is, M first extraction results (one-to-one corresponding to the M language models), etc.

[0030] Step 102: Perform inconsistency detection on the M first extraction results to obtain a detection result.

[0031] The detection result may include an inconsistency parameter, which is used to reflect the inconsistency between the M first extraction results. For the element to be extracted, the corresponding M first extraction results may have the same extraction results or different extraction results. Among the M first extraction results required to be extracted, the more the number of the same extraction results is, the stronger the consistency of the extraction results of the element to be extracted (the weaker the uncertainty (uncertainty) of the extraction results of the element to be extracted), that is, the smaller the inconsistency, the fewer the number of the same extraction results, which means that the consistency of the extraction results of the element to be extracted is weaker (the stronger the uncertainty of the extraction results of the element to be extracted), that is, the greater the inconsistency. The detection result of the element to be extracted can be used to characterize the inconsistency of the M first extraction results of the element to be extracted, that is, the degree of inconsistency / difference. For example, the smaller the detection result, the smaller the inconsistency of the M first extraction results, that is, the better the extraction effect for the element to be extracted. Conversely, the larger the detection result, the worse the extraction effect for the element to be extracted, which reflects the extraction effect of the M first extraction results to a certain extent.

[0032] Exemplarily, by counting the M first extraction results corresponding to the element to be extracted, the number of different extraction results in the M first extraction results of the element to be extracted can be obtained, and the detection result of the element to be extracted can be determined by using the number of different extraction results corresponding to the element to be extracted (i.e., how many different extraction results there are in the M first extraction results of the element to be extracted). Exemplarily, the detection result of the element to be extracted can be proportional to the number of different extraction results of the element to be extracted, etc., and is not specifically limited. For example, it can include but is not limited to taking the ratio of the number of different extraction results of the element to be extracted to M as the detection result of the element to be extracted, etc. Exemplarily, when there are multiple elements to be extracted, for each of the multiple elements to be extracted, the corresponding M first extraction results can be determined respectively, and the consistency test can be performed on the M first extraction results of each element to be extracted respectively to obtain the consistency parameter corresponding to each element to be extracted.

[0033] Step 103: Determine a second extraction result of the element to be extracted according to the detection result.

[0034] It should be noted that when there are multiple elements to be extracted, for each of the multiple elements to be extracted, the corresponding second extraction result is determined according to the corresponding test result, and the second extraction result of each element to be extracted can be obtained. For example, in the contract review scenario, the content corresponding to at least one of the elements such as Party A's name, the latest delivery time, the total contract price, and the invoice type can be determined. For example, the content of Party A's name element is A Co., Ltd., the content of the latest delivery time is December 29, the content of the total contract price is X million, and the content of the invoice type is ordinary invoice, etc.

[0035] In this embodiment, based on the elements to be extracted, information is extracted from the text to be extracted, and M first extraction results of the elements to be extracted can be obtained. The M first extraction results can be statistically analyzed to detect the inconsistency of the M first extraction results to obtain the detection result. The detection result reflects the inconsistency of the M first extraction results, or can characterize the degree of inconsistency of the M first extraction results. It can also be understood that it characterizes the differences of the M first extraction results. For the same element to be extracted, the smaller the difference between the M first extraction results, that is, the smaller the inconsistency, it can be said that the extraction effect for the same element to be extracted is better, and vice versa, the worse the extraction effect is. The inconsistency parameter reflects the extraction effect of the M first extraction results to a certain extent. Therefore, in the information extraction method of this embodiment, after performing information extraction to obtain the M first extraction results of the element to be extracted, the M first extraction results can be tested for inconsistency, that is, the extraction effect of the M first extraction results is tested, and the detection results of the M first extraction results are obtained, and the detection results of multiple first extraction results are further used to determine the second extraction result. The differences between the M extraction results are used to select the second extraction result, so as to achieve secondary screening of the extraction results, which can improve the accuracy of the extraction results of the elements.

[0036] In some embodiments, performing inconsistency detection on the M first extraction results to obtain a detection result includes:

[0037] De-duplicating the M first extraction results to obtain the first extraction results after de-duplication;

[0038] According to the number of results of the first extraction result after deduplication and M, the detection result of the element to be extracted is determined.

[0039] It should be understood that the first extraction result after deduplication of the element to be extracted can represent different extraction results among the M first extraction results of the element to be extracted, that is, the number of the first extraction results after deduplication of the element to be extracted can represent the number of different extraction results among the M first extraction results of the element to be extracted. The detection result of the element to be extracted is determined by the number of results of the first extraction result after deduplication and M to improve the accuracy of the detection result of the element to be extracted.

[0040] Exemplarily, the detection result of the element to be extracted can be directly proportional to the number of results of the first extraction result after deduplication of the element to be extracted, and inversely proportional to M. There is no specific limitation. For example, the number of results of the first extraction result after deduplication of the element to be extracted can be divided by M to determine the detection result of the element to be extracted.

[0041] In some embodiments, information is extracted from the text to be extracted based on the elements to be extracted to obtain M first extraction results, including:

[0042] Based on the elements to be extracted and the prompt information of the elements to be extracted, information is extracted from the text to be extracted to obtain M first extraction results of the elements to be extracted.

[0043] It should be understood that the prompt information of the elements to be extracted can be obtained first to guide the extraction of elements, etc., which can include element extraction examples, etc. In this embodiment, the prompt information of the elements to be extracted can be used to extract the elements to be extracted from the text to be extracted, and M first extraction results of the elements to be extracted can be obtained. In this way, the extraction of the elements to be extracted can be guided in the text to be extracted through the prompt information of the elements to be extracted, thereby improving the accuracy of the first extraction result of the elements to be extracted.

[0044] It should be understood that the prompt information of the elements to be extracted can also be called the element extraction prompt information of the elements to be extracted. For example, the prompt information can be a chain of thought prompt (COT Prompt). It should be understood that the chain of thought prompt refers to guiding the model to decompose a complex problem into many sub-problems through some examples, and finally obtain the result of the complex problem. For example, the chain of thought prompt can be a small sample chain of thought prompt (Few-Shot COTPrompt): according to the format of the chain of thought prompt, a small number of example prompts are given to the model, so that the model returns an answer that is more in line with the prompt. In the element extraction process of this example, the chain of thought prompt of the elements to be extracted is also added to improve the accuracy of the M first extraction results of the elements to be extracted.

[0045] In some embodiments, based on the elements to be extracted and the prompt information of the elements to be extracted, information extraction is performed on the text to be extracted to obtain M first extraction results of the elements to be extracted, including:

[0046] The elements to be extracted, the prompt information of the elements to be extracted and the text to be extracted are used as the input information of the information extraction model, and information extraction is performed based on the input information through M reasoning paths in the information extraction model to generate M first extraction results.

[0047] The information extraction model can be understood as a model that can be used for information extraction. There are many types of such models, which are not specifically limited in the embodiments of the present application. For example, illustratively, the information extraction model may include but is not limited to a large language model (LLM). A large language model refers to a natural language processing model with a large number of parameters. This model is usually trained based on a large amount of computing resources and training data, so it can handle various complex natural language tasks, including but not limited to language understanding, generation, translation, etc. The M first extraction results of each element to be extracted are generated by the M reasoning paths of the large language model to improve the accuracy of the M first extraction results of the element to be extracted.

[0048] In the process of extracting information to obtain the first extraction result in the present embodiment, M first extraction results of the elements to be extracted are generated by M reasoning paths of the information extraction model, that is, the M first extraction results correspond one-to-one to the M reasoning paths, which can avoid the situation where a single decoding strategy is used to generate extraction results based on only one reasoning path, which is easy to cause erroneous extraction results. In addition, in the process of extracting information by the model, prompt information is also used to guide the model to extract information according to the prompt information, thereby improving the accuracy of element extraction.

[0049] In some embodiments, the information extraction model may be a trained information extraction model, which may be trained in the following manner:

[0050] Obtain the standard extraction results of the elements to be extracted in the training text;

[0051] The information extraction model is trained using the training text, the elements to be extracted, the prompt information of the elements to be extracted, and the standard extraction results of the elements to be extracted in the training text to obtain a trained information extraction model.

[0052] That is, during the training process, the training text, the elements to be extracted, the prompt information of the elements to be extracted, and the standard extraction results of the elements to be extracted in the training text are used as inputs of the information extraction model. The information extraction model can extract the elements to be extracted from the training text based on the prompt information of the elements to be extracted, and obtain the predicted extraction results of the elements to be extracted. For example, for each element to be extracted, M predicted extraction results of the extracted element can be obtained through M reasoning paths of the information extraction model. By comparing the gap between the predicted extraction results of the elements to be extracted and the corresponding standard extraction results, the information extraction model is adjusted to achieve the training of the information extraction model. During the training process, the prompt information of the elements to be extracted is introduced to guide the model to perform reasoning, obtain the predicted extraction results of the elements to be extracted, improve the accuracy of element extraction, compare the predicted extraction results of the element extraction output by the model with the standard extraction results, iterate the model, and achieve model training. In this way, this embodiment introduces prompt information of the elements to be extracted during the training process of the information extraction model, which can guide the reasoning ability of the model, that is, the ability to guide the model to extract information, so that the model can learn the reasoning ability in the element extraction process, so that it can better extract information, and subsequently extract information through the trained information extraction model to improve the accuracy of information extraction.

[0053] In some embodiments, determining a second extraction result of the element to be extracted according to the detection result includes:

[0054] When the detection result of the element to be extracted is less than a preset threshold, a second extraction result of the element to be extracted is determined from the M first extraction results of the element to be extracted.

[0055] It should be understood that the preset threshold value can be preset, for example, it can be preset according to actual needs or historical experience, without specific limitation, for example, the preset threshold value can be set to 0.3. It should be understood that the larger the detection result of the M first extraction results of the element to be extracted, the more the number of identical extraction results in the M first extraction results, the fewer the number of different extraction results, which can indicate that the extraction effect of the element to be extracted is worse, and vice versa, it can indicate that the extraction effect of the element to be extracted is better. Therefore, in this embodiment, the detection result can be compared with the preset threshold value. When the detection result is less than the preset threshold value, it means that the detection result is smaller and the extraction effect is better, that is, the extraction of the element to be extracted is no longer optimized, and the second extraction result of the element to be extracted can be determined from the M first extraction results of the element to be extracted. Thus, while ensuring the extraction effect, more energy can be spent on the extraction of elements with higher uncertainty (i.e., higher inconsistency), which can save development time.

[0056] In some embodiments, determining a second extraction result of the element to be extracted according to the detection result includes:

[0057] When the detection result of the element to be extracted is greater than or equal to the preset threshold, the following steps are executed repeatedly until the updated detection result of the element to be extracted is less than the preset threshold. When the updated detection result of the element to be extracted is less than the preset threshold, the second extraction result of the element to be extracted is determined from the updated M first extraction results of the element to be extracted:

[0058] Update the prompt information of the elements to be extracted;

[0059] Based on the elements to be extracted and the updated prompt information of the elements to be extracted, information is extracted from the text to be extracted to obtain the updated M first extraction results of the elements to be extracted;

[0060] An inconsistency detection is performed on the updated M first extraction results, and the detection result is updated. The updated detection result is used to characterize the inconsistency of the updated M first extraction results of the element to be extracted.

[0061] When the detection result of the element to be extracted is greater than or equal to the preset threshold, the prompt information of the element to be extracted can be updated, and based on the element to be extracted and the updated prompt information of the element to be extracted, the information extraction of the text to be extracted is performed again to obtain M new extraction results of the element to be extracted, so as to update the M first extraction results, that is, the updated M first extraction results are the M new extraction results obtained by re-extracting the information of the text to be extracted, and the updated M first extraction results of the element to be extracted are re-checked for inconsistency, so as to obtain a new detection result of the element to be extracted. The test result, that is, the test result of the element to be extracted is updated, and the updated test result of the element to be extracted is continued to be compared with the preset threshold. When the updated test result is less than the preset threshold, the second extraction result of the element to be extracted can be determined from the updated M first extraction results of the element to be extracted. When the updated test result is greater than or less than the preset threshold, the above process is continued to be repeated until the updated test result of the element to be extracted is less than the preset threshold, and the second extraction result of the element to be extracted is determined from the updated M first extraction results of the element to be extracted. It should be noted that the M first extraction results after each update refer to the M first extraction results that are most recently updated in the latest round.

[0062] That is, in this embodiment, the extraction results of the elements to be extracted are evaluated for uncertainty, that is, the detection results are estimated, and the extraction of the elements to be extracted with higher inconsistency is optimized, that is, the prompt information is updated, and the updated prompt information is used to re-extract information from the text to be extracted to update the M first extraction results of the elements to be extracted, and the detection results of the elements to be extracted are updated according to the updated M first extraction results of the elements to be extracted, until the inconsistency of the extraction results of the elements to be extracted is lower than the preset threshold and becomes an element with low inconsistency, and the second extraction result is determined from the updated M first extraction results, thereby improving the accuracy of element extraction.

[0063] In some embodiments, determining the second extraction result of the element to be extracted from the M first extraction results of the element to be extracted is performed by a data processing model; the data processing model is trained in the following manner:

[0064] Obtain the standard extraction results of the elements to be extracted in the training text;

[0065] Extract information from the training text based on the elements to be extracted, and obtain the predicted extraction results of the elements to be extracted;

[0066] The data processing model is trained based on the training text, the elements to be extracted, the predicted extraction results of the elements to be extracted, and the standard extraction results of the elements to be extracted.

[0067] It should be understood that in the supervised learning training process of the model, the label training sample correctly outputs the indication. In the training process of this application, the standard extraction result of the element to be extracted can be obtained first. The standard extraction result can be understood as a label result or a standard result, which is the actual result of the element to be extracted in the training text, and the training text can be extracted based on the element to be extracted to obtain the predicted extraction result of the element to be extracted. The data processing model is trained using the training text, the element to be extracted, the predicted extraction result of the element to be extracted, and the standard extraction result of the element to be extracted. The data processing model trained in this way is then used to select the second extraction result from the M first extraction results. For example, the M first extraction results can be processed by the data processing model to obtain the predicted label of each first extraction result (for example, it can be a first value (such as 1) or a second value (such as 0), the first value indicates correct / selected, and the second value indicates incorrect / not selected), and the first extraction result with the predicted label of the first value can be selected, and the first extraction result with the predicted label of the first value can be determined as the second extraction result, etc.

[0068] Exemplarily, the data processing model may be a voting model, that is, the second extraction result of the element to be extracted may be determined by voting among the M first extraction results of the element to be extracted through the voting model.

[0069] Exemplarily, the number of predicted extraction results can be M, that is, M predicted extraction results, which can be obtained by extraction through the information extraction model. The M predicted extraction results correspond one-to-one to the M reasoning paths of the information extraction model. The M predicted extraction results of the elements to be extracted can be the elements to be extracted, the prompt information of the elements to be extracted, and the training text input information extraction model to extract information, and obtain the M predicted extraction results of the elements to be extracted generated by the M reasoning paths of the information extraction model. The process is similar to the above-mentioned process of generating M first extraction results through the information extraction model, and will not be repeated.

[0070] In addition, it should be noted that in the process of determining the second extraction result of the element to be extracted from the M first extraction results of the element to be extracted in the present application, the second extraction result can also be selected by the minority obeys the majority method, that is, the extraction result with the largest number of the same number can be selected from the M first extraction results. For example, M is 5, and the M first extraction results are A1, A2, A2, A4, and A5, that is, the number of extraction results A1, A4, and A5 are all 1, and there are 2 extraction results A2, that is, these two extraction results are the same in the 5 first extraction results, then the extraction result A2 can be selected as the second extraction result. In addition, if there is more than one extraction result with the largest number of the same number, but at least two, then any one of the at least two different extraction results can be selected.

[0071] That is, in this embodiment, for each element to be extracted, the second extraction result of the element to be extracted can be selected from the M first extraction results of the element to be extracted through the data processing model, and the data processing model is a trained model, specifically a model trained by training text, elements to be extracted, predicted extraction results of the elements to be extracted, and standard extraction results of the elements to be extracted. Through the above-mentioned trained data processing model, the second extraction result is selected from the M first extraction results to improve the efficiency of determining the second extraction result of the element to be extracted.

[0072] The process of the information extraction method is described in detail below using a specific embodiment, and the information extraction method is described by taking the application of the information extraction method to a contract review scenario as an example.

[0073] This embodiment designs a new element extraction method based on the contract element extraction based on the large language model and the thought chain prompt words. First, it is necessary to construct templates for different element extractions according to the format of the Few-shot COT prompt according to different names of the elements to be extracted. When using the large language model for element extraction, sampling is performed in the large language model decoding stage to generate multiple (for example, M) reasoning paths to generate M extraction results, and the detection results of the M extraction results are counted as the uncertainty judgment standard of the element extraction results. For the element set with high uncertainty, the Few-shot COT prompt example of each element extraction is manually updated and optimized until the element becomes an element with low uncertainty. For the elements in the element set with low uncertainty, the Few-shot COT prompt example is not updated, and the element is extracted using it. Sampling is still performed in the large model decoding stage to generate multiple reasoning paths to generate M extraction results. For these M extraction results, a voting model is trained to select one result from the M extraction results as the final result of the element extraction, that is, the second extraction result.

[0074] like Figure 2 As shown, the process of the method of the embodiment of the present application is as follows:

[0075] First, for each element to be extracted, according to the different names of the elements to be extracted, different element extraction templates are constructed in the format of Few-shot COT. The element name, Few-shot COT prompt, and contract text are combined into prompt words (i.e., corresponding to the above input information), as follows: "You are now a contract review expert, and your task is to extract the following information from the text: {element name}.

[0076] Example template:

[0077] Example 1 Text: xxxxx

[0078] A: From the text, we know that {element name} is xxx. So -{element name}-: xxx;

[0079] Example 2 Text: xxxxx

[0080] A: From the text, we know that {element name} is xxx. So -{element name}-: xxx;

[0081] …

[0082] Example text: xxxxx

[0083] A: From the text, we know that {element name} is xxx. So -{element name}-: xxx;

[0084] ---

[0085] Please output the results in the following format:

[0086] -{Element name}-:{Element name};

[0087] If no relevant attributes can be extracted, use the unknown answer;

[0088] Text: {Contract text}".

[0089] Secondly, when using a large language model for feature extraction, sampling is performed in the decoding stage of the large language model (for example, the temperature parameter is set to a preset temperature, which is not limited, for example, the preset temperature can be 0.8, etc.) to generate multiple reasoning paths to generate M (not limited, for example, M is 10) extraction results.

[0090] For each element to be extracted, the test results of its M extraction results are counted as the uncertainty judgment standard of the element extraction result. The specific statistical method of inconsistent parameters is as follows: a) Count the number of results after M extraction results are removed, represented by N; b) The test result is equal to N / M. The higher the inconsistency, the higher the uncertainty.

[0091] When the detection result of the extraction result of a to-be-extracted element is less than a preset threshold value (for example, 0.3), the to-be-extracted element is an element with low uncertainty, otherwise it is an element with high uncertainty.

[0092] For factors with high uncertainty, the few-shot prompt examples extracted from each factor are manually optimized in a loop until the factor becomes a factor with low uncertainty.

[0093] For the elements in the element set with low uncertainty, the few-shot prompt and large language model used in the above evaluation are used to extract information from the elements. Sampling is still performed in the decoding stage of the large language model to generate multiple reasoning paths and generate M extraction results.

[0094] Based on a pre-trained model such as deberta-v3-large, a reasoning path voting model (voting model) is trained. The input is the contract text, the name of the element to be extracted, and the result of the reasoning path. If the result of the reasoning path is correct, the label is 1, otherwise the label is 0.

[0095] The inference path voting model is used to select one result from the M extraction results as the final result of the feature extraction, that is, the second extraction result.

[0096] The method of the embodiment of the present application evaluates the uncertainty of the extraction results of different elements, and optimizes the extraction of elements with high uncertainty, while no longer optimizing the extraction of elements with low uncertainty. As a result, more energy can be spent on extracting elements with high uncertainty, which can save development time. At the same time, multiple rounds of optimization can be performed for the extraction of elements with high uncertainty until their uncertainty is lower than the threshold and becomes an element with low uncertainty, thereby improving the accuracy of element extraction.

[0097] In addition, in the related art, the factor extraction adopts a greedy decoding strategy and generates the final result based on only one reasoning path, which will cause the large model to select the wrong reasoning path and lead to an erroneous final result. However, this application samples the large language model in the decoding stage to generate multiple reasoning paths, generate M extraction results, and then use the trained voting model to select one of the M extraction results as the final result of factor extraction, thereby improving the accuracy of factor extraction.

[0098] See also Figure 3 , Figure 3 3 is a structural diagram of the information extraction device 300 provided in the embodiment of the present application, which can implement the details of the information extraction method in the above embodiment and achieve the same effect. Figure 3 As shown, the information extraction device 300 includes:

[0099] An extraction module 301 is used to extract information from the text to be extracted based on the elements to be extracted, and obtain M first extraction results, where M is an integer greater than 1;

[0100] A determination module 302 is used to perform an inconsistency detection on the M first extraction results to obtain a detection result, where the detection result is used to reflect the inconsistency between the M first extraction results;

[0101] The determination module 302 is further used to determine a second extraction result of the element to be extracted according to the detection result.

[0102] In some embodiments, the determination module 302 includes:

[0103] A deduplication unit, used for deduplicating the M first extraction results to obtain a first extraction result after deduplication;

[0104] The parameter determination unit is used to determine the detection result of the element to be extracted according to the number of results of the first extraction result after deduplication and M.

[0105] In some embodiments, the extraction module is specifically configured to:

[0106] Based on the elements to be extracted and the prompt information of the elements to be extracted, information is extracted from the text to be extracted to obtain M first extraction results of the elements to be extracted.

[0107] In some embodiments, the extraction module is specifically configured to:

[0108] The elements to be extracted, the prompt information of the elements to be extracted and the text to be extracted are used as the input information of the information extraction model, and information extraction is performed based on the input information through M reasoning paths in the information extraction model to generate M first extraction results.

[0109] In some embodiments, the determination module 302 is specifically configured to:

[0110] When the detection result is less than a preset threshold, a second extraction result of the element to be extracted is determined from the M first extraction results of the element to be extracted.

[0111] In some embodiments, the determination module 302 is specifically configured to:

[0112] When the detection result of the element to be extracted is greater than or equal to the preset threshold, the following steps are executed repeatedly until the updated detection result of the element to be extracted is less than the preset threshold. When the updated detection result of the element to be extracted is less than the preset threshold, the second extraction result of the element to be extracted is determined from the updated M first extraction results of the element to be extracted:

[0113] Update the prompt information of the elements to be extracted;

[0114] Based on the elements to be extracted and the updated prompt information of the elements to be extracted, information is extracted from the text to be extracted to obtain the updated M first extraction results of the elements to be extracted;

[0115] An inconsistency detection is performed on the updated M first extraction results, and the detection result is updated. The updated detection result is used to characterize the inconsistency of the updated M first extraction results of the element to be extracted.

[0116] In some embodiments, determining the second extraction result of the element to be extracted from the M first extraction results of the element to be extracted is performed by a data processing model, and the data processing model is trained in the following manner:

[0117] Obtain the standard extraction results of the elements to be extracted in the training text;

[0118] Extract information from the training text based on the elements to be extracted, and obtain the predicted extraction results of the elements to be extracted;

[0119] The data processing model is trained based on the training text, the elements to be extracted, the predicted extraction results of the elements to be extracted, and the standard extraction results of the elements to be extracted.

[0120] The information extraction device provided in the embodiment of the present application can perform each process of the information extraction method in the above embodiment. To avoid repetition, it will not be described here.

[0121] like Figure 4 As shown, an embodiment of the present application also provides an electronic device, including a processor 410, a memory 420, and a computer program stored in the memory 420 and executable on the processor 410. When the computer program is executed by the processor 410, each process of the above-mentioned information extraction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0122] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned information extraction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0123] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the various processes of the above-mentioned information extraction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0124] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0126] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An information extraction method, characterized in that: include: Extract information from the text to be extracted based on the elements to be extracted, and obtain M first extraction results, where M is an integer greater than 1; Performing inconsistency detection on the M first extraction results to obtain a detection result, where the detection result is used to reflect the inconsistency between the M first extraction results; A second extraction result of the element to be extracted is determined according to the detection result.

2. The method according to claim 1, characterized in that The performing inconsistency detection on the M first extraction results to obtain the detection result includes: Deduplication is performed on the M first extraction results to obtain a first extraction result after deduplication; The detection result is obtained according to the number of results of the first extraction result after deduplication and M.

3. The method according to claim 1, characterized in that The information extraction of the to-be-extracted text is performed based on the elements to be extracted, and M first extraction results are obtained, including: Based on the elements to be extracted and the prompt information of the elements to be extracted, information is extracted from the text to be extracted to obtain M first extraction results of the elements to be extracted.

4. The method according to claim 3, characterized in that The extracting information from the text to be extracted based on the element to be extracted and the prompt information of the element to be extracted to obtain M first extraction results of the element to be extracted includes: The elements to be extracted, the prompt information of the elements to be extracted and the text to be extracted are used as input information of the information extraction model, and information extraction is performed based on the input information through M reasoning paths in the information extraction model to generate the M first extraction results.

5. The method according to claim 3, characterized in that: The step of determining a second extraction result of the element to be extracted according to the detection result includes: When the detection result is less than a preset threshold, a second extraction result of the element to be extracted is determined from the M first extraction results of the element to be extracted.

6. The method according to claim 5, characterized in that The step of determining the second extraction result of the element to be extracted from the M first extraction results of the element to be extracted is performed by a data processing model, and the data processing model is trained in the following manner: Obtaining standard extraction results of the elements to be extracted in the training text; Extracting information from the training text based on the elements to be extracted to obtain a predicted extraction result of the elements to be extracted; The data processing model is trained based on the training text, the elements to be extracted, the predicted extraction results of the elements to be extracted, and the standard extraction results of the elements to be extracted.

7. An information extraction device, characterized in that: include: An extraction module, used for extracting information from the text to be extracted based on the elements to be extracted, and obtaining M first extraction results, where M is an integer greater than 1; A determination module, configured to perform an inconsistency detection on the M first extraction results to obtain a detection result, wherein the detection result is used to reflect the inconsistency between the M first extraction results; The determination module is further used to determine a second extraction result of the element to be extracted according to the detection result.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the information extraction method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the information extraction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images