Information extraction method and system based on multi-stage confidence coefficient calibration and medium

Abstract

The invention discloses an information extraction method and system based on multi-stage confidence coefficient calibration and a medium, belongs to the technical field of natural language processing, and is used for solving the technical problems that an existing information extraction method is insufficient in professional understanding precision, extraction result confidence coefficients cannot be quantitatively evaluated, and output formats are not uniform. The method comprises the steps of calling corresponding key fields, a common expression set and a thinking chain set in an audit domain knowledge base based on a document type uploaded by a user, constructing a dynamic Prompt and inputting the dynamic Prompt into a large language model, and outputting a preliminary information extraction result by the large language model; inputting the key field and the preliminary information extraction result into a BERT model to calculate a cosine similarity matrix between word pairs, and obtaining a comprehensive index score of the preliminary information extraction result based on the cosine similarity matrix; and based on the comprehensive index score and a preset confidence threshold, filtering out low-confidence results in the preliminary information extraction results to obtain a final information extraction result, and performing normalization processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to an information extraction method and system based on multi-stage confidence calibration and a medium. BACKGROUND

[0002] In investment audit work, auditors need to extract key information from a large amount of unstructured audit documents and enter standardized databases, which is one of the core links of audit work. The current mainstream information extraction methods in the industry mainly fall into two categories: The first category is a rule and template-based information extraction method. This method relies on auditors to manually preset matching keywords and regular expressions for key fields, and extracts information through text matching. However, this method has significant drawbacks: first, it is extremely sensitive to changes in document format and language expression. When the key information in the document is presented in a non-preset expression or non-preset format, the extraction accuracy will decrease significantly, and the generalization ability is weak; second, the maintenance cost is high. As the scope of audit business expands and new types of documents are added, the keywords and regular expressions need to be updated manually, and the adaptation efficiency is low; third, it cannot handle complex scenarios such as implicit expressions, multiple co-reference, and numerical association calculations, and cannot complete the conversion and verification of Chinese capital amounts and Arabic numerals.

[0003] The second category is a traditional machine learning / deep learning model-based information extraction method, represented by named entity recognition models. This type of method needs to rely on large-scale high-quality labeled data to train entity recognition and relationship classification sub-models, and then complete information extraction through a pipeline architecture. However, in the audit scenario, this method has the following shortcomings: first, it is difficult and costly to obtain labeled data. Audit documents are highly professional and require personnel with audit expertise to complete the labeling. The complexity of document types makes it difficult to ensure the coverage of labeled data; second, the pipeline architecture has error propagation problems. If the entity recognition stage makes a mistake, the subsequent relationship classification will inevitably fail; third, the model has poor scalability after solidification. When new key fields or document types are added, the model needs to be retrained with labeled data, and the adaptation cycle is long; fourth, the model can only output extraction results and cannot provide quantitative indicators of the reliability of the results, making it difficult for auditors to identify high-risk extraction items.

[0004] In recent years, large language models have demonstrated advantages in general text understanding tasks, and some attempts have been made to directly apply them to information extraction from audit documents. However, core issues remain unresolved: First, general large language models lack expertise in the audit domain, resulting in insufficient understanding of audit terminology and common contract expressions, which can easily lead to mis- or missed extraction of key information. Second, the output of large language models lacks quantitative confidence assessment, making it impossible to distinguish the reliability of the extraction results, resulting in low system usability and auditability. Third, the output of large language models exhibits randomness, making it difficult to stably output standardized and structured data. This results in the extraction results being unable to be directly integrated into standardized audit databases, requiring manual secondary processing, which is inefficient.

[0005] In summary, existing technologies cannot meet the core requirements for extracting unstructured document information in investment audit scenarios. There is an urgent need for an information extraction solution that integrates domain knowledge, has confidence assessment capabilities, and provides standardized output formats. Summary of the Invention

[0006] This invention provides an information extraction method, system, and medium based on multi-stage confidence calibration to address the following technical problems: existing information extraction methods lack sufficient precision in professional understanding, the confidence of extraction results cannot be quantitatively evaluated, and the output format is inconsistent.

[0007] The embodiments of the present invention adopt the following technical solutions: On the one hand, embodiments of the present invention provide an information extraction method based on multi-stage confidence calibration, the method including: preprocessing unstructured audit documents uploaded by users to obtain plain text documents with uniform format; Based on the document type of the plain text document, the corresponding key fields, common expression sets and thought chain sets are retrieved from the audit domain knowledge base, a dynamic Prompt is constructed and input into the large language model, and the large language model outputs the preliminary information extraction results. The key fields and the preliminary information extraction results are input into the BERT model to calculate the cosine similarity matrix between word pairs, and the comprehensive index score of the preliminary information extraction results is obtained based on the cosine similarity matrix. Based on the comprehensive index score and the preset confidence threshold, low-confidence results in the preliminary information extraction results are filtered out to obtain the final information extraction results. The final information extraction results are normalized according to field type, and the structured information extraction data that meets the requirements of the audit standardized database is output.

[0008] In one feasible implementation, the unstructured audit documents uploaded by users are preprocessed to obtain plain text documents with a uniform format, specifically including: For unstructured audit documents uploaded by users, the format is parsed according to the document type; wherein, the unstructured audit documents include at least: Word documents, PDF documents, and plain text documents. The encoding detection library is called to identify the encoding type of the text fragments that contain garbled characters after parsing. If an encoding mismatch is detected, the encoding conversion is re-executed to restore the correct text. Complete the corrupted and missing characters in the correct text, and filter out invisible control characters and non-printable characters in the correct text, retaining only semantically meaningful text characters; The processed audit documents are uniformly converted into plain text documents of the same format.

[0009] In one feasible implementation, based on the document type of the plain text document, the corresponding key fields, common expression sets, and thought chain sets are retrieved from the audit domain knowledge base to construct a dynamic Prompt and input into a large language model. The large language model then outputs preliminary information extraction results, specifically including: Based on the type of the plain text document, the corresponding key fields, common expression sets, and thought chain sets are retrieved from the audit domain knowledge base using a document type matching algorithm. The key fields, common expression sets, and thought chain sets are all embedded in a preset template to generate a dynamic Prompt; wherein, the preset template includes at least: audit expert role setting, field identification requirements, coreference resolution requirements, numerical calculation requirements, and JSON output format requirements; The dynamic Prompt and the plain text document are input into the large language model, and the preliminary information extraction results are output; wherein, the preliminary information extraction results include at least: field values, original phrases, original text fragments, and inference processes.

[0010] In one feasible implementation, the key fields and the preliminary information extraction results are input into the BERT model to calculate the cosine similarity matrix between word pairs, specifically including: The key fields are used as reference sentences, and the preliminary information extraction results are used as comparison sentences. Each word in the reference sentence and the comparison sentence is converted into a vector representation to obtain the reference sentence vector and the comparison sentence vector; The reference sentence vector and the comparison sentence vector are input into the BERT model to calculate the cosine similarity matrix between all word pairs in the reference sentence and the comparison sentence.

[0011] In one feasible implementation, the comprehensive index score of the preliminary information extraction result is obtained based on the cosine similarity matrix, specifically including: The recall and precision of the reference sentence and the comparison sentence are calculated based on the cosine similarity matrix. Based on the recall and precision, calculate the comprehensive index score of the preliminary information extraction results corresponding to the current key field.

[0012] In one feasible implementation, based on the comprehensive index score and a preset confidence threshold, low-confidence results in the preliminary information extraction results are filtered out to obtain the final information extraction results, specifically including: The comprehensive index score of the current key field is compared with the preset confidence threshold. If the comprehensive index score is lower than the preset confidence threshold, the current key field is determined to be a low confidence field and its corresponding preliminary information extraction result is deleted; otherwise, it is retained. The preliminary information extraction results of all remaining key fields after filtering are summarized into the final information extraction results.

[0013] In one feasible implementation, the final information extraction result is normalized according to field type to output structured information extraction data that meets the requirements of the audit standardized database, specifically including: Create a configurable rule mapping table; the rule mapping table can be configured with format mapping rules corresponding to different field types; Based on the user-configured rule mapping table, the different field types in the final information extraction result are normalized and mapped to obtain the structured information extraction data.

[0014] In one feasible implementation, the different field types in the final information extraction result are normalized and mapped according to the user-configured rule mapping table, specifically including: The strings in the final information extraction result are converted into predefined numeric types according to the mapping rules in the rule mapping table; The non-standard time representations in the final information extraction results are unified into a predefined standard date format according to the mapping rules in the rule mapping table; Texts with the same meaning but different expressions are mapped to predefined standardized codes according to the mapping rules in the rule mapping table.

[0015] On the other hand, embodiments of the present invention also provide an information extraction system based on multi-stage confidence calibration, the system comprising: The preliminary extraction module is used to preprocess the unstructured audit documents uploaded by users to obtain plain text documents with uniform format; based on the document type of the plain text documents, the corresponding key fields, common expression sets and thought chain sets are retrieved from the audit domain knowledge base, a dynamic Prompt is constructed and input into the large language model, and the large language model outputs the preliminary information extraction results; The confidence calibration module is used to input the key fields and the preliminary information extraction results into the BERT model to calculate the cosine similarity matrix between word pairs, and obtain the comprehensive index score of the preliminary information extraction results based on the cosine similarity matrix; based on the comprehensive index score and the preset confidence threshold, the low confidence results in the preliminary information extraction results are filtered out to obtain the final information extraction results; The rule constraint module is used to perform normalization processing on the final information extraction results according to the field type, and output structured information extraction data that meets the requirements of the audit standardized database.

[0016] Finally, this embodiment of the invention also provides a storage medium, which is a non-volatile computer-readable storage medium storing at least one program, each program including instructions, which, when executed by a terminal, cause the terminal to perform the information extraction method based on multi-stage confidence calibration.

[0017] Compared with the prior art, the information extraction method, system and medium based on multi-stage confidence calibration provided by the embodiments of the present invention have the following beneficial effects: This invention achieves high-precision and high-reliability information extraction from unstructured audit documents by integrating domain knowledge enhancement with a large language model through a multi-stage hybrid prompting strategy. Compared with existing technologies, it has the following significant advantages: 1. This invention introduces a knowledge base in the audit domain through a dynamic CoT semantic understanding stage. Combined with a dynamically generated thought chain reasoning framework, it can handle complex scenarios such as implicit expressions, multi-party co-references, and numerical correlation calculations. This solves the problems of weak generalization ability of traditional rule-based methods and the inability of traditional models to handle complex semantics. At the same time, the confidence calibration is completed based on the BERT model fine-tuned in the audit domain, effectively filtering out the misjudgment results of the large language model. Compared with directly using the general large language model, the accuracy of key information extraction is greatly improved.

[0018] 2. This invention uses BERTScore to quantitatively evaluate the confidence of the extraction results, outputting the recall, precision, and overall confidence score for each field. Auditors can accurately identify low-confidence extraction items and conduct manual review, solving the problem that existing technologies cannot assess the reliability of extraction results, improving the auditability of audit work, and reducing audit risks.

[0019] 3. This invention uses a rule engine in the structured constraint stage to transform the raw information output by the large language model into standardized data that meets the requirements of audit business. The output JSON data can be directly connected to the audit standardization database and downstream analysis system, which solves the integration problem caused by the randomness of the output of the large language model, eliminates the need for manual secondary processing, and improves the efficiency of audit data entry.

[0020] 4. The domain knowledge base of this invention supports flexible configuration by document type and key fields. When adding a new audit document type or key field, only the relevant configuration needs to be added to the knowledge base, without modifying the model or rule code. At the same time, the dynamic Prompt generation logic can automatically adapt to the new configuration, which solves the problems of high maintenance cost and poor scalability of traditional rule methods, and significantly reduces the system iteration cost.

[0021] 5. This invention incorporates auditing knowledge throughout the entire process. From Prompt construction and semantic understanding to confidence calibration, it optimizes auditing terminology and common contract expressions, solving the problem of insufficient accuracy in professional understanding of general large language models. It adapts to the professional needs of investment auditing scenarios and improves the level of intelligence in auditing work. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart of an information extraction method based on multi-stage confidence calibration is provided for an embodiment of the present invention; Figure 2 This is a schematic diagram of an information extraction system based on multi-stage confidence calibration, provided as an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0024] This invention provides an information extraction method based on multi-stage confidence calibration, such as...Figure 1 As shown, the information extraction method based on multi-stage confidence calibration specifically includes steps S101-S105: S101. Preprocess the unstructured audit documents uploaded by users to obtain plain text documents with uniform format.

[0025] Specifically, the multimodal data recognition and reconstruction module parses unstructured audit documents uploaded by users according to their document type and format; among them, unstructured audit documents include at least: Word documents, PDF documents, and plain text documents.

[0026] Furthermore, an encoding detection library is invoked to identify the encoding type of the parsed text fragments that contain garbled characters. If an encoding mismatch is detected, the encoding conversion is re-executed to restore the correct text. Damaged and missing characters in the correct text are completed, and invisible control characters and non-printable characters are filtered out, retaining only semantically meaningful text characters.

[0027] Finally, the processed audit documents are all converted into plain text documents in the same format.

[0028] As a feasible implementation method, a format-adaptive parsing logic is adopted for uploaded unstructured audit documents in different formats such as Word, PDF, and plain text: PDF documents: The underlying data of the document is parsed using PyMuPDF (fitz) or Adobe PDF SDK, and the original text of the text layer is extracted first. For scanned PDFs (without a text layer), the optical character recognition engine based on PaddleOCR / Alibaba Cloud OCR is called to complete the conversion from image to text, and at the same time recognize the page numbers, headers and footers and other layout marks of the document.

[0029] Word documents: Based on python-docx or Apache POI parsing components, extract the core content of the document, such as the body text, comments, and tables, and filter out style tags (such as font, color, indentation, and other non-semantic tags) in docx / xml format, retaining only the text content and structural information.

[0030] Plain text documents: Read directly through file stream, recognize text encoding formats (such as UTF-8, GBK, GB2312), and complete basic encoding adaptation reading.

[0031] S102. Based on the plain text document document type, retrieve the corresponding key fields, common expression sets and thought chain sets from the audit domain knowledge base, construct a dynamic Prompt and input it into the large language model, and output the preliminary information extraction results from the large language model.

[0032] Specifically, based on the type of plain text documents, the corresponding key fields, common expression sets, and thought chain sets are retrieved from the audit domain knowledge base through a document type matching algorithm.

[0033] Furthermore, key fields, common expression sets, and thought chain sets are all embedded in a preset template to generate a dynamic Prompt; the preset template includes at least: audit expert role settings, field identification requirements, coreference resolution requirements, numerical calculation requirements, and JSON output format requirements.

[0034] Finally, the dynamic Prompt and plain text document are input into the large language model to output preliminary information extraction results; the preliminary information extraction results include at least: field values, original phrases, original text fragments, and inference process.

[0035] As a feasible implementation method, traditional rule-based or keyword matching methods struggle to handle the following complex situations: Implicit statements often conceal key information within complex sentence structures or technical terms, such as "The price of this contract is inclusive of tax, totaling RMB 100 million." Multiple referencing: The same entity may appear in multiple forms such as full name, abbreviation, and pronoun, such as "XX Group Co., Ltd." (full name), "XX Group" (abbreviation), "Party A" (role), and "the company" (pronoun) all referring to the same entity; Numerical correlation and calculation: Contract amounts may involve capitalization conversion and addition / subtraction of amounts, requiring the model to perform correlation and preliminary calculations. Given the aforementioned complexities, this invention designs a dynamic reasoning framework based on a dynamic thought chain to guide the model in biomimetic reasoning. This framework not only includes fixed instructions but also integrates audit rules and thinking patterns dynamically loaded from a domain knowledge base.

[0036] The reasoning framework of dynamic thought chain specifically includes: Definitions: Input text T, document domain DocType, domain knowledge base K, key field list F, common expression set S, and thought chain set CoT; Model Prompt initialization: Based on DocType, load the corresponding key field list F, common expression set S, and related domain thought chain set CoT from K to form the model Prompt.

[0037] For example: For a certain contract, DocType is "Contract Information", loading F={Contract Name, Contract Amount, Guarantee Deposit Ratio, ...} and SContract Amount={Contract Price, Total Price, Investment Amount, ...}, CoT 合同金额 =F 合同金额 ×F 质保金比例Together, we construct a Prompt containing the thought-reasoning logic of the core understanding requirements, and call upon a large language model to achieve semantic understanding in text T.

[0038] S103. Input the key fields and preliminary information extraction results into the BERT model to calculate the cosine similarity matrix between word pairs, and obtain the comprehensive index score of the preliminary information extraction results based on the cosine similarity matrix.

[0039] Specifically, key fields are used as reference sentences, and preliminary information extraction results are used as comparison sentences. Each word in the reference and comparison sentences is converted into a vector representation, resulting in reference sentence vectors and comparison sentence vectors.

[0040] Furthermore, the reference sentence vector and the comparison sentence vector are input into the BERT model to calculate the cosine similarity matrix between all word pairs in the reference sentence and the comparison sentence.

[0041] Finally, the recall and precision of the reference and comparison sentences are calculated based on the cosine similarity matrix; based on the recall and precision, the comprehensive index score of the preliminary information extraction results corresponding to the current key field is calculated.

[0042] As a feasible implementation method, this invention uses BERTScore embedding cosine similarity to evaluate the initial information extraction results. Further validation of key fields is then performed. and text The similarity of the original fields in the original data.

[0043] First, obtain the embedding vector: Reference sentence: (i.e., list of key fields) Key fields defined in the code, such as the project's unified code, etc. Contrast sentences: (Model in text) The fields found correspond to the original fields and phrases, i.e., the fields in the preliminary information extraction results. Furthermore, the context embedding vector for each word is obtained using the BERT model: ,in It is the vector of the i-th word in the reference sentence. ,in It is the vector of the j-th word in the comparison sentence. Furthermore, calculate the cosine similarity matrix: calculate the cosine similarity between all word pairs in the two sentences to form a matrix. Similarity matrix Each element in the matrix The calculation formula is the standard cosine similarity formula: This matrix It reflects the degree of similarity between words in the semantic space.

[0044] Finally, the final score is calculated based on the similarity matrix: Recall rate: Recall rate measures the recall rate of the reference sentence. How much information is included in the candidate sentences? Capture. That is, capture every word in the reference sentence. Find the word that is most similar to the candidate sentence (i.e., the word with the highest similarity) among all the words in the candidate sentence, and take the average: .

[0045] Precision: Precision rate measures the accuracy of candidate sentences. How much information is related to the reference sentence? Relevant and correct. For each word in the candidate sentence. Find the word that is most similar to the reference sentence (i.e., the word with the highest similarity) among all the words in the reference sentence, and then take the average: .

[0046] Composite metric: The harmonic mean of precision and recall, used as a comprehensive indicator. .

[0047] S104. Based on the comprehensive index score and the preset confidence threshold, filter out the low confidence results in the preliminary information extraction results to obtain the final information extraction results.

[0048] Specifically, the comprehensive index score of the current key field is compared with the preset confidence threshold. If the comprehensive index score is lower than the preset confidence threshold, the current key field is determined to be a low-confidence field, and the corresponding preliminary information extraction result is deleted; otherwise, it is retained.

[0049] Then, the preliminary information extraction results of all remaining key fields after filtering are summarized into the final information extraction results.

[0050] As a feasible implementation method, the system presets a confidence threshold (configurable, default value is 0.5). If the comprehensive index F BERT If the result is below this threshold, the field is considered a low-confidence extraction result and is filtered out. This calibration stage effectively identifies and eliminates false positives from large models, significantly improving the reliability of the final output.

[0051] S105. Perform normalization processing on the final information extraction results according to the field type, and output structured information extraction data that meets the requirements of the audit standardized database.

[0052] Specifically, a configurable rule mapping table is created; the rule mapping table can be configured with format mapping rules corresponding to different field types.

[0053] Furthermore, based on the user-configured rule mapping table, the different field types in the final information extraction results are normalized and mapped to obtain structured information extraction data.

[0054] As a feasible implementation method, strong-rule post-processing and normalization are performed on the JSON data calibrated with confidence levels. Its core objective is to address the uncertainty in data type and format of large language model outputs, transforming the raw information, primarily in string format, into standardized values ​​that conform to system data protocols and business requirements, ensuring the direct usability of the data. Specific processing includes, but is not limited to, the following three categories: Numeric field quantization: Converting the string output by the model into a specific numeric type; Time field standardization: unifying various non-standard time expressions into standard date formats such as ISO 8601; Enumerated field normalization: Maps texts with the same meaning but different expressions to predefined standardized codes.

[0055] This stage ensures that the final output JSON data meets the strict requirements for system storage and business analysis in terms of data structure, data type, and data format through a series of configurable rules and mapping tables. It solves the integration problem caused by the randomness of large model output at the end of the process and completes the final step of transformation from "readable text" to "computable standardized data".

[0056] Traditional information extraction models typically divide the task into two sub-tasks: entity recognition and relation classification, and train entity models and relation models separately. However, this type of method has the following inherent drawbacks: Error propagation: If an error occurs during the entity recognition stage, subsequent relationship classification will not be able to be performed correctly.

[0057] Task fragmentation: Entity recognition and relationship classification should be closely related, but training them separately prevents the models from sharing knowledge and contextual information.

[0058] Ignoring interactions between entities: When determining the relationship between entities A and B, the existence of entity C is not considered at the same time, which may result in the loss of important contextual information.

[0059] To this end, this invention abandons the traditional multi-model pipeline architecture and adopts a large language model as a unified, end-to-end semantic understanding and information extraction engine. Through carefully designed dynamic prompts, the large language model can simultaneously complete deep semantic understanding of the text, identification and extraction of all key fields, and judgment of logical relationships between fields in a coherent reasoning process, thereby overcoming the limitations of traditional methods and achieving a leap in the accuracy and efficiency of information extraction.

[0060] In addition, embodiments of the present invention also provide an information extraction system based on multi-stage confidence calibration, such as... Figure 2 As shown, the information extraction system 200 based on multi-stage confidence calibration specifically includes: The preliminary extraction module 210 is used to preprocess the unstructured audit documents uploaded by users to obtain plain text documents with uniform format; based on the document type of the plain text documents, the corresponding key fields, common expression sets and thought chain sets are retrieved from the audit domain knowledge base, a dynamic Prompt is constructed and input into the large language model, and the large language model outputs the preliminary information extraction results; The confidence calibration module 220 is used to input the key fields and the preliminary information extraction results into the BERT model to calculate the cosine similarity matrix between word pairs, and obtain the comprehensive index score of the preliminary information extraction results based on the cosine similarity matrix; based on the comprehensive index score and the preset confidence threshold, the low confidence results in the preliminary information extraction results are filtered out to obtain the final information extraction results; The rule constraint module 230 is used to perform normalization processing on the final information extraction result according to the field type, and output structured information extraction data that meets the requirements of the audit standardized database.

[0061] Finally, this embodiment of the invention also provides a storage medium, which is a non-volatile computer-readable storage medium storing at least one program, each program including instructions, which, when executed by a terminal, cause the terminal to perform: The unstructured audit documents uploaded by users are preprocessed to obtain plain text documents with a uniform format; Based on the document type of the plain text document, the corresponding key fields, common expression sets and thought chain sets are retrieved from the audit domain knowledge base, a dynamic Prompt is constructed and input into the large language model, and the large language model outputs the preliminary information extraction results. The key fields and the preliminary information extraction results are input into the BERT model to calculate the cosine similarity matrix between word pairs, and the comprehensive index score of the preliminary information extraction results is obtained based on the cosine similarity matrix. Based on the comprehensive index score and the preset confidence threshold, low-confidence results in the preliminary information extraction results are filtered out to obtain the final information extraction results. The final information extraction results are normalized according to field type, and the structured information extraction data that meets the requirements of the audit standardized database is output.

[0062] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0063] The foregoing has described specific embodiments of the present invention. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0064] The above description is merely an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for information extraction based on multi-stage confidence calibration, characterized in that, The method comprises: Pretreatment of the user uploaded unstructured audit document to obtain a pure text document with uniform format; Based on the document type of the pure text document, the corresponding key field, common expression set and thinking chain set are retrieved from the audit field knowledge base to construct a dynamic Prompt and input to a large language model, and the large language model outputs a preliminary information extraction result; The key field and the preliminary information extraction result are input into a BERT model to calculate the cosine similarity matrix between word pairs, and the comprehensive index score of the preliminary information extraction result is obtained based on the cosine similarity matrix; Based on the comprehensive index score and the preset confidence threshold, low-confidence results in the preliminary information extraction result are filtered out to obtain a final information extraction result; The final information extraction result is normalized according to the field type to output structured information extraction data meeting the requirements of the audit standardized database.

2. The method of claim 1, wherein, Pretreatment of the user uploaded unstructured audit document to obtain a pure text document with uniform format, specifically including: For the user uploaded unstructured audit document, the document type is parsed according to the format; wherein the unstructured audit document at least includes: Word document, PDF document, pure text document: Call the encoding detection library to identify the encoding type of the text segment with garbled characters after parsing, and if the encoding does not match, re-execute the encoding conversion to restore the correct text; Complete the damaged characters and incomplete characters in the correct text, and filter out the invisible control characters and non-printing characters in the correct text, only keeping the text characters that can be semantized; Convert the processed audit document into a pure text document with the same format.

3. The method of claim 1, wherein, Based on the document type of the pure text document, the corresponding key field, common expression set and thinking chain set are retrieved from the audit field knowledge base to construct a dynamic Prompt and input to a large language model, and the large language model outputs a preliminary information extraction result, specifically including: Based on the type of the pure text document, the corresponding key field, common expression set and thinking chain set are retrieved from the audit field knowledge base through a document type matching algorithm; Embed the key field, common expression set and thinking chain set in a preset template to generate a dynamic Prompt; wherein the preset template at least includes: audit expert role setting, field identification requirement, co-reference resolution requirement, numerical calculation requirement, JSON output format requirement; Input the dynamic Prompt and the pure text document into a large language model to output the preliminary information extraction result; wherein the preliminary information extraction result at least includes: field value, original phrase, original text segment, reasoning process.

4. The method of claim 1, wherein, The key field and the preliminary information extraction result are input into a BERT model to calculate the cosine similarity matrix between word pairs, specifically including: The key field is taken as the reference sentence, and the preliminary information extraction result is taken as the comparison sentence; Convert each word in the reference sentence and the comparison sentence into a vector representation to obtain a reference sentence vector and a comparison sentence vector; inputting the reference sentence vector and the comparison sentence vector into a BERT model, and calculating a cosine similarity matrix between all word pairs in the reference sentence and the comparison sentence.

5. The method of claim 1, wherein, Based on the cosine similarity matrix, a comprehensive index score of the preliminary information extraction result is obtained, specifically including: Based on the cosine similarity matrix, the recall rate and the precision rate of the reference sentence and the comparison sentence are calculated. According to the recall rate and the precision rate, the comprehensive index score of the preliminary information extraction result corresponding to the current key field is calculated.

6. The method of claim 1, wherein, Based on the comprehensive index score and the preset confidence threshold, low-confidence results in the preliminary information extraction result are filtered out to obtain the final information extraction result, specifically including: Compare the comprehensive index score of the current key field with the preset confidence threshold. If the comprehensive index score is lower than the preset confidence threshold, it is determined that the current key field is a low-confidence field, and the corresponding preliminary information extraction result is deleted. Otherwise, it is retained. The preliminary information extraction results of all key fields remaining after filtering are summarized as the final information extraction result.

7. The method of claim 1, wherein, According to the field type, the final information extraction result is normalized to output structured information extraction data meeting the requirements of the audit standardized database, specifically including: Creating a configurable rule mapping table; the rule mapping table can configure the format mapping rules corresponding to different field types; According to the rule mapping table configured by the user, different field types in the final information extraction result are normalized to obtain the structured information extraction data.

8. The method of claim 7, wherein, According to the rule mapping table configured by the user, different field types in the final information extraction result are normalized, specifically including: Convert the string in the final information extraction result to a predefined numerical type according to the mapping rule in the rule mapping table; The non-standard time expression in the final information extraction result is uniformly converted to a predefined standard date format according to the mapping rule in the rule mapping table; Texts with the same meaning but different expressions are mapped to predefined standardized codes according to the mapping rule in the rule mapping table.

9. A multi-stage confidence calibration based information extraction system, characterized in that, The system includes: A preliminary extraction module for preprocessing the unstructured audit document uploaded by the user to obtain a pure text document with uniform format; based on the document type of the pure text document, the corresponding key field, common expression set and thought chain set are retrieved from the audit domain knowledge base, a dynamic Prompt is constructed and input into a large language model, and the large language model outputs a preliminary information extraction result; A confidence calibration module for inputting the key field and the preliminary information extraction result into a BERT model to calculate a cosine similarity matrix between word pairs, and obtaining a comprehensive index score of the preliminary information extraction result based on the cosine similarity matrix; based on the comprehensive index score and the preset confidence threshold, low-confidence results in the preliminary information extraction result are filtered out to obtain the final information extraction result; A rule constraint module for normalizing the final information extraction result according to the field type to output structured information extraction data meeting the requirements of the audit standardized database.

10. A storage medium, characterized by The storage medium is a nonvolatile computer readable storage medium, and the nonvolatile computer readable storage medium stores at least one program, and each program includes instructions, which, when executed by a terminal, causes the terminal to execute the information extraction method based on multi-stage confidence calibration according to any one of claims 1-8.

Images