Information extraction system and method based on layout analysis and multiple modes

Abstract

The invention belongs to the technical field of information extraction, and particularly relates to an information extraction system and method based on layout analysis and multiple modes. The system comprises a document analysis module used for extracting key information from documents in various formats; the query analysis module is used for deeply analyzing and understanding a query request of a user; the retrieval enhancement module is used for retrieving information most relevant to a user query request from a knowledge base by combining a large language model LLM and an information retrieval system, and integrating the information into a cue word template; the rearrangement and filtering module is used for rearranging and filtering the cue word template output by the retrieval enhancement module to obtain a final content fragment; and the document extraction module is used for acquiring a final content segment, formatting the final content segment and a query request of a user, inputting the final content segment and the query request into a large language model (LLM), and finally obtaining an answer result. The method has the characteristics that the generalization ability is higher, the use cost is lower, and the accuracy and efficiency of information extraction can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of information extraction, and in particular relates to a system and method for information extraction based on layout analysis and multimodality. Background Art

[0002] With the rapid development of the Internet, the information contained in the network is growing exponentially. How to effectively extract the required key information from massive data has become the focus of current research. Information extraction mainly extracts structured, semi-structured, and unstructured data from unstructured text data. The general process of information extraction is to use the optical character recognition (OCR) detection algorithm to output the text line content, and then use the text recognition model to convert the text line content recognition into a sequence problem, identify the text line content and save it in the database, and then use post-processing methods such as regular matching or anchor point positioning to structure and process the data so that subsequent functions can be more conveniently queried and used.

[0003] Traditional information extraction methods usually rely on manually set rules and templates, but these methods have difficulty handling complex and changeable information expressions and cannot adapt to information extraction needs in different scenarios.

[0004] Specifically, traditional information extraction methods have the following problems: 1. Regular matching is not very applicable: In mainstream information extraction systems, regular matching is widely used because of its simplicity and speed. However, in the face of complex and changing actual demand scenarios, the limitations of regular matching are becoming more and more obvious. Regular expressions rely on preset patterns and are difficult to adapt to changes in various document layouts, font sizes, colors and other content. In addition, for documents with complex structures and dynamic changes (such as merged cells in tables, character content without clear boundaries in invoices or bills), regular expressions often require post-data cleaning and correction, and specific matching rules need to be set for specific content, which reduces the generalization of the overall extraction system.

[0005] 2. Post-processing method has a large workload: Existing information extraction systems rely on complex post-processing steps to correct and improve the extraction results. These post-processing results mainly include position information conversion, regular matching, anchor matching, content structure conversion, etc. This requires a large number of manually defined rules to process. However, with the sharp increase in the number of documents that need to be processed, this process becomes time-consuming and labor-intensive, and is prone to new problems. The heavy post-processing process not only requires a lot of manual intervention, but also reduces the operating efficiency of the system.

[0006] 3. Bert-type models have difficulty solving the problem of cells crossing rows, and their accuracy is limited and the labeling cost is high: Natural language processing (NLP) technology based on deep learning, especially models such as Bert and its variants, has made significant progress in information extraction. However, these models still face challenges when processing documents with complex table structures, especially when there are cells across rows or columns in the table, Bert-type models still cannot accurately understand and recognize these special structures. In addition, since model training requires a large amount of labeled data, and complex cell standardization work is time-consuming and expensive, this further limits the promotion of these models in practical applications.

[0007] 4. Only single-mode features: Most current information extraction systems rely only on information in the textual modality, ignoring the rich information in other modalities such as images and layouts. However, in practical applications, documents often contain information in multiple modalities, which complement each other and together constitute a complete representation of the document content. For example, in financial statements, text information such as numbers and symbols is closely related to visual information such as table layout and color, and together they convey important financial information. Using only a single modality for information extraction not only limits the system's comprehensive understanding and accurate extraction of document content, but may also lead to missed and false detections of information.

[0008] In recent years, the development of deep learning technology has brought new solutions to information extraction tasks. In particular, the development of large language models (LLM) has achieved remarkable results in the field of natural language processing. Today, users can use large model technology to extract information and perform structured processing through extraction instructions. Compared with traditional methods, extraction using large model technology can reduce the problem of low generalization of regular matching in traditional methods, and can also significantly reduce post-processing work such as anchor processing.

[0009] Therefore, it is very important to design an information extraction system and method based on layout analysis and multimodality that has stronger generalization ability, lower usage cost, and can improve the accuracy and efficiency of information extraction. Summary of the invention

[0010] The present invention aims to overcome the problems in the prior art that traditional information extraction methods have difficulty in handling complex and changeable information expressions and cannot adapt to information extraction needs in different scenarios. It provides an information extraction system and method based on layout analysis and multimodality, which has stronger generalization ability, lower usage cost, and can improve the accuracy and efficiency of information extraction.

[0011] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions: Information extraction system based on layout analysis and multimodality, including: Document parsing module, used to extract key information from documents in various formats; Query analysis module, used to deeply analyze and understand user query requests; A retrieval enhancement module, which is used to retrieve the information most relevant to the user's query request from a pre-customized knowledge base by combining the large language model LLM with the information retrieval system, and integrate the information into a prompt word template; A rearrangement and filtering module is used to reorder and filter the prompt word template output by the retrieval enhancement module to obtain the final content fragment; The document extraction module is used to obtain the final content fragment, format it together with the user's query request, and input it into the large language model LLM to finally obtain the required answer result.

[0012] Preferably, the document parsing module identifies the layout structure content of the document through multimodal OCR and layout recognition technology, and converts the layout structure content into structured data; the layout structure content includes titles, paragraphs, tables and images.

[0013] Preferably, the document parsing module includes a document slicing module for segmenting a document into multiple text blocks.

[0014] Preferably, the query analysis module specifically includes the following process: Extract important keywords from the user's query request, and use the keywords to retrieve relevant information in the document library; Identify entities in the query request, including names of people, places, and organizations; Analyze the user's query request intent to determine whether the user intends to seek information, answer questions, or perform other tasks; According to the query analysis results, the original query request is appropriately rewritten so that the rewritten query request is more consistent with the user's intention.

[0015] Preferably, the search enhancement module specifically includes the following process: The vector expressions of the query request and the text block are taken as input, and the correlation between the query request and the text block is evaluated by calculating the similarity score between the query request vector and the text block vector; at the same time, a similarity threshold is set, and the text blocks whose corresponding similarity scores are higher than the similarity threshold are retained.

[0016] Preferably, the rearrangement filtering module specifically includes the following process: The text blocks retained in the prompt word template are scored in multiple dimensions; the multiple dimensions include the semantic similarity between the text block and the query request, the importance of the text block and the relevance to the query request, and the scores of the multiple dimensions are weighted and summed to obtain a final scoring result; Based on the final scoring results, several text blocks with the strongest relevance are retained and sorted.

[0017] The present invention also provides a method for extracting information based on layout analysis and multimodality, comprising the following steps: S1, the user uploads the document to be processed through the system interface, the system performs content analysis, layout analysis, and document segmentation, embeds the segmented text blocks into words, and outputs the vector expression of the text blocks; S2: The user inputs the content to be extracted, designs a prompt, and uses a large language model to analyze the keywords, entities, intent, and rewritten queries, and also converts them into vector expressions. S3, the retrieval enhancement module receives the results of document parsing and screens and filters them according to the query analysis module, and filters out the most relevant text blocks by calculating the relevance between the question and the text block; S4, the re-ranking and filtering module conducts an in-depth relevance evaluation on the data output by the retrieval enhancement module, combines the semantic similarity, the importance of the text block, and the relevance to the query question, retains the text blocks with the strongest relevance and sorts them, and further reduces the number of text blocks; S5, in the document extraction stage, the results of the rearrangement filtering module and the user's query questions are sent to the large language model; the extraction capability of the large language model is used to obtain the extraction results expected by the user.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention can parse data from different sources and different formats, greatly improving the compatibility of data and the flexibility of processing; this feature enables the present invention to perform well in processing complex and diverse data sets, and provides a solid foundation for subsequent data analysis and application; (2) Through layout analysis technology, the present invention can automatically identify and parse the layout of documents or images, such as text blocks, pictures, tables, etc. This greatly simplifies the document processing process and improves the accuracy and efficiency of information extraction, especially when processing a large number of documents, it can significantly save manpower and time costs; (3) By using advanced large model technology, the present invention has achieved significant improvements in information extraction; the large model can understand the text content more deeply, so as to extract key information more accurately; this not only improves the accuracy of information extraction, but also enables the present invention to adapt to more diverse information extraction tasks; (4) Traditional methods often rely on preset templates to extract information, but the present invention abandons this concept and realizes template-free information extraction; this means that the present invention is more flexible and can adapt to various changes, without the need to create a new template for each new document or data source, which greatly reduces the cost of use and improves the scalability and maintainability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A principle block diagram of the information extraction system based on layout analysis and multimodality in the present invention; Figure 2 A functional flow chart of the query analysis module in the present invention; Figure 3 A flow chart of the practical application of the layout analysis and multimodal information extraction system provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to more clearly illustrate the embodiments of the present invention, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other accompanying drawings and other implementation methods can be obtained based on these accompanying drawings without creative work.

[0021] like Figure 1 As shown, the present invention provides an information extraction system based on layout analysis and multimodality, including: a document parsing module, a query analysis module, a retrieval enhancement module, a rearrangement filtering module, and a document extraction module.

[0022] Document parsing module: As one of its core components, this module is mainly responsible for extracting key information from documents of various complex formats. This module can support multiple file types including Word documents, PPT, Excel tables, PDF, pictures, etc. Through multimodal OCR and layout recognition technology, it can identify the layout structure of the document, such as titles, paragraphs, tables, images, etc. And convert it into structured data to provide support for subsequent questions and answers and citations. Among them, the document may also have the problem of being too long. The document parsing module also includes a document slicing function (document slicing module), which can split longer documents into multiple text blocks, providing assistance for more efficient subsequent processing and retrieval of information.

[0023] Query analysis module: responsible for in-depth analysis and understanding of user query requests, providing precise guidance for subsequent retrieval and generation processes. Its main function is to conduct in-depth analysis of user queries and extract key information, including keywords, entities, intents, etc., which will serve as input for subsequent retrieval and generation processes. Implementation details are as follows Figure 2 Specifically, the query analyzer needs to complete the following tasks: 1. Keyword extraction: Extract important keywords from user queries. These keywords will be used to retrieve relevant information in the document library.

[0024] 2. Entity recognition: Identify entities in the query, such as names of people, places, and organizations. This entity information helps to locate relevant documents more accurately.

[0025] 3. Intent understanding: Analyze the user's query intent to determine whether the user is seeking information, answering questions, or performing other tasks, which helps generate answers that better meet user needs.

[0026] 4. Query rewriting: Based on the results of query analysis, make necessary rewrites to the original query to improve the accuracy and efficiency of retrieval.

[0027] The above functions are implemented in combination with the Large Language Model (LLM) to achieve the corresponding goals by designing different prompts.

[0028] Taking query rewriting as an example, prompt can be written in the following form.

[0029] system_prompt = """ You are a professional query decomposition expert.\ Task: Given a user problem, decompose it into multiple independent sub-problems. These sub-questions need to be answered to solve the original question.

[0030] Note: If there are abbreviations or words that you are not familiar with, do not try to rephrase them.

[0031] """ By designing a suitable prompt, we can use the powerful capabilities of LLM to implement query rewriting. This approach is not only flexible, but also can generate more accurate and efficient queries based on the actual needs of users, thereby improving the effect of information retrieval.

[0032] Retrieval enhancement module: By combining the large language model (LLM) with an efficient information retrieval system, the most relevant information to the user's query is retrieved from the customized knowledge base, and this information is integrated into a prompt word template and sent to the document extraction module, thereby enhancing the accuracy and relevance of the system when extracting information. The retrieval enhancement module uses the vector index established by the text block and the user's query vector, and retrieves the text block in the index that is closest to the query vector semantics. Specifically, the vector expression (embendding) of the query text is recorded as , the text block is As input, by calculating the cosine similarity between them To evaluate the semantic relevance between them, the specific calculation formula is as follows: ; A similarity threshold is set to 0.5, and only text blocks with similarity scores higher than this threshold will be retained.

[0033] Reordering and filtering module: In this system, the retrieval enhancement module usually returns a large amount of context related to the user, but these context relevances are not exactly the same. The reordering and filtering module is responsible for refining the retrieved context information to improve the accuracy and relevance of the content. This module reorders and filters the retrieved documents or contexts to improve the accuracy and relevance of the generated content. The reordering and filtering module considers the semantic similarity between the text block and the query. , The importance of text blocks and relevance to the query subject The final score is sorted to optimize the context information and ensure that the prompt templates finally input into the large language model are of high quality and high relevance.

[0034] Specifically, Jaccard Similarity is used as a measure of the importance of a text block. The Jaccard similarity is used to calculate the ratio of the intersection and union of two sets. In text processing, a text can be regarded as a set of words.

[0035] For query text q and text block c, Jaccard similarity Defined as: ; in: is the size of the intersection of the query text q and the text block c. is the size of the union of the query text q and the text block c.

[0036] Use mutual information as an indicator of the relevance of the query topic , defined as the relative entropy of the product of the joint distribution of two random variables and their respective marginal distributions. For the query text q and the known text block c, the mutual information can be expressed as: ; in: is a term in the query text q. is a term in the known text block c. It is a term The term appears in the query text q and The probability of appearing in a known text block c. It is a term The probability of appearing in the query text q. It is a term The probability of appearing in a known text block c.

[0037] Finally, using semantic similarity , the importance of document blocks and relevance to the query subject The weighted sum of : ; in, is a constant obtained based on experience.

[0038] Document extraction module: The extraction module obtains high-quality content fragments, formats them together with the user's query questions, inputs them into the large language model (LLM), and finally obtains context-specific answers.

[0039] In addition, the present invention also provides a method for extracting information based on layout analysis and multimodality, comprising the following steps: 1. The user uploads the document to be processed through the system interface. The system performs content analysis, layout analysis, and document segmentation on the document, embeds the segmented text blocks into words, and outputs the vector expression of the text blocks; 2. The user inputs the content requirements to be extracted, designs a prompt, and uses the large language model to analyze the keywords, entities, intent, and rewritten queries of the question, and also converts them into vector expressions; 3. The retrieval enhancement module receives the results of document parsing and screens and filters them according to the query analysis module. By calculating the relevance between the question and the text block, the most relevant text block is filtered out; 4. The rearrangement and filtering module conducts an in-depth relevance evaluation on the data output by the retrieval enhancement module, combines semantic similarity, the importance of the text block, and the relevance to the query question, retains the text blocks with the strongest relevance and sorts them, further reducing the number of text blocks; 5. In the document extraction stage, the results of the rearrangement filtering module and the user's query questions are sent to the large language model; the extraction capabilities of the large language model are used to obtain the extraction results expected by the user.

[0040] Based on the technical solution of the present invention, Figure 3 The process shown is used to illustrate the implementation process of the present invention in practical application. The specific application implementation plan is as follows: 1. The user uploads the document to be processed through the system interface. The system performs content analysis (such as OCR), layout analysis, and document segmentation on the file, embeds the segmented text blocks, and outputs the vector expression (Embendding) of the text blocks. For example, in a financial reimbursement scenario, the user uploads a reimbursement form, and the system parses the reimbursement form to identify information such as titles, paragraphs, and tables. The document is segmented (such as by paragraph), and the vector expression is calculated for each text block.

[0041] 2. The user inputs the content requirements to be extracted, designs prompts, and uses the big language model to analyze the keywords, entities, intent, and rewritten queries of the question, and also converts them into vector expressions. For example, a user asks "What is the tax-free amount of commodity A in the reimbursement form?" Taking query rewriting as an example, after the designed prompts are added to the question, the text input into the big model is: """You are a professional query decomposition expert. Task: Given a user question, decompose it into multiple independent sub-questions that need to be answered to solve the original question. Note: If there are abbreviations or words that you are not familiar with, do not try to rephrase them. The user's question is "What is the tax-free amount of item A in the reimbursement form?" Please write your rephrased answer""" Thus, the output of the rewritten question of the large model is: "What is the total amount of product A? What is the tax amount of product A? How to deduct the tax amount from the total amount to get the amount excluding tax?" 3. The search enhancement module receives the results of document parsing and screens and filters them according to the query analysis module. By calculating the relevance between the question and the text block, the most relevant text block is filtered out. For example, in the reimbursement scenario, all information irrelevant to the product will be filtered out, such as the reason for the business trip, the superior's approval, the application time and other irrelevant information in the reimbursement application form will be filtered out, and only the information related to the product will be retained.

[0042] 4. Subsequently, the rearrangement and filtering module will conduct a more detailed relevance assessment on the data output by the retrieval enhancement module, and retain and sort the most relevant document blocks based on factors such as semantic similarity, the importance of the document block, and the relevance to the query topic, which can further reduce the number of text blocks. For example, in a reimbursement scenario, there may be amounts for items A, B, and C. The rearrangement and filtering module will only retain the information of item A based on the user's question, "What is the tax-free amount of item A in the reimbursement form?"

[0043] 5. In the document extraction stage, the results of the rearrangement filtering module and the user's questions are sent to the large language model. Use the extraction capabilities of the large language model to obtain the extraction results expected by the user. For example, if the field that is finally retained is "Product A is priced at 100 yuan and the tax rate is 3%", after entering it into the large language model, the large language model will give the answer "The tax-free amount of Product A is 97.00 yuan".

[0044] The innovative features of the present invention are as follows: 1. The present invention introduces layout analysis technology and multimodal fusion technology into the field of information extraction: through advanced layout analysis technology, complex layout structures in documents or images, such as text blocks, tables, pictures, etc., are accurately identified and parsed, and combined with multimodal data fusion technology, text, image and other modal data are deeply integrated, thereby greatly improving the accuracy and comprehensiveness of information extraction.

[0045] 2. The present invention combines efficient information retrieval with a large language model: a retrieval enhancement module is constructed, which uses a large language model (LLM) and an efficient information retrieval system to accurately retrieve the most relevant information to the user's query from a customized knowledge base, and enhances the accuracy and relevance of the extraction by means of prompt templates. Subsequently, a generative large language model is used to deeply understand and process the retrieved context, generate detailed and accurate answers or texts, and realize the intelligent upgrade of information extraction.

[0046] 3. The present invention is highly automated and can achieve information extraction without templates, with high efficiency and strong flexibility: the entire system, from document parsing to information extraction, is highly automated without manual intervention, greatly improving processing efficiency. At the same time, the preset templates in the traditional method are abandoned, and a template-free information extraction mechanism is adopted, making the system more flexible and able to adapt to various complex and changeable documents and data sources, reducing the cost of use and improving the scalability and maintainability of the system.

[0047] The above description is only a detailed description of the preferred embodiments and principles of the present invention. For ordinary technicians in this field, according to the ideas provided by the present invention, there will be changes in the specific implementation methods, and these changes should also be regarded as the protection scope of the present invention.

Claims

1. A layout analysis and multimodal information extraction system, characterized in that: include: Document parsing module, used to extract key information from documents in various formats; Query analysis module, used to deeply analyze and understand user query requests; A retrieval enhancement module, which is used to retrieve the information most relevant to the user's query request from a pre-customized knowledge base by combining the large language model LLM with the information retrieval system, and integrate the information into a prompt word template; A rearrangement and filtering module is used to reorder and filter the prompt word template output by the retrieval enhancement module to obtain the final content fragment; The document extraction module is used to obtain the final content fragment, format it together with the user's query request, and input it into the large language model LLM to finally obtain the required answer result.

2. The information extraction system based on layout analysis and multimodality according to claim 1, characterized in that: The document parsing module identifies the layout structure content of the document through multimodal OCR and layout recognition technology, and converts the layout structure content into structured data; the layout structure content includes titles, paragraphs, tables and images.

3. The information extraction system based on layout analysis and multimodality according to claim 2, characterized in that: The document parsing module includes a document slicing module, which is used to divide a document into multiple text blocks.

4. The information extraction system based on layout analysis and multimodality according to claim 1, characterized in that: The query analysis module specifically includes the following processes: Extract important keywords from the user's query request, and use the keywords to retrieve relevant information in the document library; Identify entities in the query request, including names of people, places, and organizations; Analyze the user's query request intent to determine whether the user intends to seek information, answer questions, or perform other tasks; According to the query analysis result, the original query request is appropriately rewritten so that the rewritten query request is more in line with the user's intention.

5. The information extraction system based on layout analysis and multimodality according to claim 3, characterized in that: The search enhancement module specifically includes the following processes: The vector expressions of the query request and the text block are taken as input, and the correlation between the query request and the text block is evaluated by calculating the similarity score between the query request vector and the text block vector; at the same time, a similarity threshold is set, and the text blocks whose corresponding similarity scores are higher than the similarity threshold are retained.

6. The information extraction system based on layout analysis and multimodality according to claim 5, characterized in that: The rearrangement filtering module specifically includes the following processes: The text blocks retained in the prompt word template are scored in multiple dimensions; the multiple dimensions include the semantic similarity between the text block and the query request, the importance of the text block and the relevance to the query request, and the scores of the multiple dimensions are weighted and summed to obtain a final scoring result; According to the final scoring results, several text blocks with the strongest relevance are retained and sorted.

7. A method for extracting information based on layout analysis and multimodality, using the system for extracting information based on layout analysis and multimodality as claimed in any one of claims 1 to 6, characterized in that: The information extraction method based on layout analysis and multimodality comprises the following steps: S1, the user uploads the document to be processed through the system interface, the system performs content analysis, layout analysis, and document segmentation, embeds the segmented text blocks into words, and outputs the vector expression of the text blocks; S2: The user inputs the content to be extracted, designs a prompt, and uses a large language model to analyze the keywords, entities, intent, and rewritten queries, and also converts them into vector expressions. S3, the retrieval enhancement module receives the results of document parsing and screens and filters them according to the query analysis module, and filters out the most relevant text blocks by calculating the relevance between the question and the text block; S4, the re-ranking and filtering module conducts an in-depth relevance evaluation on the data output by the retrieval enhancement module, combines the semantic similarity, the importance of the text block, and the relevance to the query question, retains the text blocks with the strongest relevance and sorts them, and further reduces the number of text blocks; S5, in the document extraction stage, the results of the rearrangement filtering module and the user's query questions are sent to the large language model; the extraction capability of the large language model is used to obtain the extraction results expected by the user.

Images