Extraction and retrieval method based on multi-type documents

Abstract

The invention relates to an extraction and retrieval method based on multi-type documents, which comprises the following steps of: obtaining tens of thousands of financial text data, carrying out manual annotation on a layout, and then giving an annotation result to a model for training; identifying characters in corresponding position ranges by adopting an iText technology and an OCR (Optical Character Recognition) technology for non-picture documents and picture documents respectively; converting unstructured data contained in the text into structured expression; an HNSW algorithm is adopted, and layered and efficient data retrieval is achieved in massive vector data; through combination of semantic vector retrieval in the text and traditional keyword matching, retrieval of the most relevant text can be well realized; and finally, obtaining the most relevant to-be-retrieved text by adopting an RRF algorithm. And based on the structured data, correlating texts of the same paragraph through context, and returning the texts as a final result. The problem that an existing document retrieval technology is insufficient in real-time performance and accuracy of financial document processing is solved, and the accuracy and efficiency of document retrieval are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a text retrieval technology, and in particular to an extraction and retrieval method based on multiple types of documents. Background Art

[0002] With the acceleration of digitization, the amount of unstructured data has increased dramatically, especially in financial documents such as research reports and news reports, which contain a wealth of valuable information. However, traditional text retrieval technologies often fail to accurately capture similarities between documents when processing this unstructured data, resulting in inaccurate search results. Furthermore, financial documents contain numerous complex elements such as charts and tables, which pose challenges for layout analysis and content recognition.

[0003] Current solutions typically rely on keyword-based retrieval technology, but this approach is limited when dealing with unstructured data and cannot effectively capture similarities between documents. Furthermore, when dealing with documents containing charts and tables, these technologies often struggle to accurately analyze the layout and identify the content.

[0004] The present invention mainly solves the following technical problems:

[0005] 1. When processing unstructured data, existing document retrieval technology cannot accurately capture the similarities between documents, resulting in inaccurate retrieval results.

[0006] 2. Unstructured data lacks clear structured representation, making it difficult to directly perform effective retrieval and analysis.

[0007] 3. Existing document recognition technologies often have difficulty accurately identifying different elements in documents, such as titles, tables, pictures, etc.

[0008] 4. Real-time updating and expansion of financial documents are crucial for investors, but existing financial information retrieval systems often fail to reflect the latest market changes in a timely manner. Summary of the Invention

[0009] Aiming at the problem that existing document retrieval technology is insufficient in real-time and accuracy in processing financial documents, a multi-type document extraction and retrieval method is proposed, aiming to provide a technology that can efficiently process unstructured financial documents.

[0010] The technical solution of the present invention is:

[0011] A method for extracting and retrieving documents based on multiple types of documents includes the following steps:

[0012] Step 1: Layout Analysis

[0013] The specific training process is as follows: First, tens of thousands of financial texts are obtained, then the pages are manually annotated, and then the annotated results are fed into the model for training;

[0014] Step 2: Content Identification

[0015] Text extraction from content involves two steps: first, obtaining the results of layout analysis, and then using iText technology and OCR technology to identify text within the corresponding position range for non-image documents and image documents respectively;

[0016] Step 3: RAG topology

[0017] Through the above-mentioned layout analysis and content recognition, the unstructured data contained in the text can be converted into structured expressions;

[0018] The specific approach is: first, different text contents are analyzed and identified through natural language processing technology and entity recognition technology; then, the extracted text contents can be stored in a tree structure by combining different paragraph distribution, content classification and directory hierarchy;

[0019] The specific techniques are as follows:

[0020] Text segmentation: identifying the logical relationships between paragraphs and determining the theme and content of the paragraphs;

[0021] Entity recognition: Identify proper nouns and key terms in text through entity recognition technology;

[0022] Syntactic analysis: Analyze the grammatical structure of a sentence and identify the subject, predicate, and object components in the sentence, which helps to understand the logical relationship between sentences;

[0023] Directory structuring: parse the directory structure of the document, determine the hierarchical relationship between chapters and sub-chapter, and build the tree structure of the document;

[0024] Step 4: Content retrieval

[0025] First, the text is converted into a vector representation based on BGE-M3. After conversion to a vector, the semantic similarity between two texts can be easily compared. The specific calculation method is to use the HNSW algorithm to achieve hierarchical and efficient data retrieval in massive vector data.

[0026] By combining the semantic vector retrieval mentioned above with traditional keyword matching, the retrieval of the most relevant text can be achieved better. Specifically, the text to be retrieved is first converted into a vector representation and word segmentation is performed, and then the vector similarity and word matching of all the texts to be retrieved and the texts to be matched are calculated respectively. Finally, the RRF algorithm is used to calculate the relevance of each text to be retrieved and the text to be matched by assigning different personalized weights to obtain the most relevant text to be retrieved. Then, based on the structured data generated in the RAG topology structure mentioned above, the text of the same paragraph is associated through context and returned as the final result.

[0027] Furthermore, the method further includes the following steps:

[0028] Real-time connection: Design a real-time linkage mechanism to connect with financial information sources in real time to achieve real-time retrieval and expansion of financial documents.

[0029] Market sentiment analysis: Extract key features of news information and stock forum posts and combine them with market indicators to generate market sentiment and trend information, and then associate and match them with multiple types of documents.

[0030] Association matching: Use entity recognition capabilities to extract financial entities and key indicators in documents, and perform association matching with related content in the documents, supporting dynamic updates after real-time query data.

[0031] Develop real-time interfaces to connect with financial data providers, regularly update financial documents, and use entity recognition tools to extract features related to financial entities, key indicators, and market sentiment, and link them with documents in real time.

[0032] Optionally, use a deep learning model for text vectorization.

[0033] Optionally, use the ANNS algorithm for similarity measurement.

[0034] Optionally, layout analysis and text recognition can be performed using model positioning, fuzzification, and neural network techniques.

[0035] The beneficial effects of the present invention are:

[0036] 1. Significantly improve the accuracy and efficiency of document retrieval.

[0037] 2. Support personalized retrieval needs.

[0038] 3. Improve the efficiency and accuracy of data processing.

[0039] 4. Support more flexible data retrieval and analysis.

[0040] 5. Identify and understand document content more accurately.

[0041] 6. Provide more comprehensive and timely financial information. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the content retrieval process of the present invention;

[0043] Figure 2 This is a flowchart of the real-time linkage of public domain content in the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0045] A method for extracting and retrieving documents based on multiple types of documents includes the following steps:

[0046] Step 1: Layout Analysis

[0047] Because a large amount of text in the financial sector, such as research reports and company announcements, has unique layouts and styles, traditional layout analysis models cannot be used to analyze its content. For these reasons, we retrained our layout analysis model based on our extensive financial corpus.

[0048] The specific training process involves first acquiring tens of thousands of financial texts, then manually annotating the layouts (identifying different types of text, tables, and images), and then feeding the model with the annotations for training. This approach significantly improves the layout analysis model's ability to identify financial data, increasing its accuracy from 40% to 91.9% (using a double-blind evaluation of multiple sample sets and a test set of thousands of data items, encompassing scenarios such as plain text, images, tables, and double-column formats, and types such as financial research reports, announcements, performance reports, and IM chat logs).

[0049] Step 2: Content Identification

[0050] Extracting text from content involves two steps. First, the results of layout analysis are obtained. Then, for non-image documents and image documents (such as scanned documents), iText (an open source software for processing text files) and OCR (Optical Character Recognition) technologies are used to identify the text within the corresponding position range.

[0051] Step 3: RAG topology

[0052] Through the above-mentioned layout analysis and content recognition, the unstructured data contained in the text can be converted into structured expression.

[0053] The specific approach is to first analyze and identify different text contents using natural language processing and entity recognition technologies. Then, combining different paragraph distributions, content classifications, and directory hierarchies, the extracted text content can be stored in a tree-like structure. The advantage of this approach is that a single text can be used to find related texts in the upper and lower hierarchies, making it easier to return semantically and structurally similar text blocks.

[0054] The specific techniques are as follows:

[0055] a. Text segmentation: Identify the logical relationship between paragraphs and determine the theme and content of the paragraphs;

[0056] b. Entity recognition: Identify proper nouns and key terms in the text through entity recognition technology;

[0057] c. Syntactic analysis: Analyze the grammatical structure of a sentence and identify the subject, predicate, and object components in the sentence, which helps to understand the logical relationship between sentences;

[0058] d. Directory structuring: parse the directory structure of the document, determine the hierarchical relationship between chapters and sub-chapter, and build the tree structure of the document.

[0059] Step 4: Content retrieval

[0060] Traditional keyword-based search methods have many drawbacks, the core problem being the inability to understand the semantics of the text. With this in mind, we first convert the text into a vector representation based on BGE-M3 (General Semantic Vector Model). After conversion to a vector, we can more easily compare the semantic similarity between two texts (similar data in the vector space generally have similar semantics). The specific calculation method uses the HNSW (Hierarchical Navigable Small Worldgraphs, an implementation of an approximate nearest neighbor algorithm) algorithm to achieve hierarchical and efficient data retrieval in massive vector data. The text that is most similar to the text to be matched in the content to be searched can be returned in hundreds of milliseconds.

[0061] By combining the semantic vector retrieval mentioned above with traditional keyword matching, the retrieval of the most relevant text can be achieved better. Specifically, the text to be retrieved is first converted into a vector representation and word segmentation is performed (the text is divided into representations with words as the smallest unit), and then the vector similarity and word matching of all texts to be retrieved and texts to be matched are calculated respectively. Finally, the RRF (Reciprocal rank fusion) algorithm is used to calculate the relevance of each text to be retrieved and text to be matched by assigning different personalized weights to obtain the most relevant text to be retrieved. Then, based on the structured data generated in the RAG topology structure mentioned above, the text of the same paragraph is associated with the context and returned as the final result.

[0062] The content described above can be referred to Figure 1 .

[0063] Step 5: Extending public domain financial content in real time

[0064] The ever-changing financial markets require investors and financial institutions to quickly access and analyze large amounts of financial information. Therefore, the ability to process and analyze financial data in real time has become increasingly important. This paper provides reference and tools for financial market participants through the following key steps:

[0065] Real-time connection: Design a real-time linkage mechanism to connect with financial information sources in real time to achieve real-time retrieval and expansion of financial documents.

[0066] Market sentiment analysis: Extract key features of news information and stock forum posts and combine them with market indicators to generate market sentiment and trend information, and then associate and match them with multiple types of documents.

[0067] Association matching: Use entity recognition capabilities to extract financial entities and key indicators in documents, and perform association matching with related content in the documents, supporting dynamic updates after real-time query data.

[0068] Develop real-time interfaces to connect with financial data providers, regularly update financial documents, and use entity recognition tools to extract features related to financial entities, key indicators, and market sentiment, and link them with documents in real time.

[0069] The specific process is as follows Figure 2 As shown;

[0070] While the technical solution of this invention is highly effective, alternative solutions may exist to achieve the same results. For example, deep learning models can be used for text vectorization; algorithms such as ANNS can be used for similarity measurement; and layout analysis and text recognition can be performed using model positioning, fuzzification, neural networks, and other methods. However, these alternative solutions require thorough verification to ensure their effectiveness and reliability.

[0071] The above-described embodiment merely represents one embodiment of the present invention. While the description is relatively specific and detailed, it should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make various modifications and improvements without departing from the spirit of the present invention, and these modifications and improvements fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A method for extracting and retrieving documents based on multiple types of documents, characterized in that: The following steps are involved: Step 1: Layout Analysis The specific training process is as follows: First, tens of thousands of financial texts are obtained, then the pages are manually annotated, and then the annotated results are fed into the model for training; Step 2: Content Identification Text extraction from content involves two steps: first, obtaining the results of layout analysis, and then using iText technology and OCR technology to identify text within the corresponding position range for non-image documents and image documents respectively; Step 3: RAG topology Through layout analysis and content recognition, the unstructured data contained in the text can be converted into structured expressions; The specific approach is: first, different text contents are analyzed and identified through natural language processing technology and entity recognition technology; then, the extracted text contents can be stored in a tree structure by combining different paragraph distribution, content classification and directory hierarchy; The specific techniques are as follows: Text segmentation: identifying the logical relationships between paragraphs and determining the theme and content of the paragraphs; Entity recognition: Identify proper nouns and key terms in text through entity recognition technology; Syntactic analysis: Analyze the grammatical structure of a sentence and identify the subject, predicate, and object components in the sentence, which helps to understand the logical relationship between sentences; Directory structuring: parse the directory structure of the document, determine the hierarchical relationship between chapters and sub-chapter, and build the tree structure of the document; Step 4: Content retrieval First, the text is converted into a vector representation based on BGE-M3. After conversion to a vector, the semantic similarity between two texts can be easily compared. The specific calculation method is to use the HNSW algorithm to achieve hierarchical and efficient data retrieval in massive vector data. By combining the semantic vector retrieval mentioned above with traditional keyword matching, the retrieval of the most relevant text can be achieved better. Specifically, the text to be retrieved is first converted into a vector representation and word segmentation is performed, and then the vector similarity and word matching of all the texts to be retrieved and the texts to be matched are calculated respectively. Finally, the RRF algorithm is used to calculate the relevance of each text to be retrieved and the text to be matched by assigning different personalized weights to obtain the most relevant text to be retrieved. Then, based on the structured data generated in the RAG topology structure mentioned above, the text of the same paragraph is associated through context and returned as the final result.

2. The method for extracting and retrieving documents based on multiple types of documents according to claim 1, characterized in that: The following steps are also included: Real-time connection: Design a real-time linkage mechanism to connect with financial information sources in real time to achieve real-time retrieval and expansion of financial documents. Market sentiment analysis: Extract key features of news information and stock forum posts and combine them with market indicators to generate market sentiment and trend information, and then associate and match them with multiple types of documents. Association matching: Use entity recognition capabilities to extract financial entities and key indicators in documents, and perform association matching with related content in the documents, supporting dynamic updates after real-time query data. Develop real-time interfaces to connect with financial data providers, regularly update financial documents, and use entity recognition tools to extract features related to financial entities, key indicators, and market sentiment, and link them with documents in real time.

3. The method for extracting and retrieving documents based on multiple types of documents according to claim 1, characterized in that: Use deep learning models for text vectorization.

4. The method for extracting and retrieving documents based on multiple types of documents according to claim 1, characterized in that: The ANNS algorithm is used for similarity measurement.

5. The method for extracting and retrieving documents based on multiple types of documents according to claim 1, characterized in that: Use model positioning, fuzzification, and neural network methods to perform layout analysis and text recognition.

Images