Retrieval enhanced question and answer method and system for mixed mode document content

Abstract

The invention discloses a retrieval enhanced question and answer method and system for mixed mode document content, and the method comprises the steps: obtaining a mixed mode document of a target industry, extracting hierarchical text blocks and table information blocks from the mixed mode document, generating a Merkel tree and a vector index database on the basis of semantic feature codes of the text blocks and the table information blocks, performing similarity search in the vector index database according to query vectors of user questions to obtain contexts of the query vectors, generating Merkel paths and retrieval enhancement generation chains corresponding to the contexts according to the Merkel tree, and performing retrieval enhancement on the query vectors according to the Merkel paths and the retrieval enhancement generation chains. And inputting the prompt template, the Merkel path and the retrieval enhancement generation chain into a preset large language model, and decoding and outputting an answer. According to the method, BWT conversion, non-standard transverse table transposition and layering and blocking are utilized, a model can understand structural query of table type data, through combination of double-encoder rapid recall and cross encoder fine matching, an output result is attached with a reference mark and a source file list, and the traceability of answers is enhanced.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of intelligent question answering, and in particular to a retrieval-enhanced question answering method and system for mixed modality document content. BACKGROUND

[0002] In the fields of government procurement, enterprise management and academic research, knowledge carriers are generally in the form of mixed modality documents containing unstructured text and semi-structured tables. The tables in the documents often carry key quantitative data, comparison information and attribute relationships, while the surrounding text provides context description, conclusion analysis and logical exposition. The two complement each other and together constitute a rich-form and complete-expressing document content. In recent years, intelligent question answering systems based on large models and retrieval-enhanced generation architectures have made significant progress. However, how to accurately and efficiently obtain knowledge from such mixed modality documents has become a core challenge in the current field of intelligent information retrieval and question answering.

[0003] The existing technology is relatively mature in the pure text scenario, but when directly applied to documents with mixed text and table modalities, it usually adopts a "table to text" flattening processing method, which simply converts the table into paragraph descriptions separated by special symbols, resulting in a large loss of key structured information such as the correspondence between row and column structures, cell values and field names within the table. These factors make it difficult for the system to understand structured semantics such as "data in a certain column", "items in a certain row", or "values at the intersection of a row and a column". When the user query involves "the values of A and B in the comparison table" or "list all the data in a certain column", the semantic retrieval of pure text fragments cannot accurately locate the entire table or the correct row and column fragments in the table, resulting in retrieval failure or lack of relevant context. The existing block method mostly directly and mechanically divides the document into different text fragments, which may separate the explanatory text and the referenced table into different fragments, even far apart, causing the system to easily obtain descriptive text while losing the core data table, or only retrieving the isolated data table while lacking the key text explanation. Even if the retrieval result contains table text fragments, the large language model receives a linear text with chaotic structure, which may cause hallucination problems such as value extraction errors and relationship misunderstandings in the answer generation stage. Therefore, there is an urgent need to develop an intelligent question answering method that can quickly respond to and accurately interpret the content of mixed modality documents, which is of great significance to the development and application of mixed modality document content comprehensive understanding, retrieval and question answering technology. SUMMARY

[0004] The application aims to provide a retrieval-enhanced question answering method and system for mixed modality document content.

[0005] To achieve the above-mentioned purpose, the application is implemented according to the following technical solutions: The first aspect of the present application provides a retrieval enhanced question and answer method for mixed modal document content, comprising: A. Obtain a mixed modal document of a target industry, preprocess the mixed modal document to obtain a text content block and a table content block, and the mixed modal document includes unstructured text and semi-structured table; B. Perform BWT transformation on the string of the text content block, identify repeated semantic substrings through a suffix array according to the transformation sequence, extract the row and column structure of the table content block, and obtain hierarchical text blocks and table information blocks; C. Perform semantic feature coding based on the text blocks and table information blocks, associate the coding embedding vector with the metadata hash value of the block, generate a Merkle tree according to the hash value, and obtain a vector index library; D. Obtain a user question and encode it into a query vector through a double encoder, perform similarity search in the vector index library using the query vector, match and score the similarity search information block through a cross encoder, and take the optimal matching information block as the context of the query vector; E. Generate a Merkle path corresponding to the context according to the Merkle tree and a retrieval enhancement generation chain, take the context as a prompt template, input the prompt template, Merkle path and retrieval enhancement generation chain into a preset large language model, and decode to output an answer containing reference marks and a list of related information block source files.

[0006] Further, the method for obtaining the mixed modal document comprises: Obtain a mixed modal document of a target industry, the mixed modal document includes unstructured text and semi-structured table, the unstructured text includes paragraph text, comment text, chapter description, and context descriptive content in the document, and the semi-structured table includes merged cells, multi-level table headers, statistical tables with cross-row and cross-column association, parameter comparison tables, and data detail tables; Remove watermarks, invalid white spaces, repeated page numbers and headers and footers in the mixed modal document, and convert them into a unified document object model structure, split the normalized unstructured text based on punctuation marks, paragraph separators and chapter titles to obtain a text content block, label metadata for the text content block, and if the number of characters in a single semantic unit exceeds 1000 characters, split it into multiple text content blocks according to complete sentence boundaries; Divide the pure text area and the table area based on the table border, remove the empty cells and invalid placeholder cells in the table, restore the merged cells and multi-level table headers of the semi-structured table in the table area to a standard row and column structure through cell coordinate mapping, supplement the missing row and column indexes of the standard row and column structure, extract the table header level information, cell value and table title, obtain a table content block, and label metadata for the table content block; obtaining text content block set and table content block set based on the metadata, which includes document identification, content block identification, text starting position, character length, belonging chapter, table row and column number, table header type, data dimension, and eliminating text content block and table content block whose character recognition error rate and row and column structure missing rate exceeds 5%.

[0007] Further, the method for obtaining the repeated semantic sub-string comprises: performing BWT transformation on the string of the text content block to obtain a circular shift sequence of the string, sorting the circular shift sequence in lexicographical order, extracting the last column of characters of the sorted sequence to obtain a BWT transformation sequence, recording the original string position index in the transformation process, constructing a suffix array according to the BWT transformation sequence, calculating a longest common prefix array, and obtaining the repetition degree of the repeated semantic sub-string according to the longest common prefix array, wherein the repetition degree calculation formula is: ; wherein represents the repetition degree of the repeated semantic sub-string, represents the longest common prefix length of the first , suffixes in the suffix array, represents the semantic weight of the sub-string, which is obtained by pre-training an industry word vector model, represents the total length of the text content block, represents a distance decay coefficient, which is obtained by linear regression fitting, represents the character offset difference of the sub-string in the text content block; if the repetition degree is greater than or equal to 0.2, it is determined that the repeated semantic sub-string is valid, the repeated characters are eliminated, and the position, repetition frequency and semantic category of the repeated semantic sub-string are marked to obtain a repeated semantic sub-string list.

[0008] Further, the method for obtaining the text block comprises: aggregating text segments containing the same core repeated semantic sub-string into a first-level block based on the text content block, and the aggregation weight calculation formula is: ; wherein represents the semantic aggregation weight of the first level text block, and the input item is the repetition degree of the first repeated semantic sub-string, represents the total number of repeated semantic sub-strings, represents a hierarchical decay coefficient, which is obtained by fitting according to the semantic similarity of parent and child blocks, represents the hierarchical number of the first block. the character length of the first repeated semantic substring, the total character length of the first hierarchical text block; According to the block annotation core repeated substring, block semantic theme, the first text block is split according to the affiliation of the repeated semantic substring, and if there is no affiliation, it is split according to the complete semantic sentence boundary, and the hierarchical text block is annotated with a hierarchical index, associated with the repeated semantic substring list and the text content block metadata, and a structured hierarchical text block set is obtained.

[0009] Further, the method for obtaining the table information block comprises: Determine whether it is a standard horizontal table format based on the row and column types of the cells in the table content block. If it contains a serial number column, remove the column, and calculate the average semantic similarity of the cell contents between the data rows. The average semantic similarity of the row is used as the row semantic similarity. The average semantic similarity of the cell contents between the data columns excluding the title row and the first column is used as the column semantic similarity. The semantic similarity calculation formula is: ; ; Wherein is the row semantic similarity, is the semantic similarity of the cell and , the cosine similarity, is the cell content of the first row and the first column, is the cell content of the first row and the first column, is the semantic similarity of the cell and , is the cell content of the first row and the first column, is the cell content of the first row and the first column, is the number of table rows, is the number of table columns, is the column semantic similarity; If the row semantic similarity is less than the preset consistency threshold and the column semantic similarity is greater than or equal to the preset consistency threshold, the table is a non-standard horizontal table and is transposed. The standard format horizontal table after transposition is judged whether it contains a row header column by a row header column recognition algorithm. The calculation formula of the row header column recognition algorithm is: ; ; wherein is a row header confidence, is a cell content and a column name between a semantic similarity function, is a column header of the column, is a very small positive number, is an average row header confidence of rows in the first column; The average row header confidence of rows in the first column is greater than the preset confidence threshold, the first column is determined as a row header column, the cells of the table are traversed, if the row header column is identified, the text content of the row header and the column header is added to the cell content, if the row header column is not identified, only the text content of the column header is added, and the table information block is obtained.

[0010] Further, the method for obtaining the vector index library comprises: Based on hierarchical text blocks, the hierarchical text blocks are encoded by a pre-trained industry word vector model to obtain semantic embedding vectors of fixed dimensions, and based on table information blocks, the table information blocks are encoded by a table pre-training model to generate FAISS indexes according to the semantic feature encoding; According to the metadata of the text blocks and the table information blocks, a block identity is generated by a hash algorithm, the semantic embedding vectors and the metadata hash values are used as key-value pairs, the key-value pairs are used as leaf nodes of a Merkle tree, and intermediate node hashes are generated, and the aggregation formula of the intermediate node hashes is: ; wherein is an aggregation hash value of a Merkle tree intermediate node, is a hash algorithm, is a hash value of a left child node, is a hash value of a right child node, is a modal type identifier, which distinguishes text blocks and table information blocks; The encoding embedding vectors of the semantic feature encoding and the Merkle tree are stored in the index library to obtain the vector index library.

[0011] Further, the method for obtaining the similarity search information block comprises: The user's question text is input into a pre-trained industry-specific word vector model to obtain pure text semantic features. If the question text contains table intent keywords, the question is converted into a table semantic query. The query is then input into a lightweight table encoder to obtain a table semantic perception vector. The two-branch vectors are fused using an intent weight formula to obtain the query vector. The intent weight formula is as follows: ; in For query vector, Text intent weights, The data is based on pure text semantic features, and is dynamically adjusted according to the text intent probability of these features, with an adjustment range of 0.3-0.7. For L2 normalization, For table semantic awareness vectors, The table intent score is calculated based on the matching degree of the table intent keywords. The cosine similarity between the query vector and the encoded embedding vector is calculated based on the vector index library. Candidate information blocks are extracted using nearest neighbor search based on the cosine similarity. The cosine similarity formula is: ; in query vector With information block cosine similarity, For the first The encoded embedding vector of each information block, The strength of the association between hashes and semantic vectors in an information block; The Merkel path confidence score of the candidate information block is calculated using the following formula: ; in For information blocks Merkel path confidence For the first The hash consistency score of the Merkel path for each information block is 1 when it is completely consistent. This is the path level length. The attenuation coefficient is... Confidence level of search results; Candidate information blocks with a confidence level greater than 0.85 in the Merkel path are used as similarity search information blocks.

[0012] Furthermore, the method for obtaining the context includes: The text input sequence of the user question, the text block content and the block metadata is constructed based on the similarity search information block, and is input into a BERT-TAPAS cross encoder, and a text basic matching score is output, and the text basic matching score is corrected by a text block weight formula, and the text block weight formula is: ; wherein is a text correction score, is a repetition degree of the i-th repeated semantic substring in the text block, is a number of effective repeated substrings, is a hierarchical attenuation factor; The table input sequence of the user question, the row header, the column header, the cell and the metadata is constructed based on the similarity search information block, and a table basic matching score is output through a BERT-TAPAS cross encoder, the column semantic similarity of the key column is multiplied by the table basic matching score, and the product is taken as a table correction score, and the text correction score and the table correction score of the similarity search information block are mixed and rearranged in relevance, and the scoring formula of the mixed relevance rearrangement is: ; wherein is a rearrangement score, is a weight coefficient obtained according to the proportion of modal data, is a cosine similarity of a query vector and an information block , and is a correction score of the corresponding modal data; According to the rearrangement score, the optimal matching information block is taken as the context of the query vector.

[0013] Further, the method for obtaining the answer comprises: Based on the context, the corresponding leaf node in the Merkle tree is located according to the metadata hash value, the root node of the Merkle tree is backtracked in sequence according to the leaf node, the hash values of the intermediate nodes in the backtracking path, the block identifiers and the modal type identifiers associated with the nodes are obtained, and a Merkle path is generated; The encoding embedding vector of the context is taken as a retrieval enhancement generation chain semantic layer, the hash sequence of the Merkle path and the metadata are taken as a retrieval enhancement generation chain check layer, and the document identifier corresponding to the context information block is taken as a retrieval enhancement generation chain association layer, and a retrieval enhancement generation chain is obtained; According to the context, a structured prompt template containing a specific instruction description, a context placeholder and a question placeholder is constructed, the prompt template containing the instruction and the placeholder, the Merkle path and the retrieval enhancement generation chain are input into a preset large language model, and an answer containing a reference mark and a related information block source file list is decoded and output.​

[0014] The second aspect of the present application provides a retrieval enhanced question and answer system for mixed modal document content, comprising: A data acquisition module is configured to acquire mixed modal documents of a target industry, preprocess the mixed modal documents to obtain text content blocks and table content blocks, and the mixed modal documents include unstructured text and semi-structured tables. A data division module is configured to perform BWT transformation on strings of the text content blocks, identify repeated semantic substrings through a suffix array according to a transformation sequence, extract row and column structures from the table content blocks, and obtain hierarchical text blocks and table information blocks. An index library construction module is configured to perform semantic feature coding based on the text blocks and the table information blocks, associate the coding embedding vectors with metadata hash values of the blocks, generate a Merkle tree according to the hash values, and obtain a vector index library. A retrieval module is configured to acquire a user question, encode the question into a query vector through a double encoder, perform similarity search in the vector index library using the query vector, match and score similar information blocks through a cross encoder, and take an optimal matching information block as a context of the query vector. An answer generation module is configured to generate a Merkle path corresponding to the context according to the Merkle tree and a retrieval enhancement generation chain, take the context as a prompt template, input the prompt template, the Merkle path and the retrieval enhancement generation chain into a preset large language model, and decode and output an answer containing reference marks and a list of related information block source files.

[0015] Compared with the prior art, the embodiments of the present application have at least the following advantages or beneficial effects: The present application performs BWT transformation and identifies repeated semantic substrings through a suffix array, performs block processing on the text and tables of mixed modal documents, dynamically aggregates repeated term definition paragraphs based on semantic density, reduces the possibility of splitting key information, improves recall accuracy, automatically identifies row headers and non-standard horizontal table transposition through row and column semantic similarity calculation, preserves the original structure information of the table, enables the model to understand structural queries such as certain column data, alleviates the problem of relationship loss caused by table linearization, converts data integrity verification from post-filtering to pre-sorting weight by constructing a Merkle tree and generating a path confidence, makes the retrieval result preferentially select low-risk blocks that pass the cryptography verification, enhances the credibility of the answer, balances the calculation efficiency and semantic depth by combining the fast recall of the double encoder and the fine matching of the cross encoder, and makes the output result accompanied by reference marks and a list of source files, thereby enhancing the traceability and verifiability of the answer. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flow chart of steps of a retrieval enhanced question answering method for mixed modal document content in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0018] Referring to Figure 1 As shown in the figure, the present application provides a retrieval enhanced question answering method for mixed modal document content, comprising: A. Obtain a mixed modal document of a target industry, preprocess the mixed modal document to obtain text content blocks and table content blocks, and the mixed modal document includes unstructured text and semi-structured tables; In actual evaluation, select the pdf document related to the 2024 computer batch centralized procurement configuration standard, which contains unstructured text revenue analysis paragraphs, chapter descriptions, and annotated texts, with a total character number of about 82,000. The semi-structured table includes four quarterly configuration detail tables, containing merged cells, three-level table headers, and cross-row and cross-column related revenue data. Execute the minerU parsing command to convert to Markdown format, correct conversion errors, remove headers, footers, and noise information, remove watermarks, duplicate headers on pages 10-12, and invalid white space characters. Convert the document to a unified DOM structure, split the text based on punctuation marks, government procurement, and enterprise operation chapter titles, and label the metadata. The document identifier is 2024-GP-001, and the content block identifiers are T1-001 to TB1-003. The "government procurement" text segment has 1,200 characters, which is split into two text content blocks according to complete sentence boundaries, T1-018 with 650 characters and T1-019 with 550 characters. The T1-021 text block with a character recognition error rate of 6% is removed; Based on the table border, the table area is divided, and the three invalid placeholder cells in the revenue detail table are removed. The merged cell "core product" is merged column, and is restored to the standard row and column structure through coordinate mapping. The final table has 12 rows and 8 columns. The row and column indexes are supplemented and the metadata is labeled. The table identifier is TB2-001, the row and column number is 12x8, the table header type is three-level table header, the TB2-003 table block with a row and column structure loss rate of 7% is removed, and the character recognition error rate of each remaining block is less than 5%. The row and column structure loss rate of the table is less than 3%; B. Perform BWT transformation on the string of the text content block, identify repeated semantic substrings through the suffix array according to the transformation sequence, extract the row and column structure of the table content block, and obtain hierarchical text blocks and table information blocks. In the actual evaluation, the strings such as "2.1·Desktop Computer Procurement Requirements Standard" in the T1-018 text block were subjected to BWT transformation to obtain a cyclic shift sequence and construct a suffix array. The LCP of the longest common prefix array was calculated to be 10, and the repetition rate was calculated to be 0.22, which was determined to be a valid repetitive semantic substring. A list was generated, in which the substring position is the 20th-32nd character, the repetition frequency is 2 times, the semantic category is procurement data, and the semantic weight of the substring is 0.95, which was obtained by the industry word vector model based on "core product procurement". The total length of the text content block is 650, and the distance decay coefficient is 0.02. ; For table TB2-001, row and column semantic similarity are calculated. Rows 3 and 4 have a row semantic similarity of 0.72, below the preset threshold of 0.8. Cols 2 and 3 have a column semantic similarity of 0.91, greater than or equal to the preset threshold of 0.8. Therefore, they are classified as non-standard horizontal tables and transposed. The average row header confidence score calculated using the row header and column header recognition algorithm is 0.88, greater than or equal to the preset confidence threshold of 0.85. Therefore, the first column is identified as the row header column. Row and column header text is added to the cells. Cell C(1,2) contains 5000, with the row header "Configuration 1" and the column header "Budget Limit". The text description is "Budget Limit of Configuration 1". The text description, limited to 5000, is converted into a table information block with the corresponding metadata format {"title":"Table 2-1: Configuration Standard for Centralized Procurement of Desktop Computers","item":"Configuration 1","key":"Budget Limit","source":"2024 Computer Centralized Procurement Configuration Standard.pdf","publish_date":"2024-04-28"}. The text description "Budget Limit of Configuration 1" is generated, and the repeated semantic substring "Configuration 1" is associated. Finally, all generated text blocks and table information blocks are saved to a JSON format file. C. Based on the text blocks and table information blocks, perform semantic feature encoding, associate the encoded embedding vector with the metadata hash value of the block, generate a Merkle tree based on the hash value, and obtain a vector index library; In the actual evaluation, bge-large-zh-v1.5 was used to encode text blocks and table information blocks into 1024-dimensional vectors, where... ; based on the FAISS index, the clustering number is 100, and the metadata of the text block and the table information block are calculated SHA-256 hash value, wherein the T1-018 hash value is: 7e8d9f7a8b7c6d5e, the embedding vector and the hash value are taken as the leaf node, the intermediate node hash is generated according to the aggregation formula, and finally the Merkle tree is generated. The semantic embedding vector and the Merkle tree are stored in the FAISS index library to form a vector index library, which contains 42 text blocks, 3 table information blocks, and hash data; D, get the user question, and encode it into a query vector through a double encoder. Use the query vector to perform similarity search in the vector index library. Match and score the similar information blocks through the cross-encoder. The optimal matching information block is used as the context of the query vector. In actual evaluation, the user question "If both desktop computers and laptops are configured with 5, which has a higher budget limit?" is input into the bge-large-zh-v1.5 model to obtain pure text semantic features. Because the question contains "budget" and "configuration" table intent keywords, it is converted into a table semantic query and input into a lightweight table encoder. According to the intent weight formula, the text intent weight is 0.6, and the table intent matching degree is 0.88. The query vector is generated ; based on the nearest neighbor search ability of the FAISS index library, the basic cosine similarity is calculated, the score is calculated by bge-reranker-large, and the top-5 candidate information blocks C1, C2, C3, C4 and C5 are selected, wherein C2 includes "... laptop configuration 5... budget limit 7000 yuan...". The candidate information blocks are verified in parallel for Merkle path confidence, and the confidence of T1-018 is 0.92 and the confidence of TB2-001 is 0.91, both of which are greater than 0.85. The candidate information block is retained. The candidate is input into the bge-reranker-large cross-encoder, and the text correction score is calculated to be 0.17 and the table correction score is calculated to be 0.80. After mixed relevance rearrangement, C1 and C2 are the optimal matching information blocks, which are determined as the context; E, according to the Merkle tree, the Merkle path corresponding to the context and the retrieval enhancement generation chain are generated, the context is taken as a prompt template, the prompt template, the Merkle path and the retrieval enhancement generation chain are input into a preset large language model, and the answer containing reference marks and related information block source file list is decoded and output.

[0019] In the actual evaluation, based on the hash value of TB2-001, the Merkle tree leaf node is located, the generation path is traced back, the retrieval enhancement generation chain is constructed, the context semantic layer is modified by embedding the vector score, the data verification layer is the hash sequence plus the metadata error rate 0.02, the source file association layer is the document identifier 2024-Fin-001 plus the table identifier TB2-001, combined with the user query question Q, the context designed Prompt prompt template returned in step D, the RAG sequence execution chain is reconstructed, the basic large model Qwen2.5 is selected to receive input information, the large model is guided to give the thinking reasoning process and generate the final answer based on the factual information block, and the Prompt prompt template is "You are an expert-level assistant familiar with government procurement policies and business, help users understand information by showing your thinking process and providing clear, cited answers.

[0020] Now, you have received a user question and need to design an answer draft that already contains citations in combination with the relevant original context.

[0021] Your task is to perform the following two steps: 1. **Thinking Process**: First, please explain step by step how you will use the provided context and answer draft to construct a final, comprehensive answer. Describe your reasoning process.

[0022] 2. **Final Answer**: Second, based on your thinking process, generate a polished, easy-to-read final answer. Rewrite the answer draft into a smooth, coherent text. Convert the inline citations `[source: policy document name, chunk: information block title]` to numbered footnotes (e.g., `[1]`, `[2]`). At the end of your answer, list all sources corresponding to the numbered footnotes.

[0023] Here is the information provided to you: **Original Context:** {context} **User Question:** {question} Now, please follow the two-step structure (thinking process and final answer) to provide your answer. If the context information is insufficient to answer, please say "Cannot answer based on available information." Where context is the relevant policy context, question is the user query question, and the user input question is "If both desktop computers and laptops are selected with configuration 5, which has a higher budget limit?" The decoded output answer is that the budget limit for a laptop with configuration 5 is 7000 yuan [1]And the budget limit for desktop computers with configuration 5 is 5000 yuan [2] Therefore, under the same configuration, the budget limit for portable computers is higher, [1]: Annex 2 Portable Computer Batch Centralized Procurement Configuration Standard (2024 Edition), Information Block ID: C2 (TB1-002) Merkel Path: Leaf-8 (8f7e6d5c...) → 7a6b5c4d... → 9d8c7b6a..., [2]: Annex 1 Central State Organs Desktop Computer Batch Centralized Procurement Configuration Standard - 2024 Edition, Information Block ID: C1 (T1-002) Merkel Path: Leaf-5 (6a5b4c3d...) → 6f5e4d3c... → 9d8c7b6a...

[0024] In this embodiment, the method for obtaining the mixed modal document comprises: Obtain the mixed modal document of the target industry, the mixed modal document comprising unstructured text and semi-structured table, the unstructured text comprising paragraph text, annotation text, chapter description, and contextual descriptive content in the document, and the semi-structured table comprising merged cells, multi-level table headers, statistical tables with cross-row and cross-column association, parameter comparison tables, and data detail tables; Remove watermarks, invalid white spaces, repeated page numbers, and headers and footers in the mixed modal document, and convert them into a unified document object model structure. Split the normalized unstructured text based on punctuation marks, paragraph separators, and chapter titles to obtain text content blocks. Label metadata for the text content blocks. If the number of characters in a single semantic unit exceeds 1000 characters, split the text content blocks into multiple text content blocks according to complete sentence boundaries. Divide the pure text area and the table area based on the table border. Remove empty cells and invalid placeholder cells in the table. Restore the merged cells and multi-level table headers of the semi-structured table in the table area to a standard row-column structure through cell coordinate mapping. Supplement missing row and column indices for the standard row-column structure. Extract table header level information, cell values, and table titles to obtain table content blocks. Label metadata for the table content blocks. Remove text content blocks and table content blocks with a character recognition error rate and a row-column structure missing rate exceeding 5% based on the metadata. Obtain a set of text content blocks and a set of table content blocks. The metadata includes document identification, content block identification, text starting position, character length, belonging chapter, table row and column number, table header type, and data dimension.

[0025] In this embodiment, the method for obtaining the repeated semantic sub-string comprises: BWT transformation is performed on the string based on the text content block to obtain a circular shift sequence of the string, the circular shift sequence is sorted in lexicographical order, and the last column of characters of the sorted sequence is extracted to obtain a BWT transformed sequence, the original string position index in the transformation process is recorded, a suffix array is constructed based on the BWT transformed sequence, and a longest common prefix array is calculated, and the repetition degree of the repeated semantic substring is obtained based on the longest common prefix array, and the repetition degree calculation formula is: ; wherein represents the repetition degree of the repeated semantic substring, represents the longest common prefix length of the first , suffixes in the suffix array, represents the semantic weight of the substring, which is obtained by pre-training an industry word vector model, represents the total length of the text content block, represents a distance decay coefficient, which is obtained by linear regression fitting, represents the character offset difference of the substring in the text content block; If the repetition degree is greater than or equal to 0.2, it is determined that the repeated semantic substring is effective, repeated characters are removed, the position, repetition frequency and semantic category of the repeated semantic substring are marked, and a list of repeated semantic substrings is obtained.

[0026] In the embodiment, the method for obtaining the text block includes: Text segments containing the same core repeated semantic substring are aggregated into a first-level block based on the text content block, and the aggregation weight calculation formula is: ; wherein represents the semantic aggregation weight of the first level text block, and the input item is the repetition degree of the first repeated semantic substring, represents the total number of repeated semantic substrings, represents a hierarchical decay coefficient, which is obtained by fitting based on the semantic similarity of parent and child blocks, represents the hierarchical number of the first block, represents the character length of the first repeated semantic substring, represents the total character length of the first level text block; Based on the core repeating substrings and semantic topics of the blocks, the first-level text blocks are split according to the subordinate relationship of the repeating semantic substrings. If there is no subordinate relationship, they are split according to the boundary of the complete semantic sentence. This forms a hierarchical index for the hierarchical text block annotation. The list of repeating semantic substrings and the metadata of the text content blocks are associated to obtain a structured hierarchical set of text blocks.

[0027] In this embodiment, the method for obtaining the table information block includes: Based on the row and column types of cells within the table content block, it is determined whether it is a standard horizontal table format. If it contains a serial number column, that column is removed. The average semantic similarity of cell content between data rows is calculated, and this average row semantic similarity is used as the row semantic similarity. The average semantic similarity of cell content between data columns excluding the header row and the first column is used as the column semantic similarity. The formula for calculating semantic similarity is as follows: ; ; in For line semantic similarity, For cells and The semantic similarity is expressed as cosine similarity. For the first Line number The content of the column cells, For the first Line number The content of the column cells, For cells and semantic similarity, For the first Line number The content of the column cells, For the first Line number The content of the column cells, For the number of rows in the table, For the number of columns in the table, For semantic similarity of columns; If the row semantic similarity is less than a preset consistency threshold and the column semantic similarity is greater than or equal to the preset consistency threshold, then the table is a non-standard horizontal table and is transposed. The transposed standard format horizontal table is then evaluated using a row header column recognition algorithm to determine whether it contains a row header column. The formula for this algorithm is as follows: ; ; in For the confidence level of the line, For cell content and the column name a semantic similarity function between, is the first column header of the column, is a very small positive number, is the average row header confidence of the rows in the first column; Calculate the row header confidence of the first column cell content and each column header, if the average row header confidence of the rows in the first column of the entire table is greater than the pre-set confidence threshold, then determine that the first column is a row header column, traverse the table cells, if the row header column is identified, add the text content of the row header and column header to the cell content, if the row header column is not identified, only add the text content of the column header, and obtain the table information block.

[0028] In the embodiment, the method for obtaining the vector index library comprises: Based on hierarchical text segmentation, the hierarchical text segmentation is encoded by a pre-trained industry word vector model to obtain a fixed dimension semantic embedding vector, and the table information block is encoded by a table pre-training model based on semantic feature coding, and the FAISS index is generated according to the semantic feature coding; According to the metadata of the text segmentation and the table information block, a segmentation identity is generated by a hash algorithm, the semantic embedding vector and the metadata hash value are used as a key-value pair, the key-value pair is used as a leaf node of a Merkle tree, and an intermediate node hash is generated, and the aggregation formula of the intermediate node hash is: ; Wherein is the aggregation hash value of the intermediate node of the Merkle tree, is a hash algorithm, is the hash value of the left child node, is the hash value of the right child node, is a modal type identifier, which distinguishes between text segmentation and table information block; The encoding embedding vector of the semantic feature coding and the Merkle tree are stored in the index library to obtain the vector index library.

[0029] In the embodiment, the method for obtaining the similarity search information block comprises: The problem text of the user question is input into a pre-trained industry word vector model to obtain pure text semantic features, if the problem text contains a table intent keyword, the question is converted into a table semantic query formula, the query formula is input into a lightweight table encoder to obtain a table semantic perception vector, and the double-branch vector is fused by an intent weight formula to obtain a query vector, and the intent weight formula is: ; Wherein is a query vector, Text intent weights, The data is based on pure text semantic features, and is dynamically adjusted according to the text intent probability of these features, with an adjustment range of 0.3-0.7. For L2 normalization, For table semantic awareness vectors, The table intent score is calculated based on the matching degree of the table intent keywords. The cosine similarity between the query vector and the encoded embedding vector is calculated based on the vector index library. Candidate information blocks are extracted using nearest neighbor search based on the cosine similarity. The cosine similarity formula is: ; in query vector With information block cosine similarity, For the first The encoded embedding vector of each information block, The strength of the association between hashes and semantic vectors in an information block; The Merkel path confidence score of the candidate information block is calculated using the following formula: ; in For information blocks Merkel path confidence For the first The hash consistency score of the Merkel path for each information block is 1 when it is completely consistent. This is the path level length. The attenuation coefficient is... Confidence level of search results; Candidate information blocks with a confidence level greater than 0.85 in the Merkel path are used as similarity search information blocks.

[0030] In this embodiment, the method for obtaining the context includes: The text input sequence, consisting of user questions, text chunk content, and chunk metadata, is constructed based on similarity search information blocks and input into the BERT-TAPAS cross-encoder. The output is a basic text matching score, which is then corrected using a text chunk weighting formula. The text chunk weighting formula is as follows: ; in Score the text corrections. For the first text block The degree of repetition of repeated semantic substrings. To determine the effective number of repeating substrings, Hierarchical attenuation factor; The table input sequence of user questions, column headers, row headers, cells and metadata is constructed based on the similarity search information block, and a table-based matching score is output through a BERT-TAPAS cross encoder, the column semantic similarity of a key column is multiplied by the table-based matching score, and the product is used as a table correction score, and the text correction score of the similarity search information block and the table correction score are mixed correlation rearranged, and the scoring formula of the mixed correlation rearrangement is: ; Wherein is the rearrangement score, is a weight coefficient obtained according to the proportion of modal data, is a query vector and the cosine similarity of the information block , is the correction score of the corresponding modal data; According to the rearrangement score, the optimal matching information block is used as the context of the query vector.

[0031] In this embodiment, the method for obtaining the answer comprises: Based on the context, the corresponding leaf node in the Merkle tree is located according to the metadata hash value, the root node of the Merkle tree is backtracked in turn according to the leaf node, the hash value of the intermediate node in the backtracking path, the block identifier and the modal type identifier associated with the node are obtained, and a Merkle path is generated; The encoding embedding vector of the context is used as a retrieval enhancement generation chain semantic layer, the hash sequence of the Merkle path and the metadata are used as a retrieval enhancement generation chain check layer, and the document identifier corresponding to the context information block is used as a retrieval enhancement generation chain association layer, and a retrieval enhancement generation chain is obtained; According to the context, a structured prompt template containing specific instruction descriptions, context placeholders and question placeholders is constructed, the prompt template containing instructions and placeholders, the Merkle path and the retrieval enhancement generation chain are input into a preset large language model, and the answer containing reference marks and a list of information block source files is decoded and output.

[0032] The second aspect of the application also provides a retrieval enhancement question and answer system for mixed modal document content, comprising: A data acquisition module is used to acquire mixed modal documents of a target industry, preprocess the mixed modal documents, obtain text content blocks and table content blocks, and the mixed modal documents include unstructured text and semi-structured tables; A data division module is used to perform BWT transformation on the string of the text content block, identify repeated semantic substrings through a suffix array according to the transformation sequence, extract row and column structures from the table content block, and obtain hierarchical text blocks and table information blocks; An index library construction module is configured to perform semantic feature coding based on the text blocks and table information blocks, associate the coding embedding vectors with metadata hash values of the blocks, generate a Merkle tree according to the hash values, and obtain a vector index library; A retrieval module is configured to obtain a user question, encode the question into a query vector through a double encoder, perform a similarity search in the vector index library using the query vector, match and score information blocks obtained through the similarity search through a cross encoder, and take an optimal matching information block as a context of the query vector. An answer generation module is configured to generate a Merkle path corresponding to the context and a retrieval enhancement generation chain according to the Merkle tree, take the context as a prompt template, input the prompt template, the Merkle path, and the retrieval enhancement generation chain into a preset large language model, and decode an output answer containing a reference mark and a list of related information block source files.

[0033] The above is merely an example and a description of the structure of the present application. Those skilled in the art can make various modifications or supplements to the described specific embodiments or use similar ways to replace them, as long as they do not deviate from the structure of the present application or exceed the scope defined by the present claims.

Claims

1. A retrieval augmented question answering method for mixed modality document content, characterized in that, The method comprises the following steps: A. Obtain a mixed modal document of a target industry, preprocess the mixed modal document to obtain a text content block and a table content block, and the mixed modal document comprises unstructured text and semi-structured tables; B. Perform BWT transformation on the string of the text content block, identify repeated semantic sub-strings through a suffix array according to the transformation sequence, extract the row and column structure of the table content block, and obtain hierarchical text blocks and table information blocks; C. Perform semantic feature coding based on the text blocks and the table information blocks, associate the coding embedding vector with the metadata hash value of the block, generate a Merkle tree according to the hash value, and obtain a vector index library; D. Obtain a user question, encode it into a query vector through a double encoder, perform similarity search in the vector index library using the query vector, match and score the similar information blocks through a cross encoder, and take the optimal matching information block as the context of the query vector; E. Generate a Merkle path corresponding to the context and a retrieval enhancement generation chain according to the Merkle tree, take the context as a prompt template, input the prompt template, the Merkle path and the retrieval enhancement generation chain into a preset large language model, and decode to output an answer containing reference marks and a list of related information block source files.

2. The retrieval augmented question answering method for mixed modality document content according to claim 1, wherein, The method for obtaining the mixed modal document comprises: Obtain a mixed modal document of a target industry, wherein the mixed modal document comprises unstructured text and semi-structured tables, the unstructured text comprises paragraph text, annotation text, chapter description, and context descriptive content in the document, and the semi-structured tables comprise merged cells, multi-level table headers, statistical tables with cross-row and cross-column association, parameter comparison tables, and data detail tables; Remove watermarks, invalid white spaces, repeated page numbers and headers and footers in the mixed modal document, and convert them into a unified document object model structure, split the normalized unstructured text based on punctuation marks, paragraph separators and chapter titles to obtain text content blocks, label metadata for the text content blocks, and if the number of characters in a single semantic unit exceeds 1000 characters, split the text content blocks into multiple text content blocks according to complete sentence boundaries; Divide the pure text area and the table area based on the table border, remove empty cells and invalid placeholder cells in the table, restore the merged cells and multi-level table headers of the semi-structured tables in the table area to a standard row-column structure through cell coordinate mapping, supplement missing row and column indexes, and extract table header level information, cell values and table titles to obtain table content blocks, and label metadata for the table content blocks; Remove text content blocks and table content blocks with a character recognition error rate and a row-column structure missing rate exceeding 5% based on the metadata to obtain a text content block set and a table content block set, and the metadata comprises a document identifier, a content block identifier, a text starting position, a character length, a belonging chapter, a table row and column number, a table header type, and a data dimension.

3. The retrieval augmented question answering method for mixed modality document content of claim 1, wherein, The method for obtaining the repeated semantic sub-strings comprises: BWT transformation is performed on the string based on the text content block to obtain a circular shift sequence of the string, the circular shift sequence is sorted in lexicographical order, and the last column of characters of the sorted sequence is extracted to obtain a BWT transformed sequence, the original string position index in the transformation process is recorded, a suffix array is constructed based on the BWT transformed sequence, and a longest common prefix array is calculated, and the repetition degree of the repeated semantic substring is obtained based on the longest common prefix array, and the repetition degree calculation formula is: ； wherein is the repetition degree of the repeated semantic substring, is the length of the longest common prefix of the first , prefixes in the suffix array, is the semantic weight of the substring, obtained by pre-training industry word vector model, is the total length of the text content block, is the distance attenuation coefficient, obtained by linear regression fitting, is the character offset difference of the substring in the text content block. If the repetition degree is greater than or equal to 0.2, it is determined that the repeated semantic substring is valid, repeated characters are removed, and the position, repetition frequency and semantic category of the repeated semantic substring are marked to obtain a list of repeated semantic substrings.

4. The retrieval augmented question answering method for mixed modality document content of claim 1, wherein, The method for obtaining the text block includes: The text segments containing the same core repeated semantic substring are aggregated into a first-level block based on the text content block, and the aggregation weight calculation formula is: ； in For the first The semantic aggregation weights of each level of text chunks are used as input items. No. The degree of repetition of repeated semantic substrings. This represents the total number of repeating semantic substrings. The hierarchical decay coefficient is obtained by fitting the semantic similarity between parent and child blocks. For the first The number of levels in each block, For the first The character length of a repeating semantic substring. For the first The total character length of each level of text block; The core repeated substring is marked according to the block, the semantic theme of the block is marked, the first-level text block is split according to the affiliation of the repeated semantic substring, if there is no affiliation, the first-level text block is split according to the complete semantic sentence boundary, the hierarchical index of the hierarchical text block is marked, the repeated semantic substring list and the text content block metadata are associated, and a structured hierarchical text block set is obtained.

5. The retrieval augmented question answering method for mixed modality document content of claim 1, wherein, The method for obtaining the table information block includes: Whether it is a standard horizontal table format is determined based on the row and column types of the cells in the table content block, if a serial number column is contained, the column is removed, and the average semantic similarity of the cell contents between data rows is calculated, the row average semantic similarity is taken as the row semantic similarity, the average semantic similarity of the cell contents between the data columns excluding the title row and the first column is taken as the column semantic similarity, and the semantic similarity calculation formula is: ； ； wherein is a semantic similarity of a row, is a cell with a semantic similarity of a row, is a cell content of a row is a cell content of a row is a cell content of a row is a cell content of a row is a cell content of a row is a cell content of a row is a cell with a semantic similarity of a row, is a cell content of a row is a cell content of a row is a cell content of a row is a cell content of a row is a cell content of a row is a cell content of a row is a number of rows of a table, is a number of columns of a table, is a column semantic similarity; If the row semantic similarity is less than a preset consistency threshold and the column semantic similarity is greater than or equal to the preset consistency threshold, the table is a non-standard horizontal table and is transposed, and whether the transposed standard format horizontal table contains a row header column is determined by a row header column recognition algorithm, and the row header column recognition algorithm calculation formula is: ； ； wherein is a row header confidence, is a cell content and a first column name between the semantic similarity function, is a column header of a second column, is a very small positive number, is an average row header confidence of rows in the first column; The row header confidence of the cell contents in the first column and the column headers is calculated, if the average row header confidence of the rows in the first column of the entire table is greater than a preset confidence threshold, the first column is determined to be a row header column, the table cells are traversed, if the row header column is recognized, the text content of the row header and the column header is added to the cell content, if the row header column is not recognized, only the text content of the column header is added, and the table information block is obtained.

6. The retrieval augmented question answering method for mixed modality document content of claim 1, wherein, The method for obtaining the vector index library includes: The hierarchical text block is semantically feature encoded by a pre-trained industry word vector model based on the hierarchical text block to obtain a fixed-dimension semantic embedding vector, the table information block is semantically feature encoded by a table pre-training model, and a FAISS index is generated according to the semantic feature encoding; A block identity is generated by a hash algorithm according to the metadata of the text block and the table information block, the semantic embedding vector and the metadata hash value are taken as a key-value pair, the key-value pair is used as a leaf node of a Merkle tree, and an intermediate node hash is generated, and the aggregation formula of the intermediate node hash is: ； wherein is an aggregated hash value of an intermediate node in the Merkle tree, is a hash algorithm, is a hash value of a left child node, is a hash value of a right child node, is a modal type identifier, distinguishing between a text chunk and a table information chunk; The encoding embedding vector of the semantic feature encoding and the Merkle tree are stored in the index library to obtain the vector index library.

7. The retrieval augmented question answering method for mixed modality document content of claim 1, wherein, The method for obtaining the similarity search information block comprises: inputting the question text of the user question into a pre-trained industry word vector model to obtain pure text semantic features, and if the question text contains a table intent keyword, converting the question into a table semantic query formula, inputting the query formula into a lightweight table encoder to obtain a table semantic perception vector, and fusing the double-branch vectors through an intent weight formula to obtain a query vector, the intent weight formula being: ； wherein is a query vector, is a text intent weight, is a pure text semantic feature, a text intent probability is dynamically adjusted according to the pure text semantic feature, and the adjustment range is 0.3-0.7, is L2 normalization, is a table semantic perception vector, is a table intent score, which is obtained by matching degrees of table intent keywords. calculating the cosine similarity of the query vector and the encoding embedding vector based on a vector index library, and extracting a candidate information block through nearest neighbor search according to the cosine similarity, the cosine similarity formula being: ； wherein is a query vector is a cosine similarity with an information block is an encoding embedding vector of an information block is a correlation strength of a hash and semantic vector in the information block calculating the Merkle path confidence of the candidate information block, the Merkle path confidence calculation formula being: ； wherein is the information block Merkle path confidence, is the hash consistency score of the Merkle path of the information block, being 1 for a perfect match, is the path level length, is the decay coefficient, is the search result confidence; taking the candidate information block with a Merkle path confidence greater than 0.85 as the similarity search information block.

8. The retrieval augmented question answering method for mixed modality document content of claim 1, wherein, The method for obtaining the context comprises: constructing a text input sequence of the user question, the text block content and the block metadata based on the similarity search information block, and inputting the text input sequence into a BERT-TAPAS cross encoder to output a text-based matching score, and correcting the text-based matching score through a text block weight formula, the text block weight formula being: ； wherein is a text revision score, is a repetition degree of the th repeated semantic substring in the text chunk, is a number of valid repeated substrings, is a hierarchical decay factor; constructing a table input sequence of the user question, the row header, the column header, the cell and the metadata based on the similarity search information block, and outputting a table-based matching score through the BERT-TAPAS cross encoder, multiplying the column semantic similarity of the key column by the table-based matching score to obtain a table correction score, and performing hybrid correlation rearrangement on the text correction score and the table correction score of the similarity search information block, the scoring formula of the hybrid correlation rearrangement being: ； in To score for rearrangement, The weighting coefficients are obtained based on the proportion of modal data. query vector With information block cosine similarity, The corrected score for the corresponding modal data; taking the optimal matching information block as the context of the query vector according to the rearrangement score.

9. The retrieval augmented question answering method for mixed modality document content of claim 1, wherein, The method for obtaining the answer comprises: locating to the corresponding leaf node in the Merkle tree according to the metadata hash value based on the context, and backtracking to the root node of the Merkle tree in sequence according to the leaf node to obtain the hash value of the intermediate node in the backtracking path, the block identifier and the modal type identifier associated with the node, and generating a Merkle path; taking the encoding embedding vector of the context as a retrieval enhancement generation chain semantic layer, taking the hash sequence of the Merkle path and the metadata as a retrieval enhancement generation chain verification layer, and taking the document identifier corresponding to the context information block as a retrieval enhancement generation chain association layer to obtain a retrieval enhancement generation chain; constructing a structured prompt template containing a specific instruction description, a context placeholder and a question placeholder according to the context, inputting the prompt template containing the instruction and the placeholder, the Merkle path and the retrieval enhancement generation chain into a preset large language model, and decoding to output an answer containing a reference mark and a related information block source file list.

10. A retrieval-augmented question-answering system for mixed modality document content, configured to perform the retrieval-augmented question-answering method for mixed modality document content according to any one of claims 1-9. The system comprises: a data acquisition module configured to acquire a mixed modal document of a target industry, and preprocess the mixed modal document to obtain text content blocks and table content blocks, wherein the mixed modal document comprises unstructured text and semi-structured tables; The data division module is configured to perform BWT transformation on the string of the text content block, identify a repeated semantic substring through a suffix array according to a transformation sequence, extract a row-column structure from the table content block, and obtain hierarchical text blocks and table information blocks; The index library construction module is configured to perform semantic feature coding based on the text blocks and the table information blocks, associate the coded embedding vectors with metadata hash values of the blocks, generate a Merkle tree according to the hash values, and obtain a vector index library; The retrieval module is configured to obtain a user question, encode the question into a query vector through a double encoder, perform similarity search in the vector index library using the query vector, match and score information blocks obtained through the similarity search through a cross encoder, and take an optimal matching information block as a context of the query vector; The answer generation module is configured to generate a Merkle path corresponding to the context and a retrieval enhancement generation chain according to the Merkle tree, take the context as a prompt template, input the prompt template, the Merkle path and the retrieval enhancement generation chain into a preset large language model, and decode and output an answer containing a reference mark and a list of source files of related information blocks.

Images