A multimodal retrieval enhancement method and system for government and enterprise institutional documents

Abstract

This invention proposes a multimodal retrieval enhancement method and system for government and enterprise policy documents. The method includes: performing OCR recognition of the table of contents of the document to be processed, parsing the title hierarchy and performing preliminary segmentation; converting the segmented parts to Markdown using OCR and performing post-processing, slicing by title and integrating attachment information; segmenting the main text according to a length threshold and setting overlap to generate fine-grained and coarse-grained text slices. An improved semantic coding model is used to vectorize and store the slices in a vector database; the user query is encoded and candidate slices are retrieved, and after reordering, a target slice set is obtained; the target slices and the query are combined to construct a prompt input multimodal model, generating question-and-answer results and evaluating them. This invention extracts the document structure through deep post-processing and integrates attachment content to improve the quality of the answer; intelligent segmentation and parallel processing are used for long documents to improve efficiency; the coding model is fine-tuned and subjected to contrastive learning to enhance recognition capabilities, and modal verb weighting improves comprehension accuracy.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and information retrieval technology, and in particular to a multimodal retrieval enhancement method and system for government and enterprise policy documents. Background Technology

[0002] Government and enterprise policy documents are characterized by their diverse categories, length, and frequent updates. Traditional processing methods struggle to achieve unified, efficient management and rapid retrieval, leading to inconvenience in accessing policy content. Existing document slicing methods are mostly based on fixed lengths or delimiters. Although they support recursive slicing and overlapping fragment settings, they still fail to fully guarantee the semantic coherence of the context. For policy documents with hierarchical structures, existing methods easily disrupt the relationship between titles and body text, thus failing to support accurate retrieval based on a combination of hierarchical structure and text content. In question-and-answer application scenarios, existing systems struggle to adapt to users' varying levels of query granularity and can only return text content, failing to meet the query needs for multimodal information such as tables, images, and processes in business scenarios. Furthermore, the serial processing of documents is inefficient. Moreover, directly applying a general semantic model to encode policy text fails to perform semantic parsing of abbreviations or contractions, and does not distinguish the difference in intensity between compliance terms such as "should" and "ought," resulting in inaccurate semantic representation and insufficient retrieval matching precision, making it difficult to meet the high rigor requirements of government and enterprise policy documents. To address the aforementioned shortcomings, this invention proposes a multimodal retrieval enhancement method and system for government and enterprise policy documents. This method achieves multi-granularity question answering and multimodal content return, improves processing efficiency through multi-threading technology, and enhances the accuracy of semantic representation of policy texts, thereby improving the practicality and reliability of retrieval and question answering. Summary of the Invention

[0003] This invention aims to at least address the technical problems existing in the prior art, and in particular, innovatively proposes a multimodal retrieval enhancement method and system for government and enterprise institutional documents.

[0004] To achieve the above-mentioned objectives of this invention, this invention provides a multimodal retrieval enhancement method for government and enterprise executive policy documents, the method comprising: S1. Perform OCR recognition on the table of contents of the policy document to be processed, parse the hierarchical structure of the main document title, and perform preliminary segmentation of the document according to the page number range corresponding to the main title; perform OCR recognition on the segmented document and convert it into Markdown format text, and perform post-processing on the Markdown format text; S2. Slice the post-processed Markdown text according to heading level to obtain multiple text slices containing main headings, first-level headings, second-level headings, and body text. Integrate the corresponding images or tables in the attachments into the body text references. Divide the multiple text slices of the body text according to the set length threshold and set the overlap interval to generate text slices for encoding. The text slices for encoding contain fine-grained heading and body text combined slices and coarse-grained document frame slices. S3. The text slices are vectorized using an improved semantic coding model, and the coding results and corresponding metadata are stored in a vector database. S4. The user query is vector-encoded, and a similarity search is performed in the vector database to obtain candidate text slices; the candidate text slices are reordered to obtain a target text slice set. S5. Combine the target text slices with the user query to construct prompt information, input the multimodal large model to generate question and answer results, wherein the question and answer results include text and corresponding image or table information; S6. Evaluate the question-and-answer results based on preset evaluation indicators, which include performance indicators for the retrieval stage and effectiveness indicators for the generation stage.

[0005] As an optional embodiment of the present invention, the post-processing in step S1 may include: Extract visual features of text blocks, and perform heading level completion on the Markdown formatted text based on the visual features of text blocks; The heading level of the Markdown formatted text is corrected based on regular expression rules; The Markdown formatted text is processed for noise removal and typo correction.

[0006] As an optional embodiment of the present invention, the step of completing the title hierarchy based on the visual features of text blocks may include determining the title hierarchy relationship by using font size, character weight, line spacing and spatial position.

[0007] As an optional embodiment of the present invention, the improved semantic coding model in step S3 may include: Obtain a hybrid dataset consisting of four tasks: retrieval, semantic similarity, classification, and clustering. The dataset is specifically designed for the characteristics of government and enterprise documents, including query document pairs constructed using abbreviations specific to government and enterprise, modal verbs related to legal compliance, and long document paragraphs. Fine-tuning was performed through a two-stage training process. The first stage involved basic capability training, which applied a hybrid retrieval approach to the text slices to be encoded, incorporating dense retrieval, lexical retrieval, and multi-vector retrieval, and optimizing the model using infoNCE loss. The second stage involved joint fine-tuning for government and enterprise scenarios, which introduced a self-knowledge distillation mechanism. After generating teacher signals, weighted contrast loss, no-distillation loss, and distillation loss were calculated. Finally, the average of these three losses was used as the final loss to fine-tune the model, resulting in an improved semantic encoding model adapted to government and enterprise policy document scenarios.

[0008] As an optional embodiment of the present invention, the vector database may be a Milvus database, and an index structure including text vectors, title information, path information and time information may be constructed.

[0009] As an optional embodiment of the present invention, the index structure may be an HNSW index.

[0010] As an optional embodiment of the present invention, steps S1, S2 and S3 may be executed in parallel.

[0011] On the other hand, the present invention also provides a multimodal retrieval enhancement system for government and enterprise executive documents, the system comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the multimodal retrieval enhancement method for government and enterprise institutional documents when executing the executable instructions.

[0012] The beneficial effects of this invention are as follows: By performing deep post-processing on Markdown files, this invention can accurately extract heading levels and text content, thereby optimizing the document structure, ensuring the accuracy of hierarchical relationships, and providing high-quality semantic structure support for subsequent retrieval and question-answering tasks. Furthermore, by effectively integrating attachment content into the corresponding positions in the original text, the model can fully utilize the contextual information of the attachments when generating answers, significantly improving the accuracy and completeness of the answers and avoiding answer bias caused by information fragmentation. Simultaneously, by constructing a complete document framework, the system supports the understanding and presentation of the overall document structure; when users ask coarse-grained questions such as "What is the document structure?" or "Which chapters does it contain?", the system can quickly return a clear document outline, thereby enhancing the user experience. For long documents in government and enterprise categories, this invention adopts an intelligent segmentation strategy based on a directory structure, performing OCR recognition, post-processing, and data entry operations on each segment in parallel, significantly improving the processing efficiency of a single policy document and effectively solving the performance bottleneck in long document processing. Furthermore, the encoding model is fine-tuned for domain adaptation, and a contrastive learning mechanism is introduced into the loss function to enhance the model's ability to recognize specific expressions in areas such as abbreviations and acronyms. Meanwhile, differentiated weights are assigned to modal verbs such as "must," "should," and "may," enabling the model to accurately distinguish semantic differences at different levels of regulations, thereby improving the accuracy of understanding institutional texts.

[0013] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0014] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating a multimodal retrieval enhancement method for government and enterprise executive documents provided in Embodiment 1 of this application; Figure 2 This is a logical block diagram of a multimodal retrieval enhancement system for government and enterprise executive documents provided in Embodiment 2 of this application; Figure 3 This is a flowchart of the coding model fine-tuning training for a multimodal retrieval enhancement system for government and enterprise institutional documents, provided according to Embodiment 2 of this application. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0016] Example 1 like Figure 1 As shown, a multimodal retrieval enhancement method for government and enterprise institutional documents is proposed, the method comprising: S1. Perform OCR recognition on the table of contents of the policy document to be processed, parse the hierarchical structure of the main document title, and perform preliminary segmentation of the document according to the page number range corresponding to the main title; perform OCR recognition on the segmented document and convert it into Markdown format text, and perform post-processing on the Markdown format text; The post-processing described in step S1 includes: Extract visual features of text blocks, and perform heading level completion on the Markdown formatted text based on the visual features of text blocks; Regarding heading level completion, it's important to note that due to the common issues of inconsistent formatting, nested headings across pages, and the omission of some subheadings in unified heading tables, relying solely on the OCR results of the table of contents page is insufficient to fully reconstruct all heading levels. This method introduces visual features of text blocks as an auxiliary basis, extracting five categories of visual features for each text block: font size, font weight, pre-paragraph spacing, post-paragraph spacing, and alignment. These features are input into a pre-trained lightweight heading level classifier, which determines whether the current text block is a heading and to which heading level it belongs, thus completing the heading level correction for the initial parsing results. After heading level completion, document slicing strictly adheres to the actual heading level division, avoiding incorrectly segmenting cross-page headings of the same level into body text and ensuring no subheadings embedded within the body text are missed. This provides a clearer structured slicing foundation for subsequent encoding and retrieval. Simultaneously, the tree_path metadata of document slices in the knowledge base more accurately reflects the document's logical structure, providing users with clear guidance on the content source when returning the final results.

[0017] The heading level of the Markdown formatted text is corrected based on regular expression rules; Regarding the correction of title hierarchy, it should be noted that many titles in government and enterprise system documents have standardized hierarchical numbering, such as "Chapter 1", "Section 1", "1.", "1.1", "I.", "(I)", etc. This method pre-constructs multi-level regular expression matching rules for different level numbering formats, which can batch correct the hierarchical misalignment problem that still exists in the output of the hierarchical classifier: for example, correcting the first-level numbered title "1.×××" which is misclassified as a second-level title to the corresponding first-level title, reclassifying embedded secondary titles with non-standard numbering formats to the correct level, and batch correcting the numbering distortion problem caused by OCR recognition, such as correcting "1," which is misidentified as a full-width comma back to the original full-width dot "1.", and completing the hierarchical realignment after unifying the numbering format.

[0018] This batch correction method based on regular expressions eliminates the need for manual adjustments segment by segment. It can correct all title numbering errors in the entire document at once, adapting to the common numbered title specifications for government and enterprise documents and solving formatting errors caused by OCR recognition in batches, further improving the accuracy of title hierarchy classification.

[0019] The Markdown formatted text is processed for noise removal and typo correction.

[0020] Regarding noise removal and typo correction, it should be noted that government and enterprise policy documents are mostly scanned documents converted to digital format. OCR recognition is highly susceptible to introducing various types of noise, including repeated headers and footers, redundant page numbers, background watermark text, garbled characters caused by line break errors, and invisible special control characters. It can also lead to misidentification of similar-looking characters and homophones. This method first removes various invalid noises in batches through rule matching: it locates and deletes repeated headers, footers, and page numbers at fixed positions on each page, then filters out consecutive redundant blank lines and special control characters, and corrects typo errors caused by OCR pagination. For typo correction, this method uses a pre-built dictionary of government and enterprise policy terminology, combined with the context-aware capabilities of a large language model, to perform corrections. It prioritizes the correction of high-frequency proprietary terms in government and enterprise documents such as "must," "responsibility," "approval," and "compliance," avoiding errors in general correction logic that could misinterpret domain-specific vocabulary. This step effectively eliminates various interferences introduced during the document digitization process, significantly improves the accuracy of subsequent text encoding, and prevents noisy content from interfering with retrieval and ranking results after being entered into the database, thus ensuring the overall quality of the knowledge base content from the source.

[0021] It should also be elaborated in detail in step S1 that each text block output by OCR recognition will carry visual attributes such as coordinate position, font size, line spacing, and block spacing. We extract these attributes as the visual features of the text block and train a lightweight hierarchical classifier to distinguish whether the text block belongs to a first-level title, a second-level title, or the main text based on the visual features. For text blocks that lack hierarchical markers in the original document or lose hierarchical markers during OCR recognition, the corresponding hierarchical Markdown title markers are automatically completed to solve the problems of missing and inconsistent hierarchical markers commonly existing in government and enterprise long-form documents.

[0022] Regarding the regular batch correction of title levels, government and enterprise system documents generally use multi-level manual numbering, and there are situations where multiple numbering formats such as "1.", "1.1", "一、", "(一)" are mixed and misused. We preset regular matching rules that cover the general numbering rules of domestic government and enterprise official documents to batch identify all numbers in the text of the entire document, and automatically correct the misplaced and incorrect Markdown title level markers according to the logical levels of the numbers, completing the level correction of the entire document in batches without manual adjustment one by one, greatly improving the processing efficiency of long documents.

[0023] Regarding noise processing, during the OCR recognition of scanned documents, it is easy to introduce redundant whitespace, duplicate page numbers, redundant information in headers and footers, and misspelled characters and garbled codes caused by scanning artifacts. We first batch remove headers and footers, duplicate page numbers, and redundant whitespace through rule filtering, and then combine a Chinese error correction model fine-tuned for the field of government and enterprise official documents to correct形近字 (similar-shaped characters) and homophone errors caused by scanning recognition based on the context, and at the same time filter out unrecognizable garbled code segments, finally obtaining a clean and standardized Markdown format text.

[0024] S2. Slice the post-processed Markdown format text according to the title level, obtaining multiple text slices including main titles, first-level titles, second-level titles, and main text content, and integrating the image or table information corresponding to the attachments in the document into the in-text citation positions; segment the multiple text slices of the main text according to the set length threshold and set an overlapping interval to generate text slices for encoding, where the text slices for encoding include fine-grained title and main text combined slices and coarse-grained document framework slices; In step S2, it's important to explain in detail that this title-level slicing method perfectly aligns with the structured logic of the government and enterprise leadership documents. Compared to the traditional RAG fixed-character-length slicing method, it doesn't disrupt the original title-content correspondence and avoids splitting content on the same topic into multiple unrelated slices. For images and tables included in the document, this method integrates and stores the OCR-recognized text of the images, the structured parsed text of the tables, and the citation descriptions in the main text, while retaining the original image / table access addresses. This provides complete multimodal information for text encoding and facilitates the output of the original formatted image and table content according to the rules when generating the final response.

[0025] The purpose of setting a length threshold for secondary segmentation is to prevent individual slices from exceeding the encoder's maximum input length while ensuring semantic concentration in the encoding. Setting overlapping intervals compensates for semantic breaks that may result from fine-grained segmentation, ensuring that continuous semantics across segmentation boundaries are fully preserved in at least one slice, preventing the omission of key information during retrieval. Designing both coarse and fine-grained slices is to adapt to different types of user queries: when users ask broad framework questions, such as "What are the components of our unit's safety production management system?", coarse-grained document framework slices are prioritized, directly returning the complete system structure; when users ask specific detailed questions, such as "What is the approval process for employee leave requests?", fine-grained title-body combination slices accurately match the question, reducing redundant content in the search results. This balances retrieval efficiency and accuracy across different query scenarios, solving the problems of semantic fragmentation and low matching accuracy in long structured documents caused by traditional fixed-length slices.

[0026] S3. The text slices are vectorized using an improved semantic coding model, and the coding results and corresponding metadata are stored in a vector database. The improved semantic coding model in step S3 includes: Obtain a hybrid dataset consisting of four tasks: retrieval, semantic similarity, classification, and clustering. The dataset is specifically designed for the characteristics of government and enterprise documents, including query document pairs constructed using abbreviations specific to government and enterprise, modal verbs related to legal compliance, and long document paragraphs. Fine-tuning was performed through a two-stage training process. The first stage involved basic capability training, which applied a hybrid retrieval approach to the text slices to be encoded, incorporating dense retrieval, lexical retrieval, and multi-vector retrieval, and optimizing the model using infoNCE loss. The second stage involved joint fine-tuning for government and enterprise scenarios, which introduced a self-knowledge distillation mechanism. After generating teacher signals, weighted contrast loss, no-distillation loss, and distillation loss were calculated. Finally, the average of these three losses was used as the final loss to fine-tune the model, resulting in an improved semantic encoding model adapted to government and enterprise policy document scenarios.

[0027] In step S3, it is necessary to explain in detail that this step, based on the Top N candidate text slices output from the initial retrieval, combines the structured hierarchical attributes unique to government and enterprise long documents to complete the secondary sorting optimization: First, the user query and candidate slices are input into the re-sorting model fine-tuned for the government and enterprise domain, and the basic semantic matching score is calculated; then, a hierarchical weighting rule is introduced: if the user's question is a framework question, extra points are added to the matched coarse-grained document framework slices; if the user's question is a specific detail question, extra points are added to the matched corresponding level fine-grained slices; at the same time, for continuous semantic slices from the same parent title, a continuity confidence score is added to avoid the problem of low ranking scores caused by semantic fragmentation of a single slice. Finally, the candidate slices are sorted in descending order according to the weighted total score, and the Top K slices are concatenated and sent to the generation module to obtain the final response result.

[0028] Compared to traditional re-ranking schemes that rely solely on semantic similarity, this method incorporates the structured logical characteristics of government and enterprise policy documents. It can adaptively adjust the ranking weights of different granularity slices according to the query type, solving the problem of high semantic matching scores but mismatch between slice granularity and query requirements. This further improves the accuracy of search results and adapts to the ranking requirements of two different scenarios: framework-type queries and fine-grained rule queries.

[0029] S4. The user query is vector-encoded, and a similarity search is performed in the vector database to obtain candidate text slices; the candidate text slices are reordered to obtain a target text slice set. In step S4, it's important to explain in detail that the re-ranking process combines the structured hierarchical attributes of government and enterprise policy documents with multi-granularity slice features, rather than relying solely on semantic similarity ranking. The specific process is as follows: First, the user query and all candidate text slices are input into a re-ranking model fine-tuned for the government and enterprise domain, calculating a basic semantic matching score. Then, an adaptive hierarchical weighting rule is introduced to initially identify the type of user query. If it's identified as a framework question, such as querying the module divisions included in a certain type of policy, additional weight is added to the coarse-grained document framework slices in the candidates. If it's identified as a specific detail question, such as querying the processing requirements for a certain matter, additional weight is added to the corresponding fine-grained text slices. Finally, all candidate slices are sorted in descending order according to the weighted total score. The top K slices are concatenated to form the target text slice set, which is then fed into the subsequent large language model generation module to generate the final response.

[0030] Compared to traditional RAG and GraphRAG re-ranking schemes that rely solely on semantic similarity, this re-ranking method fully utilizes the structured information obtained during the processing of long-term government and enterprise policy documents. It can adaptively adjust the ranking priority of different granularity slices according to the type of user query, solving the problem often encountered in traditional methods where "semantic matching scores are high but the slice granularity does not match the query requirements." For example, if a user asks about a specific approval process, the method may incorrectly recall the entire framework slice of the policy. This further improves the accuracy of the search results sent to the generation module. At the same time, it takes into account the ranking requirements of two different scenarios: framework-type queries and fine-grained rule queries. As can be seen from the comparative experimental results above, this method outperforms the comparative methods in both contextual recall and contextual precision, two core retrieval indicators. The fidelity, relevance, and overall correctness of the final output answer are also significantly higher than those of the RAG and GraphRAG schemes. In addition, the overall document processing time is shorter, making it more suitable for scenarios involving the storage and retrieval of a large number of long-term government and enterprise policy documents.

[0031] S5. Combine the target text slices with the user query to construct prompt information, input the multimodal large model to generate question and answer results, wherein the question and answer results include text and corresponding image or table information; In step S5, it is necessary to explain in detail that this method organizes the recalled content according to the hierarchical logical order of the target slices, and concatenates the user query with the structured recalled slices into prompt words that meet the input requirements of the large model: the prompt begins by clearly stating the user's query needs, and then puts all the target slice content into the order according to the original logic of the document. For the image and table information associated with the slices, on the one hand, the OCR-recognized text and structured parsing content are embedded in the corresponding prompt positions to provide complete semantic information for the generation of the large model. On the other hand, the placeholders of the original resources are retained so that they can be replaced with the original format images and tables that can be directly displayed in the final output.

[0032] To address the compliance requirements of Q&A in government and enterprise scenarios, this method also incorporates specific constraint rules into the prompts: requiring the large model to strictly generate answers based on the recalled document content, and prohibiting the fabrication or concoction of rules and requirements not explicitly mentioned in the regulations; if the user's question is not effectively covered by the recalled content, the large model is required to clearly inform the user that "no relevant regulations were found," thereby constraining the generation behavior of the large model from the prompt level and avoiding the generation of incorrect or misleading answers.

[0033] Compared to traditional plain text retrieval enhancement generation solutions, this method's multimodal fusion mechanism can fully preserve key information such as processes, standards, and division of responsibilities contained in images and tables in institutional documents. It does not lose structured and visualized content due to plain text extraction. When users query specific procedures, the flowcharts in the original regulations can be directly displayed in the response, which is clearer and more intuitive than plain text descriptions. This can effectively improve the experience and efficiency of government and enterprise users in querying institutional information, while ensuring the compliance, accuracy, and fidelity of the generated answers.

[0034] S6. Evaluate the question-and-answer results based on preset evaluation indicators, which include performance indicators for the retrieval stage and effectiveness indicators for the generation stage.

[0035] In step S6, it is necessary to explain in detail that this step, targeting the core needs of the Q&A scenario for government and enterprise executive system documents, sets up multi-dimensional hierarchical evaluation indicators, divided into two categories: performance indicators in the retrieval stage and effectiveness indicators in the generation stage. The retrieval stage uses two core indicators: contextual recall and contextual precision, to quantitatively evaluate the coverage and filtering accuracy of the target text slice set obtained after reordering, which corresponds to the core performance of the retrieval process. The generation stage, combined with the compliance requirements of the government and enterprise scenario, sets up four evaluation dimensions: compliance, fidelity, relevance, and usability. The compliance indicator is used to detect whether the generated content contains illegal expressions or misleading content that violates system requirements. The fidelity indicator is used to verify whether the generated content is entirely derived from the recalled target slices and whether there are any fabricated or fabricated contents not mentioned in the system. The relevance indicator is used to evaluate the degree of matching between the generated answer and the user's query, filtering out redundant content that is irrelevant to the question. The usability indicator is used to evaluate the clarity and logical rationality of the generated answer and whether it can effectively solve the user's question needs.

[0036] When the evaluation results do not meet the preset qualified threshold, the system will automatically trigger the backtracking optimization mechanism: automatically adjust the hierarchical weighted threshold of the re-sorting process, re-select the target slice set from the candidate slices, and then send it to the generation module to regenerate the question and answer results. If the results still fail to meet the standard after multiple iterations, they will be automatically transferred to manual review and processing. At the same time, the question and the corresponding processing record will be stored in the optimization sample library for subsequent iterations to fine-tune the semantic encoding model and re-sorting model, and continuously improve the question and answer performance and accuracy of the system in government and enterprise scenarios.

[0037] The preset qualification thresholds (preset evaluation indicators) for the retrieval stage are: context recall rate of not less than 85% and context precision of not less than 80%. If either of the two indicators fails to meet the threshold requirements, the retrieval stage is deemed unqualified.

[0038] The preset pass / fail thresholds for each indicator during the generation phase are as follows: Compliance indicators: The generated content must not contain any illegal statements or misleading content that violates the disclosure system. A perfect score is awarded for passing the test, and a score of zero is awarded for failing the test. The passing threshold is a perfect score. If the compliance standard is not met, the entire question and answer result will be deemed unqualified. Loyalty index: The semantic overlap between the generated content and the recalled target slice is calculated by the semantic alignment model. At the same time, the large model is used to verify whether there is any fabrication or fabrication of content not mentioned in the system. The qualified threshold is that the semantic overlap is not less than 90% and there is no fabricated content. Relevance metrics: The semantic matching degree between generated content and user query is calculated by cosine similarity. The default threshold is a matching similarity of not less than 0.75. Usability metrics: The scoring method combines automatic scoring from a large model with periodic manual verification through sampling. The maximum score is 100 points, and the pass threshold is preset to be no less than 80 points.

[0039] For high-risk core business issues involving the division of powers and responsibilities, administrative approvals, and compliance requirements in government and enterprise scenarios, this embodiment will automatically raise the qualification threshold of the corresponding indicators by 5% to further improve the accuracy and compliance of the results. For general navigation and common-sense policy queries, the qualification threshold can be appropriately lowered to improve evaluation and response efficiency while meeting usage requirements. Only when all indicators meet the corresponding qualification threshold requirements will the question and answer result be deemed qualified; otherwise, the backtracking optimization mechanism described above will be triggered.

[0040] As an optional embodiment of the present invention, the step of completing the title hierarchy based on the visual features of text blocks may include determining the title hierarchy relationship by using font size, character weight, line spacing and spatial position.

[0041] It's important to note that this visual feature-based approach aligns perfectly with the common practices of manually formatting government and enterprise policy documents. Whether it's an original electronic document or a document transcribed from a scan, the formatting stage naturally distinguishes different levels of headings and body text through font size, font weight, line spacing, and block position. The lack of explicit hierarchical markers doesn't prevent the loss of crucial clues for hierarchical differentiation. Compared to methods relying solely on semantic textual content to determine heading levels, visual features provide a more intuitive and direct basis for judgment, resulting in higher accuracy. Even for documents with manual annotation errors or inconsistent numbering, preliminary hierarchical classifications can be obtained through visual features. Combined with subsequent regular expression-based batch correction steps, this ultimately improves the accuracy of hierarchical structuring in long-form government and enterprise policy documents, significantly reducing the cost of manually organizing long, structured documents.

[0042] As an optional embodiment of the present invention, the vector database may be a Milvus database, and an index structure including text vectors, title information, path information and time information may be constructed.

[0043] It's worth noting that Milvus natively supports efficient near-nearest neighbor retrieval for over a million vectors, meeting the retrieval needs of government and enterprise units with thousands or tens of thousands of long policy documents and hundreds of thousands of slices. Retrieval latency can be stably controlled within milliseconds, adapting to the performance requirements of high-frequency question-and-answer queries. The multi-field composite index structure designed in this solution fully leverages the combined advantages of vector retrieval and scalar filtering: time information indexing quickly filters out obsolete and invalid policy slices, prioritizing the retrieval of the latest effective policy content, solving the pain point of traditional retrieval schemes frequently retrieving outdated regulations; title information indexing pre-screens candidate slices containing user query keywords, narrowing the scope of vector retrieval and further improving retrieval efficiency; path information indexing directly locates the original document storage location of the slice, facilitating users to trace and view the complete policy document, meeting the compliance management and record retention needs of government and enterprise units. This composite index design and the multi-granularity slicing and adaptive reordering process of this method form a complete technical synergy, ensuring retrieval efficiency in large-scale scenarios while further improving the accuracy of the final retrieval results, adapting to the full-process needs of long policy document management in government and enterprise units.

[0044] As an optional embodiment of the present invention, the index structure may be an HNSW index.

[0045] It should be noted here that the HNSW index, by constructing a multi-layered hierarchical navigation graph structure, can achieve an excellent balance between retrieval efficiency and recall precision. Compared with other vector index structures such as inverted indexes and FLAT indexes, HNSW can still maintain extremely fast retrieval response speed and high recall precision in vector sets of millions or more, which is fully suitable for the retrieval needs of this solution for large-scale long-term policy documents of government and enterprises.

[0046] Meanwhile, this solution will continuously generate new slice vectors as government and enterprise policy documents are dynamically updated. The excellent dynamic expansion capability of the HNSW index supports the rapid insertion of incremental slice vectors without the need for a complete reconstruction of the entire index. It can adapt to the dynamic management needs of government and enterprise units to continuously add new policies and periodically abolish old policies, further reducing the daily maintenance cost of the vector index and ensuring that the system maintains stable and efficient retrieval performance during long-term operation.

[0047] As an optional embodiment of the present invention, steps S1, S2 and S3 may be executed in parallel.

[0048] It's important to note that the parallel pipeline operation mode can fully utilize the CPU and GPU computing resources of the heterogeneous computing cluster, avoiding the wasted computing power of traditional serial processing modes where GPUs spend long periods waiting for the CPU to complete pre-processing tasks such as document parsing and structured segmentation. In actual batch processing scenarios, the CPU handles CPU-intensive tasks like document structured parsing and slice generation. Sliced ​​documents automatically enter the GPU encoding queue, allowing the GPU to continuously and uninterruptedly perform vectorized encoding tasks, fully leveraging the computing power advantages of the heterogeneous architecture. For scenarios involving government and enterprise units importing thousands of long policy documents, the overall document import processing time is significantly reduced, substantially lowering the time cost of large-scale document import and adapting to the actual business needs of government and enterprise units for batch updates and dynamic maintenance of policy documents.

[0049] Example 2 This embodiment also provides a multimodal retrieval enhancement system for government and enterprise executive documents. This embodiment mainly consists of 6 modules, see reference. Figure 2 and 3 .

[0050] Data processing module Government and enterprise policy documents typically encompass documents from different departments, such as risk and compliance, finance, and technology, but they all exhibit hierarchical characteristics. Let's select a specific type of document, denoted as document '...'. The table of contents is subjected to OCR recognition, and the resulting ordered set of titles is obtained by parsing the table of contents: in, The text for the main heading. This indicates the total number of chapter headings. Calculate the page range covered by each main heading: in, Page number, Total number of pages in the document. Indicates the first The page number starting with the document. Indicates the first End page number of the document.

[0051] Organize the document according to the page range covered by the main heading. Perform preliminary sectioning to obtain The document is processed as follows: For the i-th document, the processing method is: ... The document structure was converted to Markdown using MinerU OCR recognition. Since the initial Markdown still contained some unrecognized headings, redundant text, and misrecognized characters, further post-processing was required. Specifically, the Markdown post-processing included: 1. Visual feature assistance Based on the structured JSON file output by MinerU, visual attribute features (font size, font weight, line spacing, coordinate position, and text block area) of text blocks are extracted to construct heading level determination rules: text blocks in the beginning area of ​​the line (within the Y-axis coordinate threshold), with a font size ≥ 1.5 times that of the body text, a font weight of bold, and a line spacing > 2 times that of the body text are identified as missing headings; combined with the spatial hierarchy of text blocks (e.g., high-feature text blocks below a first-level heading are second-level headings), Markdown hierarchy symbols are mapped to complete the accurate heading level, while filtering out text blocks such as headers and footers that interfere with visual features to reduce the false positive rate.

[0052] 2. Batch editing with regular expressions The document title numbering system is used to construct multi-dimensional regular expression matching rules: by anchoring the beginning of the line, it matches numerical level titles (such as \d+[. / 、], \d+.\d+[. / 、]) and Chinese level titles (such as Chapter 1|2[、.], Chapter

[12] ), and batches Markdown level symbols are added according to the preset mapping relationship (Level 1 #, Level 2 ##). The rules support greedy matching and non-capturing group design to avoid mismatching of similar text in the body text, while being compatible with multiple delimiters (. / 、) to achieve automatic correction of all title levels.

[0053] 3. Noise Control To handle text redundancy, regular expressions were written to batch remove redundant spaces, line breaks, special characters, etc.; and the fuzzywuzzy method was used for global deduplication of paragraphs. The lightweight Autochecker4Chinese method for correcting Chinese typos was employed, and a dictionary of common OCR errors was created and used to replace strings.

[0054] The improved Markdown document will be available soon. Convert to the first Markdown documents Post-processing makes the Markdown document more accurate in recognizing titles and attachments.

[0055] Document slicing module Based on the Markdown document, perform initial segmentation using # to identify the document's first-level headings, second-level headings, and body text. The ordered collection of Markdown documents is as follows: in, This indicates the first line of a Markdown document. A slice, Indicates the first There are Markdowns. The slice; the first The titles of the Markdown documents are: , Indicates the main title. This indicates a first-level heading. This indicates a second-level heading; the main text content. Including text Image URL and table address .

[0056] Since attachments to some policy documents are usually stored at the end of the document, and are mostly tables or images, but contain little text information, they need to be specially processed. For attachments mentioned in the preceding text, the URLs of tables and images from the following text should be embedded in the location of the previously mentioned attachments to ensure that relevant information can be retrieved and to reduce data loss.

[0057] 1. Assembly and combination information The main headings, first-level headings, second-level headings, and body text obtained from the table of contents will be combined according to the following logic: When the The first Markdown document The main text of each slice Not exceeding the specified text length At that time, the text will not be split; if the text length exceeds the limit... When that happens, the text is processed according to the string length. ( ), with an overlap of ( ), and perform text splitting.

[0058] The obtained main text is combined according to the order of the main headings, first-level headings, second-level headings obtained from the table of contents parsing, and the processed main text. This allows users to obtain precise recall when searching for relevant information, i.e., the text form to be encoded. The first Markdown document The slice was modified as follows: These represent the main heading, first-level heading, and second-level heading of the j-th slice in the i-th Markdown document, respectively. 2. Added coarse-grained information.

[0059] To accommodate users' varying levels of question granularity, the main headings identified in the table of contents, along with the first-level and second-level headings under each main heading, are also extracted and used as text slices to be encoded. Similarly, the entire document's title and the main headings from the table of contents are also included in the text slices to be encoded. This indicates the title of the entire article: Encoding and Input Module Text encoding For the text slices to be encoded obtained by the document slicing module mentioned above , The vector was encoded using a fine-tuned BGEM3. The reason for making minor adjustments to BGEM3 is that some abbreviations in government and enterprise documents were not trained, and the meaning of these abbreviations is unknown; the strength of terms related to law or compliance was not differentiated, such as "should" and "ought to".

[0060] BGEM3 Fine-tuning Path Reference Figure 3 The specific steps are as follows.

[0061] (1) Data preparation layer Data Construction Reference Please refer to Table 1 below for the meaning of each field.

[0062] Table 1 There are four possible values ​​for type: retrieval: a retrieval task, using normal comparative learning and inbatch negative sampling; sts: semantic similarity, only calculates positive pairs, does not add negative pairs in batch; classification: Classification task, without adding inbatch negative examples; Clustering: Clustering task, without adding inbatch negative examples.

[0063] In light of the characteristics of government and enterprise documents, the following types of data need to be supplemented during the data construction phase: Government and Enterprise Terminology Data: Query document pairs are constructed for abbreviations and terms specific to government and enterprises.

[0064] Legal compliance data: Distinguishing samples are constructed for legal modal verbs such as "should / ought to / must / may", and semantic weight differences are clearly defined.

[0065] Long document data: For long texts such as government and enterprise reports and policy documents, construct paragraph-to-text relationship pairs.

[0066] (2) First stage training: basic ability pre-training Based on the FlagEmbedding unified fine-tuning framework, the BAAI / bgem3 model is loaded, employing a hybrid retrieval approach. This stage utilizes large-scale unsupervised data for model pre-training, establishing a foundation for multilingual semantic understanding. BGEM3 employs a Transformer encoder architecture based on XLMRoBERTa, supporting input lengths up to 8192 tokens. The model is pre-trained and initialized using the RetroMAE method to enhance retrieval-oriented semantic encoding capabilities.

[0067] Dense retrieval: Converting query q into a hidden layer based on the text encoder. This allows the [CLS] to be hidden to represent a query: This means normalizing the hidden vector of query q, and the same applies to paragraph p: This means that the hidden vector of paragraph (slice) p is normalized, and the relevance score between the query and the paragraph is the inner product of the two: .

[0068] Vocabulary retrieval: For each term t in the query, the term weight is calculated as follows: , It maps the hidden state to a floating-point matrix, which is used to convert the hidden state of each token into the word importance weight of that token for sparse retrieval. Indicates the first The hidden state vector of each token. Paragraph p is processed similarly, by querying terms that coexist with the paragraph (denoted as...). The relevance score between the query and the paragraph is calculated based on the joint importance of the two elements. . Both represent the importance of a word in the query, and are the word importance scores of each token in the query.

[0069] Multi-vector retrieval: embeds information from the entire output for query and paragraph representation. , , It is a learnable projection matrix that uses post-interactive computation to calculate a fine-grained relevance score: , where N and M represent the length of the query and the paragraph, respectively.

[0070] The loss function used in the first stage is the InfoNCE comparative loss: , This represents the positive and negative samples for query q. for Any one of them, This is the temperature coefficient.

[0071] (3) Second phase training: joint fine-tuning of government and enterprise scenarios Based on the pre-trained model in the first stage, joint fine-tuning was performed using labeled data from government and enterprise scenarios and synthetic data, and the three retrieval methods were optimized simultaneously through a self-knowledge distillation mechanism.

[0072] The retrieval process employs a hybrid approach. First, each method can retrieve candidate results independently (this step can be skipped for the multi-vector method due to its high cost). Then, results are analyzed based on a comprehensive relevance score. Reorder the final search results: , and The weighting depends on the downstream scenario (recommended weighting for government and enterprise scenarios:). =0.15, =0.5, =0.35).

[0073] To address the potential conflict in training objectives among the three retrieval methods, a self-knowledge distillation mechanism is introduced: Teacher signal generation involves weighted fusion of three retrieval scores to form the teacher signal. Without distillation loss, calculate the InfoNCE loss for each retrieval method separately, and also calculate the loss of the ensemble score: During training =1, =0.1, =1.

[0074] Distillation loss uses the softmax distribution of the teacher signal as the soft label, and the loss function for each retrieval method is... This guides the output distribution for each retrieval method. To address the need for differentiating the semantic strength of legal terms in government and enterprise documents, a weighted comparison loss is added to the distillation loss: in As a term weighting coefficient, higher weights are assigned to mandatory words such as "must" (e.g., ...). =1.5), giving lower weight to suggestive words such as "can" (e.g., =0.5), to enhance the model's ability to perceive the strength of legal semantics.

[0075] The final loss function is: An efficient batch processing strategy was used to train the model, and the trained model was deployed to obtain the fine-tuned BGEM3.

[0076] 4. Knowledge Base Construction Using the vector database Milvus, the vector data schema design is shown in Table 2 below.

[0077] Table 2 The schema design in the vector database includes fields such as chunk_id, title, content, document_title, tree_path, vector, date, and created_time. Among these, to return text after retrieval, the primary key field chunk_id, the vector field vector, and the text slice field content are typically necessary. For fast text filtering, document slice metadata document_title is designed; to specify the paragraph from which a document originates when pushing it to a user, the document slice path tree_path is designed; considering that policy documents may be updated, updates are usually performed on a specific policy, which can be understood as the text corresponding to a main heading in a table of contents. A date field is added to the schema to indicate the date the main heading in the table of contents was published.

[0078] Set the vector database index to HNSW and the similarity metric to IP (inner product).

[0079] The three modules mentioned above are designed to run in parallel to improve computational efficiency.

[0080] Search and sorting module 1. Preliminary search The content of user queries is also vector-encoded using a slightly modified BGEM3, and the resulting vectors are retrieved from the Milvus vector database. The results will be retained.

[0081] 2. Reordering The BGERERANKERM3 algorithm was used to re-rank the data obtained from the initial retrieval and calculate scores to select... The data will be retained.

[0082] Question and answer result generation module The qwen3vl32b large model was adopted. In order to utilize the understanding and reasoning capabilities of the large model and adapt to image and table addresses, the prompt words were designed.

[0083] Results Evaluation Module This invention uses three primary indicators and six secondary indicators to evaluate 200 pre-designed question-and-answer pairs. For a document with a length of 611 characters, the method of this invention is compared with the RAG and GraphRAG methods. The RAG method handles context by segmenting the text into a database based on character length and overlapping character length; GraphRAG encodes the document and stores it in a Neo4j database. The designed entities are main title, first-level headings, second-level headings, and body text, with relationships defined as sub-levels. Both RAG and GraphRAG use unadjusted BGEM3 for encoding and unadjusted BGERERANKERM3 for reordering, using the same set of prompt words. Refer to Table 3 below: Table 3 In the data processing stage, the policy documents are structured through OCR parsing and table of contents segmentation, converted into Markdown format, and supplemented with attachment information. Then, a finely tuned BGE-M3 model is used for encoding before storage in the Milvus vector database, supporting parallel processing to improve efficiency. Next, in the retrieval and ranking stage, user questions are encoded using the same model, and relevant fragments are recalled through vector similarity matching. The BGE model is then used to re-rank and filter highly relevant text. Subsequently, in the question-and-answer generation stage, user questions and selected text are combined to construct prompts, and the qwen3-vl-32b large-scale model is used to generate accurate answers. Finally, in the results evaluation stage, the accuracy, relevance, and compliance of the answers are assessed using multiple indicators, forming a complete closed loop from document storage to answer output.

[0084] The fine-tuning process of the BGE-M3 model is divided into two stages. The first stage is basic capability training, which loads dense retrieval, lexical retrieval, and multi-vector retrieval through a hybrid retrieval method for the text slices to be encoded, and optimizes it using infoNCE loss. The second stage is joint fine-tuning for government and enterprise scenarios, which introduces a self-knowledge distillation mechanism: after generating teacher signals, weighted contrastive loss, no distillation loss, and distillation loss are calculated, and finally the average of the three is used as the final loss to fine-tune the model, resulting in a fine-tuned BGE-M3 model adapted to government and enterprise policy document scenarios.

[0085] Example 3 This embodiment provides a multimodal retrieval enhancement system for government and enterprise policy documents, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement multimodal retrieval enhancement for government and enterprise policy documents when executing the executable instructions. It should be noted that the computer device includes a processor and a memory, and may also include one or more of a multimedia component, an input / output (I / O) interface, and a communication component. The processor controls the overall operation of the computer device, completing all or part of the steps of multimodal retrieval enhancement for government and enterprise policy documents. The memory stores various types of data to support device operation and can be implemented by volatile or non-volatile storage devices or combinations thereof, such as SRAM, EEPROM, etc. The multimedia component includes a screen (such as a touch screen) and an audio component. The audio component has a microphone to receive external audio signals and also includes at least one speaker to output audio signals. The I / O interface provides an interface for the processor and other interface modules (such as a keyboard, mouse, buttons, etc.). The communication component is used for wired or wireless communication between devices. Wireless communication includes WiFi, Bluetooth, etc., and the communication component includes a WiFi module, etc. As a preferred option, the computer device can be implemented using electronic components such as ASICs and DSPs to execute a multimodal retrieval enhancement method for government and enterprise executive documents.

[0086] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multimodal retrieval enhancement method for government and enterprise executive documents, characterized in that, The method includes: S1. Perform OCR recognition on the table of contents of the policy document to be processed, parse the hierarchical structure of the main document title, and perform preliminary segmentation of the document according to the page number range corresponding to the main title; perform OCR recognition on the segmented document and convert it into Markdown format text, and perform post-processing on the Markdown format text; S2. Slice the post-processed Markdown text according to heading level to obtain multiple text slices containing main headings, first-level headings, second-level headings, and body text. Integrate the corresponding images or tables in the attachments into the body text references. Divide the multiple text slices of the body text according to the set length threshold and set the overlap interval to generate text slices for encoding. The text slices for encoding contain fine-grained heading and body text combined slices and coarse-grained document frame slices. S3. The text slices are vectorized using an improved semantic coding model, and the coding results and corresponding metadata are stored in a vector database. S4. The user query is vector-encoded, and a similarity search is performed in the vector database to obtain candidate text slices; the candidate text slices are reordered to obtain a target text slice set. S5. Slice the target text and construct prompt information with the user query, input the multimodal large model to generate question and answer results, wherein the question and answer results include text and corresponding image or table information; S6. Evaluate the question-and-answer results based on preset evaluation indicators, which include performance indicators for the retrieval stage and effectiveness indicators for the generation stage.

2. The multimodal retrieval enhancement method for government and enterprise executive documents as described in claim 1, characterized in that, The post-processing described in step S1 includes: Extract visual features of text blocks, and perform heading level completion on the Markdown formatted text based on the visual features of text blocks; The heading level of the Markdown formatted text is corrected based on regular expression rules; The Markdown formatted text is processed for noise removal and typo correction.

3. The multimodal retrieval enhancement method for government and enterprise executive documents as described in claim 2, characterized in that, The title hierarchy completion based on the visual features of text blocks includes determining the title hierarchy relationship by using font size, character weight, line spacing, and spatial position.

4. The multimodal retrieval enhancement method for government and enterprise executive documents as described in claim 1, characterized in that, The improved semantic coding model in step S3 includes: Obtain a hybrid dataset consisting of four tasks: retrieval, semantic similarity, classification, and clustering. The dataset is specifically designed for the characteristics of government and enterprise documents, including query document pairs constructed using abbreviations specific to government and enterprise, modal verbs related to legal compliance, and long document paragraphs. Fine-tuning was performed through a two-stage training process. The first stage involved basic capability training, which applied a hybrid retrieval approach to the text slices to be encoded, incorporating dense retrieval, lexical retrieval, and multi-vector retrieval, and optimizing the model using infoNCE loss. The second stage involved joint fine-tuning for government and enterprise scenarios, which introduced a self-knowledge distillation mechanism. After generating teacher signals, weighted contrast loss, no-distillation loss, and distillation loss were calculated. Finally, the average of these three losses was used as the final loss to fine-tune the model, resulting in an improved semantic encoding model adapted to government and enterprise policy document scenarios.

5. The multimodal retrieval enhancement method for government and enterprise executive documents as described in claim 1, characterized in that, The vector database uses the Milvus database and constructs an index structure that includes text vectors, title information, path information, and time information.

6. The multimodal retrieval enhancement method for government and enterprise executive documents as described in claim 1, characterized in that, The index structure is an HNSW index.

7. The multimodal retrieval enhancement method for government and enterprise executive documents as described in claim 1, characterized in that, Steps S1, S2 and S3 are executed in parallel.

8. A multimodal retrieval enhancement system for government and enterprise executive documents, characterized in that: The system includes: processor; Memory used to store processor-executable instructions; The processor is configured to implement the multimodal retrieval enhancement method for government and enterprise institutional documents as described in any one of claims 1 to 7 when executing the executable instructions.

Images