A multi-modal retrieval augmented generation system and method for enterprise-level heterogeneous knowledge

Abstract

The application discloses a kind of multi-modal retrieval enhancement generation system and method for enterprise-level heterogeneous knowledge, and the system includes the following modules: knowledge resource construction module M1, query structured module M2, retrieval gate module M3, iterative retrieval module M4, generation gain reordering module M5, evidence organization and answer generation module M6, answer verification and feedback control module M7.Compared with the prior art that mainly bases on semantic similarity to carry out one recall and directly generates answer, the application maintains heterogeneous knowledge structure, realizes source backtracking, controls retrieval opportunity, reduces evidence generation, improves the evidence completeness of complex problem, makes candidate sorting target consistent with answer quality, and improves answer credibility and auditability.

Description

Technical Field

[0001] This invention relates to a multimodal retrieval enhancement generation system and method for enterprise-level heterogeneous knowledge, belonging to the field of information technology, and particularly to the field of retrieval enhancement generation and multimodal information processing technology. Background Technology

[0002] Retrieval Enhanced Generation (RAG) technology mitigates problems such as model knowledge lag, lack of enterprise proprietary knowledge, and factual illusions by introducing external knowledge bases into the large language model generation process. In enterprise-level application scenarios, knowledge sources typically include heterogeneous data such as PDFs, Word documents, Excel spreadsheets, images, audio, and video. How to achieve multimodal knowledge parsing, unified retrieval, candidate re-ranking, and reliable generation is a key research focus. Currently, the following six patents disclose related technical solutions:

[0003] Patent CN117725183A proposes a re-ranking method and apparatus to improve the retrieval performance of AI large language models. The specific process is as follows: First, the user question is vectorized and the initial recall range is expanded in the vector database; then, information such as the embedding features, word frequency features, and semantic similarity of the candidate texts are extracted, and the candidate texts are re-scored; finally, the top-ranked texts are selected as the context of the large language model to improve the matching degree between the retrieval results and the user question.

[0004] Patent CN119646192A proposes a multimodal scientific data retrieval method and system based on abstract-assisted cognitive enhancement. The specific process is as follows: First, scientific data such as text, tables, and structural diagrams are converted into unified feature representations, and corresponding structured abstracts are generated. Then, a keyword entity tree is constructed based on the abstracts, and the retrieval scope is expanded through multi-step dynamic reasoning. Finally, a semantic matching model is used to calculate the relevance between the question and candidate data, and the recall results are reordered.

[0005] Patent CN120705275A proposes a method and system for intelligent document parsing and question answering based on RAG and multimodal knowledge graphs. The specific process is as follows: First, text and image information in the document are extracted using document parsing tools and a multimodal large model; then, text and image knowledge graphs are constructed separately and fused; finally, the entities in the graphs are vectorized for user question retrieval, and answers are generated based on the retrieval results.

[0006] Patent CN118885627A proposes a multi-turn dialogue processing method and system based on RAG and knowledge graph. The specific process is as follows: First, the related entities in the user's question are identified and reasoning links are constructed based on the proprietary knowledge graph; then, text, image, and audio data related to the user's question are retrieved; finally, the multimodal data, knowledge graph reasoning results, and historical dialogue content are input into a multimodal large model to generate multi-turn dialogue responses.

[0007] Patent CN118193714A proposes a dynamic adaptive question-answering system and method based on hierarchical structure and retrieval enhancement. The specific process is as follows: First, a hierarchical knowledge representation is established, and question-answering is completed through a retrieval enhancement generation mechanism; then, user feedback and historical interactions are analyzed to adjust the knowledge representation, retrieval strategy, and model parameters; finally, through model fine-tuning and cross-domain knowledge integration, the question-answering system can be continuously updated according to changes in business data and user needs.

[0008] Patent publication number CN118013020A proposes a patent query method and system based on joint training of retrieval and generation. The specific process is as follows: First, the original query is refined using a question generation model; then, a document retrieval model recalls candidate patent documents from the patent database, and documents that are semantically relevant but do not meet the query intent are designated as difficult negative samples; finally, the question generation model and the document retrieval model are jointly optimized to improve the discriminative ability of patent document retrieval.

[0009] While the aforementioned existing technologies have made some progress in retrieval enhancement generation, multimodal knowledge processing, and candidate re-ranking, each solution typically focuses on semantic relevance, unified representation, knowledge graph question answering, or long-term model adaptation, and has not yet formed a complete processing loop around enterprise-level complex question answering, from retrieval triggering, evidence completion, evidence ranking to answer verification. Specific shortcomings are as follows:

[0010] The aforementioned patent CN117725183A re-ranks candidates by expanding the candidate recall scope and integrating text embedding, word frequency, and semantic similarity. Its scoring results mainly reflect the relevance between candidate texts and queries. This scheme primarily processes text and does not characterize the actual improvement in the correctness, completeness, and certainty of the answer before and after adding candidate evidence, nor does it determine whether to initiate a search based on whether the model's current knowledge is sufficient.

[0011] The aforementioned patent CN119646192A achieves multimodal scientific data retrieval through unified feature representation, structured summarization, and multi-step dynamic reasoning. However, its candidate ranking still primarily relies on the semantic matching degree between the question and the data. This scheme does not break down questions into verifiable fact slots for enterprise question answering, making it difficult to determine which specific numerical, time, cause, version, or source information is missing from the current evidence. Furthermore, it does not utilize the actual contribution of the evidence to the generated results for ranking.

[0012] The aforementioned patent CN120705275A achieves document question answering by constructing and integrating text and image knowledge graphs, focusing on the construction of multimodal knowledge graphs and graph entity retrieval. However, this solution does not adequately handle merged cells, multi-level headers, statistical units, and cell coordinates in enterprise reports, nor does it cover audio / video semantic slicing and time positioning. Furthermore, its question answering process lacks retrieval gating based on model self-awareness boundaries and multi-round supplementary retrieval based on evidence gaps.

[0013] The aforementioned patent CN118885627A utilizes knowledge graph reasoning links and multimodal data to support multi-turn dialogues, focusing on dialogue intent alignment and graph reasoning. However, this solution lacks a unified checking mechanism for fact slot coverage, evidence conflict, and the value of newly added evidence, making it difficult to clearly control whether to continue or end the search based on the current evidence status. Furthermore, its candidate selection process does not disclose the use of poor answer quality to construct gain labels and train a re-ranking model.

[0014] The aforementioned patent CN118193714A achieves continuous adaptation of the question-answering system through user feedback, model fine-tuning, and cross-domain knowledge integration. Its dynamism is mainly reflected in the long-term updating of knowledge representation and model parameters. This solution does not first determine the model's knowledge boundaries for a single query, and then perform a limited number of rounds of targeted retrieval around the uncovered facts. Nor does it establish a unified knowledge unit that retains table coordinates, image regions, audio and video timestamps, version, and permission information.

[0015] The aforementioned patent CN118013020A improves patent document retrieval capabilities through refined queries, construction of difficult negative samples, and joint training. Its training objective remains the same: to distinguish relevant documents from those that do not meet the query intent. This approach does not compare the quality changes of the generated answers before and after adding candidate evidence, does not address the unified ranking of text, tables, images, and audio / video evidence, and does not establish a feedback relationship between search results and answer fact-checking.

[0016] In summary, how to solve the problems that still exist in existing enterprise-level multimodal RAG technology, such as the difficulty in fully preserving knowledge structure and source information, lack of control over retrieval timing and depth, difficulty in diagnosing evidence gaps in complex questions, inconsistency between candidate ranking targets and the quality of the final answer, and lack of item-by-item verification of generated results, has become an urgent technical challenge in the field of multimodal retrieval enhancement generation technology for enterprise-level heterogeneous knowledge. Summary of the Invention

[0017] In view of this, the purpose of this invention is to provide a system and method for realizing unified parsing, on-demand retrieval, and verifiable generation of heterogeneous enterprise knowledge.

[0018] To achieve the above objectives, this invention proposes a multimodal retrieval enhancement generation system for enterprise-level heterogeneous knowledge, the system comprising the following modules:

[0019] Knowledge Resource Construction Module M1: This module is used to access heterogeneous knowledge sources within the enterprise, such as PDFs, Word documents, web pages, Excel files, images, audio, and video. It performs version registration, permission registration, and structure-preserving parsing on these heterogeneous knowledge sources, and establishes unified knowledge units and hybrid indexes for online retrieval. The module's inputs are enterprise knowledge source files, media streams, and version and access control information associated with the knowledge sources. The outputs are unified knowledge units, keyword indexes, vector indexes, and metadata indexes. The outputs are written to a shared knowledge storage area and serve as the basis for retrieval by the iterative retrieval module M4. This module includes a resource access and version permission registration unit, a structure-preserving parsing unit, a unified knowledge unit, an encapsulation unit, and a hybrid index construction unit.

[0020] The query structuring module M2: This module converts the user's current question, historical dialogues, historical evidence, and user-selected charts or media objects into structured queries that can directly perform retrieval and evidence integrity judgment. The inputs to this module are the current user question, historical messages within a preset window, historical evidence objects, currently selected objects on the interface, and a glossary of enterprise terms. The output is a structured query object Q. This structured query object Q serves as the input to the retrieval gating module M3 and the iterative retrieval module M4. This module includes a context object location unit, a terminology and spatiotemporal constraint normalization unit, a fact slot decomposition unit, and a structured query generation unit.

[0021] The retrieval gating module M3 is used to determine whether the current question requires access to the enterprise knowledge base and to generate a retrieval plan when necessary. Its inputs are the structured query object Q output by the query structuring module M2, the mandatory retrieval rules, and the knowledge sufficiency assessment results of internal candidate answers. Its outputs are the retrieval decision D(q), the retrieval trigger reason, the fact slots to be covered, the priority retrieval modality, and the filtering conditions. When D(q) = 1, the output of this module serves as the initial input to the iterative retrieval module M4; when D(q) = 0, the output serves as the direct answer constraint for the evidence organization and answer generation module M6. This module includes a mandatory retrieval determination unit, a knowledge sufficiency assessment unit, and a retrieval plan generation unit.

[0022] Iterative Retrieval Module M4: This module performs one or more rounds of retrieval around the fact slots in the structured query, and determines whether to continue, complete, or stop the retrieval based on evidence coverage, evidence conflict, retrieval rounds, and remaining budget. The inputs to this module are the retrieval plan output by the retrieval gating module M3, the indexes and unified knowledge units in the shared knowledge storage area, and the gap information returned by the answer verification and feedback control module M7. The outputs are the current candidate unified knowledge unit set, the retrieval status object, and the final evidence set. The current candidate unified knowledge unit set is sent to the gain reordering module M5, and the final evidence set is sent to the evidence organization and answer generation module M6. This module includes a retrieval status maintenance unit, a subquery planning and hybrid recall unit, a candidate unified knowledge unit merging and filtering unit, an evidence gap and conflict diagnosis unit, and a state transition and supplementary retrieval unit.

[0023] The Gain Re-ranking Module M5 is used to evaluate the actual gain on the quality of the final answer after the candidate unified knowledge units (CQUs) constructed by the Knowledge Resource Construction Module M1 and recalled by the Iterative Retrieval Module M4 are added to the current evidence set, and to rank the multimodal candidate CQUs accordingly. The inputs to this module are a structured query object Q, the currently selected evidence set, and the current candidate CQU set. The outputs are the predicted gain, overall ranking score, and suggested evidence set for each candidate CQU. This output is returned to the Iterative Retrieval Module M4, which uses it for evidence selection, fact slot coverage determination, and next state determination. This module includes a training sample and gain label construction unit, a gain prediction unit, a modality adaptation and overall scoring unit, and an evidence selection unit.

[0024] Evidence Organization and Answer Generation Module M6: This module transforms the final evidence set into a structured context centered on fact slots and retaining source anchors, and generates initial answers that can bind key conclusions to evidence item by item. The inputs to this module are a structured query object Q, the final evidence set, and the answer structure requirements. The output is the initial answer and the conclusion-evidence mapping relationship. This output serves as the input to the answer verification and feedback control module M7. This module includes an evidence object construction unit, a structured context assembly unit, and a fact anchoring generation unit.

[0025] Answer Verification and Feedback Control Module M7: This module verifies the source, access permissions, document version, numerical values, units, time, semantic support relationships, and evidence conflicts in the initial answer, and controls answer correction, supplementary retrieval, or final output based on the verification results. The inputs to this module are the initial answer, the conclusion-evidence mapping relationship, the final evidence set, and the retrieval status object. The outputs are the final answer, evidence source, uncovered facts, conflict explanation, and verification status. When key conclusions lack evidence but there is still retrieval budget, this module also outputs the missing slots and conflict reasons and returns to the iterative retrieval module M4. This module includes a source permission version verification unit, a numerical semantic conflict verification unit, an answer correction and re-check control unit, and a final output unit.

[0026] During the online question-and-answer process, the structured query objects output by the query structure module M2 form a retrieval decision through the retrieval gating module M3. When a retrieval is needed, the iterative retrieval module M4 performs recall based on the hybrid index formed by the knowledge resource construction module M1, and completes the ranking of candidate unified knowledge units in each round of calls to the gain generation and reordering module M5. After the retrieval is completed, the evidence organization and answer generation module M6 generates an initial answer based on the final evidence, and the answer verification and feedback control module M7 verifies the answer and conclusion-evidence mapping. When M7 finds that key evidence is missing or there are unresolved conflicts and there is still a retrieval budget, it returns the gap information to M4 to perform supplementary retrieval. When the verification is passed or the budget is exhausted, the final result is output, thus forming a closed loop of "retrieval - reordering - generation - verification - supplementary retrieval".

[0027] The knowledge resource construction module M1 and the shared knowledge storage area constitute the knowledge construction link, while modules M2 to M7 constitute the online question-and-answer link. The online question-and-answer link selects either a direct answer path or an enterprise knowledge base retrieval path based on the retrieval decision, and feeds back the missing fact slots and conflict reasons to the iterative retrieval module M4 through the answer verification results.

[0028] The knowledge resource construction module M1 processes enterprise knowledge sources in the order of "resource registration - modal parsing - unified knowledge unit encapsulation - hybrid index construction", so that the keyword index, vector index and metadata index all point to the same unified knowledge unit, and retain version, permission and source location information;

[0029] In the knowledge resource construction module M1:

[0030] The resource access and version permission registration unit consists of a connector adaptation subunit, a document primary key generation subunit, a version relationship subunit, and a permission mapping subunit. The connector adaptation subunit retrieves source files or media streams from a file system, document management platform, object storage, database, or media platform. The document primary key generation subunit generates a document primary key based on file identifier, content summary, version number, and modification time to identify duplicate files. The version relationship subunit records inheritance, substitution, and invalidation relationships between different versions and marks invalidated versions as non-default recall. The permission mapping subunit converts user, department, role, document, and field-level access control rules into searchable permission metadata. The resource access and version permission registration unit outputs a resource registration object with document primary key, source path, version status, validity period, and access permissions.

[0031] The structure preservation parsing unit includes a text layout parsing subunit, a complex table parsing subunit, an image parsing subunit, and an audio / video parsing subunit;

[0032] The text layout parsing subunit is used to identify layout areas such as titles, body text, lists, headers, footers, footnotes, images, and tables. It restores the relationship between these areas according to the reading order, removes duplicate headers and footers, watermarks, and blank areas, and performs block segmentation according to chapters, paragraphs, list items, and semantic transitions. When a single semantic block exceeds the preset maximum length L, a secondary segmentation with overlapping windows is used, and the preceding block, succeeding block, chapter path, page number, and paragraph number are recorded. Here, L is a positive integer used to limit the maximum number of characters or the maximum number of words in a single text block.

[0033] The complex table parsing subunit is used to sequentially perform merged cell expansion, table orientation recognition, multi-level header compression, unit and primary key recognition, multi-granular segmentation, and source location. Merged cell expansion propagates the original values ​​spanning rows or columns to the underlying cells it covers, preserving the original coordinates of the merged area. Table orientation recognition determines the record orientation based on the text ratio, numerical ratio, repetition pattern, and data type distribution of the first row and column, performing row and column transposition when necessary. Multi-level header compression forms unique column names based on the parent-child hierarchy of the header. Unit and primary key recognition binds statistical categories such as amount, ratio, quantity, date, region, and product with corresponding values. Multi-granular segmentation forms a table summary, sub-table blocks, and row blocks. Source location records the table number, worksheet name, page number, row and column range, and cell coordinates. When the structure recognition confidence level is lower than a preset threshold, the OCR text, original table image, and original file reference are simultaneously preserved to avoid losing original evidence due to structuring failure.

[0034] The image parsing subunit includes OCR processing, layout area detection, and visual semantic description. It is used to extract text, main objects, object relationships, coordinate axes, legends, key values, trends, and directly obtainable conclusions from flowcharts, equipment diagrams, dashboards, and screenshots, and to record image page numbers, area coordinates, and parsing confidence.

[0035] The audio and video parsing subunit is used to perform timestamped speech recognition, speaker segmentation, scene transition detection, keyframe extraction, and audio-visual alignment. For audio, this subunit forms semantic segments with speaker and start and end times according to topic changes or silence intervals. For video, this subunit merges keyframe OCR results, visual descriptions, and speech transcription within the same time window into a spatiotemporal slice, and records the media file identifier, start time, end time, speaker, and keyframe number.

[0036] The unified knowledge unit encapsulation unit is used to encapsulate text blocks, table blocks, image semantic proxies, audio segments, and video spatiotemporal slices into unified knowledge units; the i-th unified knowledge unit u i Represented as:

[0037]

[0038] In the above formula, i is the index of the unified knowledge unit, and i can take values ​​from 1 to N. u For any positive integer N, u The total number of unified knowledge units in the system; id i m is the unique identifier for the i-th unified knowledge unit; i The modal type can be one of TEXT, TABLE, IMAGE, AUDIO, or VIDEO; x iThis refers to the original content, a fragment of original data, or a reference to the original file; p i Semantic proxy text for keyword retrieval and vector retrieval; s i For structural information, text includes chapter paths, page numbers, and paragraph numbers; tables include table headers, primary keys, units, and row and column coordinates; images include region coordinates and object relationships; and audio / video includes speaker and keyframe information. i For business time or media time information, it can represent a single point in time or a time interval consisting of start and end times; i Source anchors include document name, file path, page number, table coordinates, image region, or media time interval; v i For version number, version status, and validity period; a i For access permission set; e i Let e ​​be the vector representation of the i-th unified knowledge unit. i Belonging to the d-dimensional real vector space R d d is the output dimension of the embedded model;

[0039] The hybrid index building unit consists of a keyword index subunit, a vector index subunit, and a metadata index subunit. The keyword index subunit builds an inverted index for semantic proxies, table headers, units, entities, and enterprise terms. The vector index subunit adds modal identifiers and structural summaries before the semantic proxies, generates vectors through an embedding model, and builds a vector index. The metadata index subunit builds filtering indexes for permissions, versions, validity periods, departments, document types, time ranges, page numbers, and media timestamps. During online recall, the system first performs metadata filtering, then performs keyword retrieval and vector retrieval in parallel, and calculates the initial recall score B of the i-th candidate unified knowledge unit using the following formula. i :

[0040]

[0041] In the above formula, i is the index of the candidate unified knowledge unit; B i B is the initial recall comprehensive score for the i-th candidate unified knowledge unit. i Located between 0 and 1; B i kw B is the keyword matching score for the i-th candidate unified knowledge unit. i vec B is the vector similarity score for the i-th candidate unified knowledge unit. i metaLet η1 be the metadata matching score for the i-th candidate unified knowledge unit. All three scores are normalized to 0 to 1 before weighting. η1, η2, and η3 are the weights of the keyword score, vector score, and metadata score, respectively, all of which are between 0 and 1 and satisfy η1 + η2 + η3 = 1. For numerical and statistical queries, increase the weight of η1 and the keywords corresponding to the table header, indicator name, and unit; for time-sensitive queries, increase the weight of η3 and the metadata corresponding to the version and time conditions.

[0042] In the query structure module M2:

[0043] The context object localization unit is used to determine the object identifier and source anchor point corresponding to the indicated message from historical messages, historical evidence objects, and the chart or media object currently selected by the user; for example, "this image" is located as the selected dashboard screenshot in the current interface, "the previous table" is located as the table evidence referenced in the most recent Q&A, and "yesterday's meeting" is located as the audio and video range of the meeting one day before the query occurred; when there are multiple candidate objects and the highest matching confidence is lower than a preset threshold, the system retains multiple candidate objects for separate retrieval, or outputs a clarification request;

[0044] The terminology and spatiotemporal constraint standardization unit is used to map department abbreviations, product codes, indicator aliases and document abbreviations to standard names based on the enterprise terminology table, and to convert relative times such as "latest", "current", "this quarter" and "yesterday" into specific time points or time ranges, while extracting constraints such as department, region, version, permissions, required modality and answer format;

[0045] The fact slot decomposition unit is used to break down complex questions into fact slots that can be independently retrieved and verified according to the query intent; each fact slot includes at least the slot name, target entity, time condition, expected data type, priority modality, and slot importance.

[0046] The structured query generation unit is used to encapsulate the above processing results into a structured query object Q; the structured query object Q includes at least a standard query q. std The query intent, standardized entities, constraints, preferred modalities, freshness requirement, fact slots, and answer structure are all specified. schema ; where q std To address the semantic self-containment problem arising from omitting entities and time, answer schema This is used to specify the conclusions, comparative data, reasons, conditions, and source formats that the answer should include.

[0047] In the retrieval gating module M3:

[0048] The forced retrieval determination unit sets a forced retrieval flag F. q When a structured query involves internal company policies, business data, unpublished information, the latest or specified version, a specified source, page numbers, table coordinates, media timestamps, facts that must be answered based on images or audio / video, or falls under pre-defined high-risk business areas such as finance, compliance, and security, then F... q =1; otherwise, let F =1; q =0;F q When =1, regardless of whether the large language model can generate candidate answers, the system will enter the enterprise knowledge base retrieval path;

[0049] When F q When the answer score is 0, the knowledge sufficiency assessment unit generates candidate answers for internal evaluation only and calculates the answer confidence score C. q Fact slot completeness P q and uncertainty risk U q Then, the knowledge sufficiency K of query q is calculated according to the following formula. q :

[0050]

[0051] In the above formula, q represents the current structured query; K q K represents the model's knowledge sufficiency regarding query q. q The value is between 0 and 1, with a higher value indicating a higher probability of being able to answer correctly and completely without external evidence; C q P represents the confidence level of the internal candidate answers. q U represents the complete coverage of fact slots by internal candidate answers. q For the uncertainty risk of internal candidate answers, C q P q and U q All are between 0 and 1; 1-U q Used to convert uncertainty risk into certain contribution; λ1, λ2 and λ3 are the weights of answer confidence, fact slot completeness and certain contribution, respectively, all of which are between 0 and 1 and satisfy λ1+λ2+λ3=1;

[0052] The retrieval gating module M3 is based on F q and K q The retrieval decision D(q) is obtained by the following formula:

[0053]

[0054] In the above formula, D(q) represents the retrieval decision for query q, and D(q) can be 0 or 1; D(q) = 0 indicates that the enterprise knowledge base will not be searched, and the evidence organization and answer generation module M6 will generate an answer according to the direct answer constraint; D(q) = 1 indicates that the iterative retrieval module M4 will be entered; F q This is a forced search flag, and can be either 0 or 1; K q The calculated knowledge sufficiency; θ k θ is the knowledge sufficiency threshold. k Located between 0 and 1; the symbol "∧" represents logical AND. F ∈ F. q =0 and K q Not less than θ k When , D(q) = 0; in other cases, including F q =1 or K q Below θ k Let D(q) = 1 for all cases;

[0055] When D(q)=1, the retrieval plan generation unit generates an initial retrieval plan based on fact slots, priority modalities, time ranges, versions, permissions, and source requirements; numerical values, statistical definitions, and comparative data are preferentially allocated to the table index, institutional clauses and definitions are preferentially allocated to the text index, structures and trends are preferentially allocated to the image index, and meeting decisions, speech content, and operation processes are preferentially allocated to the audio and video index.

[0056] In the iterative retrieval module M4:

[0057] The retrieval status maintenance unit maintains the retrieval status object R for the t-th round. t ;R t At least the following should be recorded: current round t, structured query object Q, fact slot set, and currently selected evidence set E. t The current set of candidate unified knowledge units C t Uncovered slots, conflict information, used search criteria, maximum search rounds T max Remaining executable budget b t and the current state; where t is a non-negative integer, T max The maximum number of search rounds is preset and T max Greater than zero, b t It can be represented by remaining search attempts, remaining computation time, remaining tokens, or a combination thereof. Search status transitions between PLAN, SEARCH, RERANK, CHECK, REFINE, DONE, and STOP.

[0058] The subquery planning and hybrid recall unit generates one or more subqueries for each uncovered fact slot in the PLAN state. Each subquery includes at least the target entity, time condition, keywords, priority modality, and metadata filtering conditions. In the SEARCH state, knowledge units are first filtered according to permissions, version, time, department, and document type. Then, keyword retrieval and vector retrieval are performed in parallel. Each subquery obtains the top K0 candidate unified knowledge units, where K0 is a positive integer. The candidate unified knowledge unit merging and filtering unit merges and deduplicates multiple candidate unified knowledge units according to unified knowledge unit identifier, source location, and content similarity, and filters candidate unified knowledge units with no access permissions, invalid versions, or resolution confidence below the minimum threshold.

[0059] The evidence gap and conflict diagnosis unit, in the CHECK state, determines one by one whether each fact slot is supported by evidence. For the j-th fact slot in the t-th round, when evidence that can directly and completely support that slot is obtained, let h be the evidence. j,t =1, when only partial information is obtained, let h j,t =0.5, when no evidence is obtained or there are unresolved key conflicts, let h = 0.5. j,t =0; Fact slot coverage Cov in round t t Calculate according to the following formula:

[0060]

[0061] In the above formula, t is the current retrieval round; j is the fact slot index; J is the total number of fact slots, where J is a positive integer; w j w represents the importance weight of the j-th fact slot. j A value greater than zero indicates that key conclusions, key values, statistical definitions, and sources can be set with a larger w. j h j,t The degree of support for the j-th fact slot by the evidence in round t, taking a value of 0, 0.5, or 1; Cov t For the fact slot coverage in round t, Cov t The value is between 0 and 1, with a larger value indicating a more complete coverage of the facts to be answered in the current evidence set;

[0062] The evidence gap and conflict diagnosis unit also generates a key conflict marker X. t When two or more pieces of evidence provide different values, different valid versions, or contradictory conclusions regarding the same entity, the same indicator, and the same time period, and these conclusions cannot be resolved based on source level, approval status, or version validity period, let X... t =1; otherwise let X =1; t =0. This unit also records the maximum prediction generation gain G from the newly added evidence in this round. tAnd the number of newly covered fact slots in this round; when G t Below the preset gain threshold θ g In such cases, the same search criteria should not be used repeatedly in the next round; instead, the search terms, modality, or search scope should be changed.

[0063] The state transition and supplementary retrieval unit determines the next state based on coverage, conflict flag, current round, and remaining budget, according to the following formula:

[0064]

[0065] In the above formula, State t+1 For the state in round t+1, choose one of DONE, REFINE, or STOP; Cov t Let θ be the actual slot coverage obtained in round t; c θ is the evidence coverage threshold. c Located between 0 and 1; X t The key conflict flag for round t is 0 or 1; t is the current search round; T max b is the maximum number of search rounds; t This represents the remaining executable budget after round t; the symbol "∧" represents logical AND, and the symbol "∨" represents logical OR; when Cov t Not less than θ c And X t When =0, enter the DONE state; when Cov... t Below θ c or X t =1, and t is less than T max And b t When the value is greater than 0, the system enters the REFINE state; otherwise, it enters the STOP state.

[0066] When entering the REFINE state, the state transition and supplementary retrieval unit only generates new subqueries around uncovered fact slots or conflict reasons; when indicator values ​​are obtained but the reasons for the decline are lacking, only business analysis reports and meeting minutes are retrieved; when there are conflicting system versions, filter conditions such as "latest effective version," "approved," and explicit effective date are added; when the table has values ​​but lacks units, the table header, footnotes, or adjacent table blocks are retrieved. When entering the DONE state, the final evidence set and covered fact slots are output; when entering the STOP state, confirmed facts, uncovered facts, and unresolved conflicts are output.

[0067] In the gain reordering module M5:

[0068] The training samples and gain label construction units are used to construct the generation gain training labels for candidate unified knowledge units. Under the same generation model, the same prompt template, and the same decoding parameters, the system first generates a baseline answer A0 based on the query q and the current existing evidence set, and then generates an enhanced answer A after adding the i-th candidate unified knowledge unit to the existing evidence set. i The system calculates the answer quality Q(A) from four dimensions: factual accuracy, fact slot completeness, evidence consistency, and uncertainty.

[0069]

[0070] In the above formula, A represents the answer to be evaluated; Q(A) is the overall quality score of answer A, which is between 0 and 1; Acc(A) is the factual correctness score of answer A, Comp(A) is the completeness score of answer A to the fact slot, Cons(A) is the consistency score of answer A with the given evidence, and Unc(A) is the uncertainty score of answer A, all four indicators are between 0 and 1; 1-Unc(A) represents the certainty contribution; ω1, ω2, ω3 and ω4 are the weights of factual correctness, slot completeness, evidence consistency and certainty contribution, respectively, all four weights are between 0 and 1 and satisfy ω1+ω2+ω3+ω4=1;

[0071] The generation gain label ΔG of the i-th candidate unified knowledge unit i Calculate according to the following formula:

[0072]

[0073] In the above formula, i is the candidate unified knowledge unit index; A i A0 is the enhanced answer generated after adding the i-th candidate unified knowledge unit; A0 is the baseline answer generated without adding the candidate unified knowledge unit; Q(A i Q(A0) and Q(A0) are the enhanced answer quality and baseline answer quality obtained according to equation (7), respectively; ΔG i Let ΔG be the true generation gain label for the i-th candidate unified knowledge unit. i Located between -1 and 1; ΔG i A value greater than 0 indicates that the candidate unified knowledge unit can improve the quality of the answer, ΔG i A value close to 0 indicates that the candidate unified knowledge unit overlaps with or is only minimally helpful to existing evidence; ΔG i A value less than 0 indicates that the candidate unified knowledge unit may be irrelevant, outdated, contain conflicting information, or lead to a decrease in the quality of the answer;

[0074] The generation gain prediction unit employs a cross-coding reordering model, combining the standard query, query intent, fact slot, modality of the candidate unified knowledge unit, semantic proxy, structural information, version information, and source information of the candidate unified knowledge unit as joint input, and outputting the predicted generation gain of the i-th candidate unified knowledge unit. The model is trained using mean squared error loss.

[0075]

[0076] In the above formula, L gain To generate the training loss for the gain prediction model, L gain Greater than or equal to 0; M is the number of training samples, where M is an integer greater than 0; i is the index of the training sample; The generation gain of the model for the i-th training sample is preferably limited to between -1 and 1; ΔG i The calculated true generated gain labels; the goal of model training is to minimize L. gain This makes the predicted generation gain close to the actual generation gain;

[0077] During the online phase, the modality adaptation and comprehensive scoring unit calculates the modality adaptation degree Mod of the i-th candidate unified knowledge unit based on the query intent. i Furthermore, semantic relevance, prediction generation gain, modality fit, source and version validity, and degree of duplication with existing evidence are integrated into a comprehensive ranking score S. i :

[0078]

[0079] In the above formula, S i S is the comprehensive ranking score of the i-th candidate unified knowledge unit. i Rel is a real number used for the relative ranking of candidate unified knowledge units; i To determine the semantic relevance between the i-th candidate unified knowledge unit and the structured query, Mod i Val represents the modal fit between the i-th candidate unified knowledge unit and the query intent. i For the source reliability and version validity of the i-th candidate unified knowledge unit, Dup i Rel represents the degree of overlap between the i-th candidate unified knowledge unit and the currently selected evidence set. i Mod i Val i and Dup i Normalize to 0 to 1; The prediction generation gain for the i-th candidate unified knowledge unit is preferably located between -1 and 1; α, β, γ, κ, and δ are the non-negative weights of semantic relevance, prediction generation gain, modality fit, source version validity, and duplication penalty, respectively; since S i For relative sorting only, the sum of the five weights is not required to be 1; a negative sign is set before the repetition level item so that the higher the repetition level, the lower the overall sorting score.

[0080] The evidence selection unit is ranked according to a comprehensive score S. i The top N candidate unified knowledge units are selected from high to low, where N is an integer greater than zero. When the comprehensive scores are close, the candidate unified knowledge units that can add key fact slots, have valid sources, reliable parsing, and can accurately locate the original position are selected first. The ranking results, predicted generation gains, and suggested evidence sets are returned to the iterative retrieval module M4.

[0081] The iterative retrieval module M4 migrates between the states of PLAN, SEARCH, RERANK, CHECK, REFINE, DONE, and STOP, and calls the gain reordering module M5 in the RERANK state; the CHECK state determines the subsequent state based on the fact slot coverage, key conflict flags, retrieval rounds, and remaining budget.

[0082] In the evidence organization and answer generation module M6:

[0083] The evidence object construction unit is used to assign an evidence number to each piece of evidence in the final evidence set and record the fact slot, modality type, source name, version, page number or timestamp, semantic content, and original location supported by the evidence; for tabular evidence, the evidence object retains the table header, primary key, unit, and cell coordinates; for image evidence, the evidence object retains the object relationship, key area, and OCR result; for audio and video evidence, the evidence object retains the speaker, transcription segment, start time, and end time.

[0084] The structured context assembly unit organizes and generates context in the order of "fact slot - evidence number - modality - source location - evidence content", rather than directly splicing candidate unified knowledge units according to the recall order. For multiple pieces of evidence for the same fact slot, the main evidence that directly supports the conclusion is arranged first, followed by the auxiliary evidence used for cross-validation; when the context length exceeds the input budget of the generative model, fact slots with higher importance and evidence with higher prediction generation gain are retained first.

[0085] The fact anchoring generation unit generates data based on the answers in the structured query object Q. schemaA prompt template should be created, requiring the generated model to answer only based on the given evidence, and to mark the corresponding evidence number after each key value, clause, reason, and conclusion; for the direct answer path where D(q)=0, only questions that do not involve corporate private facts, time-sensitive facts, designated sources, and high-risk business are allowed to be answered; when evidence is insufficient or conflicting, the generated model must clearly state the missing or conflicting items, and may not fill in corporate facts that are not supported by sources.

[0086] In the answer verification and feedback control module M7:

[0087] The source permission version verification unit is used to check whether the evidence number cited in the initial answer exists, whether the current user has access rights, whether the source anchor is valid, and whether the corresponding document version is valid. The numerical semantic conflict verification unit is used to trace the numerical values, units, times, and calculation relationships in the answer back to table cells, OCR areas, or original text fragments to determine whether the evidence directly supports the corresponding conclusion and to check whether the same conclusion cites contradictory sources.

[0088] When the answer correction and review control unit detects missing references, inconsistent values, lack of direct support for conclusions, or conflicting key evidence, it first deletes, downgrades, or rewrites the relevant conclusions based on existing evidence; if the missing conclusion is a key fact and the remaining budget b is found in the retrieval status object... t If the remaining budget is greater than 0, the missing fact slot, conflict cause, and existing evidence status will be returned to the iterative retrieval module M4 for supplementary retrieval; if the remaining budget is 0, the confirmed facts will be retained and the insufficient evidence status will be output.

[0089] After verification, the final output unit outputs the user-facing answer, the evidence number corresponding to each key conclusion, the document name, page number, table coordinates, image area or audio / video timestamp, covered facts, uncovered facts, and explanations of any remaining conflicts.

[0090] Evidence organization, fact anchoring generation, and answer verification form a closed loop; when verification finds insufficient key evidence or unresolved conflicts and there is still budget, the answer verification and feedback control module M7 returns the missing fact slot and conflict cause to the iterative retrieval module M4; when there is no remaining budget, the confirmed facts and insufficient evidence status are output.

[0091] A multimodal retrieval enhancement generation method for enterprise-level heterogeneous knowledge based on the aforementioned system includes the following steps:

[0092] Step S100: Through the knowledge resource construction module M1, access the heterogeneous knowledge sources of the enterprise, register the document primary key, version and access permissions of the knowledge sources, perform structure-preserving parsing on knowledge sources of different modalities, and build a unified knowledge unit as well as keyword index, vector index and metadata index;

[0093] Step S200: Through the generation gain re-ranking module M5, training samples are constructed based on historical question-and-answer, manual test questions and business cases. The changes in answer quality before and after adding candidate unified knowledge units are compared to generate generation gain labels for candidate unified knowledge units. A generation gain prediction model for online ranking of candidate unified knowledge units is trained.

[0094] Step S300: By querying the structured module M2, perform referential resolution, terminology normalization, spatiotemporal constraint extraction, and fact slot decomposition on the current user's question, historical messages, historical evidence, and the chart or media object selected by the user, and output the structured query object.

[0095] In step S400, the retrieval gate module M3 determines whether to initiate an enterprise knowledge base retrieval based on the mandatory retrieval conditions and the model knowledge sufficiency calculation results. If the retrieval decision is not to retrieve, proceed to step S600. If the retrieval decision is to retrieve, generate a retrieval plan and proceed to step S500.

[0096] Step S500: Through the iterative retrieval module M4 and the generation gain reordering module M5, perform hybrid recall, candidate unified knowledge unit merging and filtering, generation gain prediction, modality-aware reordering, evidence coverage check, evidence conflict diagnosis and gap filling retrieval around the fact slot to be covered, until the completion condition is met or the stopping condition is reached.

[0097] In step S600, the final evidence is organized into a structured context that retains the source anchor point through the evidence organization and answer generation module M6 and the answer verification and feedback control module M7, and the fact-anchored answer is generated. The source, permissions, version, value, unit, time, semantic support relationship and evidence conflict in the answer are verified one by one. If the key evidence is insufficient and there is still a search budget, the process returns to step S500 to perform a supplementary search. Otherwise, the final answer and the corresponding evidence source are output.

[0098] Step S100 specifically includes the following steps:

[0099] Step S110: Receive enterprise knowledge sources such as PDFs, Word documents, web pages, Excel files, images, audio, and video through the resource access and version permission registration unit; obtain file identifiers, content summaries, version numbers, modification times, source paths, and access control information; generate document primary keys based on file identifiers, content summaries, version numbers, and modification times; if the document primary key and content summary are identical, it is determined to be a duplicate resource and duplicate entries are skipped; when a new version is detected, an inheritance or replacement relationship between the new and old versions is established, and expired versions are marked as non-default recalls; convert user, department, role, document, and field-level permissions into permission metadata.

[0100] Step S120: The structure-preserving parsing unit performs corresponding parsing based on the knowledge source modality: For PDFs, Word documents, web pages, and scanned documents, it identifies page layout areas such as titles, body text, lists, headers and footers, footnotes, images, and tables, restores the reading order, and segments the document by chapter, paragraph, list item, and semantic transition; for complex tables, it sequentially performs merged cell expansion, record direction recognition, row and column transposition when necessary, multi-level header compression, unit and primary key binding, multi-granular segmentation, and cell coordinate recording; for images, it performs OCR, region detection, and visual semantic description; for audio and video, it performs time-stamped speech recognition, speaker segmentation, scene transition detection, keyframe extraction, and audio-visual alignment. When the parsing confidence is lower than a preset threshold, it simultaneously retains simplified semantic proxy and the original image, original file reference, or original media time interval.

[0101] Step S130: By encapsulating text blocks, table blocks, image semantic proxies, audio segments, and video spatiotemporal slices into unified knowledge units through a unified knowledge unit encapsulation unit, and writing a unique identifier, modality type, original content or original file reference, semantic proxy, structural information, business time or media time, source anchor, version validity period, access permissions, and vector representation for each unified knowledge unit.

[0102] Step S140: By constructing a unit through a hybrid index, a keyword inverted index is built for the semantic agent, table header, unit, entity and enterprise terminology of the unified knowledge unit. Vectors are generated by embedding model and vector indexes are built. Metadata indexes are built for permissions, version, validity period, department, document type, time range, page number and media timestamp. The three types of indexes all point to the same unified knowledge unit identifier. During online recall, the initial recall score is calculated and the results from multiple paths are merged and deduplicated.

[0103] Step S200 specifically includes the following operation steps:

[0104] Step S210: Obtain training questions from enterprise historical Q&A, manual test questions, and business cases; form structured queries and fact slots through the query structure module M2; and recall candidate unified knowledge units in text, table, image, audio, and video modalities through the iterative retrieval module M4; each training sample includes at least a structured query, fact slot, current existing evidence, semantic proxy of the candidate unified knowledge unit, modality, structural information, source location, version information, and time information of the candidate unified knowledge unit; retain candidate unified knowledge units with high similarity but no answer, duplicate existing evidence, outdated version, or factual conflict as difficult negative samples;

[0105] Step S220: Under the same generative model, the same prompt template, and the same decoding parameters, generate a baseline answer A0 based on the training question and the existing evidence. Then, add the i-th candidate unified knowledge unit to the existing evidence to generate an enhanced answer A. i Calculate the baseline answer quality Q(A0) and the enhanced answer quality Q(A0) respectively. i Then calculate the generation gain label ΔG for the i-th candidate unified knowledge unit. i The correctness of facts is determined by the standard answer, knowledge base rules, or manual verification results; the completeness of fact slots is determined by the fact slots in the structured query; the consistency of evidence is determined by whether the candidate unified knowledge unit directly supports the answer conclusion; and the uncertainty is determined by fuzzy expressions, internal contradictions in the answer, and source conflicts.

[0106] Step S230: Combine the standard query, query intent, fact slot, modality of candidate unified knowledge unit, semantic proxy of candidate unified knowledge unit, structural information, source information and version information into the joint input of the generation gain prediction model, so that the model outputs the predicted generation gain of candidate unified knowledge unit; calculate the training loss and update the model parameters until the preset convergence condition is reached; select the model parameters with smaller generation gain prediction error and better ranking effect of candidate unified knowledge unit on the validation set, and write the selected model parameters into the model parameter storage area;

[0107] In step S240, the generated gain prediction model trained in step S230 is deployed to the generated gain reordering module M5 for online use in step S530. The manual review results, answer verification results, and newly added business data are used as new training samples, and incremental training or retraining is performed on the same generated gain prediction model according to a preset cycle. The updated model parameters replace the original model parameters in the model parameter storage area and are then used in subsequent steps S530.

[0108] Step S300 specifically includes the following operation steps:

[0109] Step S310: Using the context object positioning unit, determine the object identifier and source anchor point corresponding to the delivery of indicators such as "this chart", "previous table", "this product", and "yesterday's meeting" from historical messages, historical evidence objects and the chart or media object currently selected by the user; when there are multiple candidate objects and the highest matching confidence is lower than the preset threshold, retain multiple candidate objects for processing or output clarification requests respectively;

[0110] Step S320: Through the terminology and spatiotemporal constraint standardization unit, the abbreviations of departments, product codes, indicator aliases and document abbreviations are mapped to standard names according to the enterprise terminology table. Relative times such as "latest", "current", "this quarter" and "yesterday" are converted into specific time points or time ranges, and constraints such as department, region, version, permission, required modality and answer format are extracted.

[0111] Step S330: Through the fact slot decomposition unit, the complex problem is broken down into fact slots that can be independently retrieved and verified according to the query intent; each fact slot includes at least the slot name, target entity, time condition, expected data type, priority modality and slot importance.

[0112] In step S340, the structured query generation unit encapsulates the standard query, query intent, standardized entity, constraint condition, priority mode, timeliness requirement, fact slot set and answer structure into a structured query object Q, and sends the structured query object to the retrieval gating module M3.

[0113] Step S400 specifically includes the following operation steps:

[0114] Step S410: The forced retrieval judgment unit determines whether the structured query involves internal enterprise policies, business data, unpublished information, the latest or specified version, a specified source, page number, table coordinates, media timestamps, facts that must be answered based on images or audio / video, or preset high-risk business such as finance, compliance, and security; if any condition is met, the forced retrieval flag F is activated. q =1, otherwise let F =1. q =0;

[0115] Step S420, when F q When =0, candidate answers are generated for internal evaluation only, and the confidence C of the answers is calculated respectively. q Fact slot completeness P q and uncertainty risk U q And calculate the knowledge sufficiency K. q When F q When =1, regardless of whether internal candidate answers can be generated, the forced retrieval state is directly retained;

[0116] Step S430, calculate the retrieval decision D(q); when F q =0 and K q Not lower than the knowledge sufficiency threshold θ k When D(q) is set to 0, proceed to step S600; otherwise, set D(q) to 1, generate the retrieval triggering reason, the fact slot to be covered, the priority mode, the retrieval range and the filtering conditions through the retrieval plan generation unit, initialize the retrieval state object R0 and proceed to step S500.

[0117] Step S500 specifically includes the following operation steps:

[0118] Step S510: Establish the retrieval state object R for the t-th round through the retrieval state maintenance unit. t Record the current round t, the structured query object Q, the set of fact slots, and the set of selected evidence E. t Candidate unified knowledge unit set C t Uncovered slots, conflict information, used search criteria, maximum search rounds T max Remaining executable budget b t And the current state; the initial state is set to PLAN;

[0119] Step S520: In the PLAN state, generate one or more subqueries for each uncovered fact slot. Each subquery includes at least the target entity, time condition, keywords, priority modality, and metadata filtering conditions. Then, enter the SEARCH state. First, filter unified knowledge units according to permissions, version, time, department, and document type. Then, perform keyword retrieval and vector retrieval in parallel and calculate the initial recall score. For each subquery, obtain the top K0 candidate unified knowledge units and merge and deduplicate them according to unified knowledge unit identifier, source location, and content similarity. Filter out candidate unified knowledge units with no access permissions, expired versions, or parsing confidence below the minimum threshold to form the t-th round candidate unified knowledge unit set C. t ;

[0120] Step S530: Combine the structured query object Q and the previously selected evidence set E. t-1 and the set of unified knowledge units C for candidates in round t t Input the generative gain re-ranking module M5, and output the predicted generative gain for each candidate unified knowledge unit from the generative gain prediction model trained in step S200; calculate the semantic relevance, modality fit, source and version validity, and degree of duplication with existing evidence for each candidate unified knowledge unit, and calculate the comprehensive ranking score S. i The top N candidate unified knowledge units are selected from high to low based on their comprehensive ranking scores. Priority is given to retaining candidate unified knowledge units that can add key fact slots, have valid sources, reliable parsing, and can accurately locate the original position.

[0121] Step S540: Enter the CHECK state and determine the support level h for the j-th fact slot in round t. j,t The complete evidence corresponds to h. j,t =1, partial evidence corresponds to h j,t =0.5, no evidence obtained or unresolved key conflicts corresponding to h j,t =0; Calculate the fact slot coverage Cov in round t. t When two or more pieces of evidence provide different values, valid versions, or contradictory conclusions regarding the same entity, the same indicator, and the same time period, and these cannot be resolved based on source level, approval status, or version validity period, then the key conflict indicator X is set. t =1, otherwise let X =1. t =0;

[0122] Step S550: Record the maximum prediction generation gain and the number of newly added covered fact slots in the newly added evidence in this round; when the maximum prediction generation gain is lower than the preset gain threshold θ g In such cases, the same search criteria must not be reused in the next round; instead, the search keywords, search modality, or search scope should be changed. When low gain occurs consecutively and no new fact slots are added, the remaining executable budget b will be allocated. t Set to 0; then determine the next state;

[0123] In step S560, when the next state is REFINE, generate a new subquery only around the uncovered fact slots or the reasons for evidence conflicts and return to step S520; when the next state is DONE, output the final evidence set, the covered fact slots and the source anchor and proceed to step S600; when the next state is STOP, output the confirmed facts, the uncovered facts and the unresolved conflicts, and proceed to step S600 in a conservative manner.

[0124] Step S600 specifically includes the following operation steps:

[0125] Step S610: Assign an evidence number to each piece of evidence in the final evidence set through the evidence object construction unit, and record the fact slot, modality type, source name, version, page number or timestamp, semantic content and original location supported by the evidence; table evidence retains the table header, primary key, unit and cell coordinates; image evidence retains the object relationship, key area and OCR result; audio and video evidence retains the speaker, transcription segment and start time and end time.

[0126] Step S620: The context is organized in the order of "fact slot - evidence number - modality - source location - evidence content" through the structured context assembly unit; under the same fact slot, the main evidence that directly supports the conclusion is arranged first, and then the auxiliary evidence used for cross-validation is arranged; a prompt template is generated according to the answer structure requirements in the structured query object Q, requiring the generated model to answer only based on the given evidence, and the corresponding evidence number is marked after each key value, clause, reason and conclusion, thereby generating the initial answer; when D(q)=0, direct answers are only performed for questions that do not involve enterprise private facts, time-sensitive facts, designated sources and high-risk businesses;

[0127] Step S630: The initial answer is broken down into several key conclusions and a conclusion-evidence number mapping is established. The source existence verification, access permission verification, version validity verification, numerical value and unit verification, time verification, semantic support verification, and evidence conflict verification are performed in sequence. For non-key conclusions that do not exist, have inconsistent numerical values, lack direct support, or have evidence conflicts, they are deleted, downgraded, or rewritten based on existing evidence.

[0128] Step S640, when the missing or conflicting conclusion pertains to key facts and the remaining executable budget b t When the value is greater than 0, the missing fact slot, conflict cause, and existing evidence status will be returned to step S500 to perform supplementary retrieval; when the remaining budget is 0 or there are no key facts that need to be supplemented, the retrieval feedback will be stopped.

[0129] Step S650: Output the final answer for the user, the evidence number corresponding to each key conclusion, document name, page number, table coordinates, image area or audio / video timestamp, covered facts, uncovered facts, and explanations of any remaining conflicts; record the manual review results and the verification status generated in this step as an updated sample that can be used when executing step S240.

[0130] Parsing failure handling: When the confidence level of table structure recognition, image OCR or speech recognition is lower than the preset threshold, the knowledge resource construction module M1 simultaneously retains the original file, original image or original audio and video range and simplifies semantic proxy, and reduces the parsing reliability, without directly deleting the evidence;

[0131] Modality missing handling: When a query explicitly requires a certain modality but the shared knowledge storage does not contain that modality, the iterative retrieval module M4 outputs a modality missing value; other modalities can be retrieved as supplementary evidence, but textual speculations must not be presented as non-existent visual or audio-visual facts;

[0132] Empty Recall Handling: When the initial recall is empty, the iterative retrieval module M4 sequentially expands the entity aliases, synonyms, and allowed time range, and changes the retrieval modality or index combination; if no candidate unified knowledge unit is still obtained, it enters the STOP state and outputs that insufficient evidence was not retrieved.

[0133] Handling of conflicting evidence: Prioritize evidence with updated validity, clear approval status, higher source level, and verifiability by other sources; if conflicting evidence still cannot be resolved, present the conflicting content and corresponding sources in parallel, and reduce the certainty of the conclusion.

[0134] Permissions and budget handling: Access permission filtering is performed before recall, and the final answer only shows the content and source that the current user is authorized to access; when the maximum retrieval rounds or computation budget are exhausted, confirmed conclusions, uncovered facts and unresolved conflicts are output based on the current fact slot coverage, and further calls are stopped;

[0135] Model or tool anomaly handling: When the embedded model, OCR tool, speech recognition tool, reordering model, or generation model fails to be invoked, a backup model of the same type is invoked first or the corresponding degradation processing is adopted; if the backup model is still unavailable, the current retrieval status and confirmed facts are retained, the reason for the anomaly, the incomplete steps, and the retry flag are output, and no conclusions that are not supported by evidence are generated.

[0136] The beneficial effects of this invention are as follows: Compared with the existing technology that mainly relies on semantic similarity to perform a single recall and directly generate answers, this invention maintains the heterogeneous knowledge structure and realizes source tracing, controls the timing of retrieval and reduces the generation of no evidence, improves the completeness of evidence for complex problems, makes the candidate ranking target consistent with the answer quality, and improves the credibility and auditability of the answer. Attached Figure Description

[0137] Figure 1 This is a schematic diagram of the module structure of a multimodal retrieval enhancement generation system for enterprise-level heterogeneous knowledge proposed in this invention.

[0138] Figure 2 This is a schematic diagram of the overall structure, data flow, and feedback relationship of the system of the present invention.

[0139] Figure 3 This is a schematic diagram of the knowledge resource construction module M1 of the present invention.

[0140] Figure 4 This is a schematic diagram of the structure of the query structure module M2 of the present invention.

[0141] Figure 5 This is a schematic diagram of the retrieval gate module M3 of the present invention.

[0142] Figure 6This is a schematic diagram of the state transition and supplementary retrieval process of the iterative retrieval module M4 of the present invention.

[0143] Figure 7 This is a schematic diagram of the training and online sorting process of the gain reordering module M5 of this invention.

[0144] Figure 8 This is a schematic diagram of the structure of the evidence organization and answer generation module M6 of the present invention.

[0145] Figure 9 This is a flowchart of the answer verification and feedback control module M7 of the present invention.

[0146] Figure 10 This is a schematic diagram of the overall process of a multimodal retrieval enhancement generation method for enterprise-level heterogeneous knowledge proposed in this invention. Detailed Implementation

[0147] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0148] See Figure 1 This invention introduces a multimodal retrieval enhancement generation system for enterprise-level heterogeneous knowledge, comprising the following modules:

[0149] Knowledge Resource Construction Module M1: This module is used to access heterogeneous knowledge sources within the enterprise, such as PDFs, Word documents, web pages, Excel files, images, audio, and video. It performs version registration, permission registration, and structure-preserving parsing on these heterogeneous knowledge sources, and establishes unified knowledge units and hybrid indexes for online retrieval. The module's inputs are enterprise knowledge source files, media streams, and version and access control information associated with the knowledge sources. The outputs are unified knowledge units, keyword indexes, vector indexes, and metadata indexes. The outputs are written to a shared knowledge storage area and serve as the basis for retrieval by the iterative retrieval module M4. This module includes a resource access and version permission registration unit, a structure-preserving parsing unit, a unified knowledge unit, an encapsulation unit, and a hybrid index construction unit.

[0150] The query structuring module M2: This module converts the user's current question, historical dialogues, historical evidence, and user-selected charts or media objects into structured queries that can directly perform retrieval and evidence integrity judgment. The inputs to this module are the current user question, historical messages within a preset window, historical evidence objects, currently selected objects on the interface, and a glossary of enterprise terms. The output is a structured query object Q. This structured query object Q serves as the input to the retrieval gating module M3 and the iterative retrieval module M4. This module includes a context object location unit, a terminology and spatiotemporal constraint normalization unit, a fact slot decomposition unit, and a structured query generation unit.

[0151] The retrieval gating module M3 is used to determine whether the current question requires access to the enterprise knowledge base and to generate a retrieval plan when necessary. Its inputs are the structured query object Q output by the query structuring module M2, the mandatory retrieval rules, and the knowledge sufficiency assessment results of internal candidate answers. Its outputs are the retrieval decision D(q), the retrieval trigger reason, the fact slots to be covered, the priority retrieval modality, and the filtering conditions. When D(q) = 1, the output of this module serves as the initial input to the iterative retrieval module M4; when D(q) = 0, the output serves as the direct answer constraint for the evidence organization and answer generation module M6. This module includes a mandatory retrieval determination unit, a knowledge sufficiency assessment unit, and a retrieval plan generation unit.

[0152] Iterative Retrieval Module M4: This module performs one or more rounds of retrieval around the fact slots in the structured query, and determines whether to continue, complete, or stop the retrieval based on evidence coverage, evidence conflict, retrieval rounds, and remaining budget. The inputs to this module are the retrieval plan output by the retrieval gating module M3, the indexes and unified knowledge units in the shared knowledge storage area, and the gap information returned by the answer verification and feedback control module M7. The outputs are the current candidate unified knowledge unit set, the retrieval status object, and the final evidence set. The current candidate unified knowledge unit set is sent to the gain reordering module M5, and the final evidence set is sent to the evidence organization and answer generation module M6. This module includes a retrieval status maintenance unit, a subquery planning and hybrid recall unit, a candidate unified knowledge unit merging and filtering unit, an evidence gap and conflict diagnosis unit, and a state transition and supplementary retrieval unit.

[0153] The Gain Re-ranking Module M5 is used to evaluate the actual gain on the quality of the final answer after the candidate unified knowledge units (CQUs) constructed by the Knowledge Resource Construction Module M1 and recalled by the Iterative Retrieval Module M4 are added to the current evidence set, and to rank the multimodal candidate CQUs accordingly. The inputs to this module are a structured query object Q, the currently selected evidence set, and the current candidate CQU set. The outputs are the predicted gain, overall ranking score, and suggested evidence set for each candidate CQU. This output is returned to the Iterative Retrieval Module M4, which uses it for evidence selection, fact slot coverage determination, and next state determination. This module includes a training sample and gain label construction unit, a gain prediction unit, a modality adaptation and overall scoring unit, and an evidence selection unit.

[0154] Evidence Organization and Answer Generation Module M6: This module transforms the final evidence set into a structured context centered on fact slots and retaining source anchors, and generates initial answers that can bind key conclusions to evidence item by item. The inputs to this module are a structured query object Q, the final evidence set, and the answer structure requirements. The output is the initial answer and the conclusion-evidence mapping relationship. This output serves as the input to the answer verification and feedback control module M7. This module includes an evidence object construction unit, a structured context assembly unit, and a fact anchoring generation unit.

[0155] Answer Verification and Feedback Control Module M7: This module verifies the source, access permissions, document version, numerical values, units, time, semantic support relationships, and evidence conflicts in the initial answer, and controls answer correction, supplementary retrieval, or final output based on the verification results. The inputs to this module are the initial answer, the conclusion-evidence mapping relationship, the final evidence set, and the retrieval status object. The outputs are the final answer, evidence source, uncovered facts, conflict explanation, and verification status. When key conclusions lack evidence but there is still retrieval budget, this module also outputs the missing slots and conflict reasons and returns to the iterative retrieval module M4. This module includes a source permission version verification unit, a numerical semantic conflict verification unit, an answer correction and re-check control unit, and a final output unit.

[0156] See Figure 2 During the online question-and-answer process, the structured query objects output by the query structure module M2 form a retrieval decision through the retrieval gating module M3. When a retrieval is needed, the iterative retrieval module M4 performs recall based on the hybrid index formed by the knowledge resource construction module M1, and completes the ranking of candidate unified knowledge units in each round of calling the gain reordering module M5. After the retrieval is completed, the evidence organization and answer generation module M6 generates an initial answer based on the final evidence, and the answer verification and feedback control module M7 verifies the answer and conclusion-evidence mapping. When M7 finds that key evidence is missing or there are unresolved conflicts and there is still a retrieval budget, it returns the gap information to M4 to perform supplementary retrieval. When the verification is passed or the budget is exhausted, the final result is output, thus forming a closed loop of "retrieval - reordering - generation - verification - supplementary retrieval".

[0157] The knowledge resource construction module M1 and the shared knowledge storage area constitute the knowledge construction link, while modules M2 to M7 constitute the online question-and-answer link. The online question-and-answer link selects either a direct answer path or an enterprise knowledge base retrieval path based on the retrieval decision, and feeds back the missing fact slots and conflict reasons to the iterative retrieval module M4 through the answer verification results.

[0158] The knowledge resource construction module M1 processes enterprise knowledge sources in the order of "resource registration - modal parsing - unified knowledge unit encapsulation - hybrid index construction", so that the keyword index, vector index and metadata index all point to the same unified knowledge unit, and retain version, permission and source location information;

[0159] See Figure 3 In the knowledge resource construction module M1:

[0160] The resource access and version permission registration unit consists of a connector adaptation subunit, a document primary key generation subunit, a version relationship subunit, and a permission mapping subunit. The connector adaptation subunit retrieves source files or media streams from a file system, document management platform, object storage, database, or media platform. The document primary key generation subunit generates a document primary key based on file identifier, content summary, version number, and modification time to identify duplicate files. The version relationship subunit records inheritance, substitution, and invalidation relationships between different versions and marks invalidated versions as non-default recall. The permission mapping subunit converts user, department, role, document, and field-level access control rules into searchable permission metadata. The resource access and version permission registration unit outputs a resource registration object with document primary key, source path, version status, validity period, and access permissions.

[0161] The structure preservation parsing unit includes a text layout parsing subunit, a complex table parsing subunit, an image parsing subunit, and an audio / video parsing subunit;

[0162] The text layout parsing subunit is used to identify layout areas such as titles, body text, lists, headers, footers, footnotes, images, and tables. It restores the relationship between these areas according to the reading order, removes duplicate headers and footers, watermarks, and blank areas, and performs block segmentation according to chapters, paragraphs, list items, and semantic transitions. When a single semantic block exceeds the preset maximum length L, a secondary segmentation with overlapping windows is used, and the preceding block, succeeding block, chapter path, page number, and paragraph number are recorded. Here, L is a positive integer used to limit the maximum number of characters or the maximum number of words in a single text block.

[0163] The complex table parsing subunit is used to sequentially perform merged cell expansion, table orientation recognition, multi-level header compression, unit and primary key recognition, multi-granular segmentation, and source location. Merged cell expansion propagates the original values ​​spanning rows or columns to the underlying cells it covers, preserving the original coordinates of the merged area. Table orientation recognition determines the record orientation based on the text ratio, numerical ratio, repetition pattern, and data type distribution of the first row and column, performing row and column transposition when necessary. Multi-level header compression forms unique column names based on the parent-child hierarchy of the header. Unit and primary key recognition binds statistical categories such as amount, ratio, quantity, date, region, and product with corresponding values. Multi-granular segmentation forms a table summary, sub-table blocks, and row blocks. Source location records the table number, worksheet name, page number, row and column range, and cell coordinates. When the structure recognition confidence level is lower than a preset threshold, the OCR text, original table image, and original file reference are simultaneously preserved to avoid losing original evidence due to structuring failure.

[0164] The image parsing subunit includes OCR processing, layout area detection, and visual semantic description. It is used to extract text, main objects, object relationships, coordinate axes, legends, key values, trends, and directly obtainable conclusions from flowcharts, equipment diagrams, dashboards, and screenshots, and to record image page numbers, area coordinates, and parsing confidence.

[0165] The audio and video parsing subunit is used to perform timestamped speech recognition, speaker segmentation, scene transition detection, keyframe extraction, and audio-visual alignment. For audio, this subunit forms semantic segments with speaker and start and end times according to topic changes or silence intervals. For video, this subunit merges keyframe OCR results, visual descriptions, and speech transcription within the same time window into a spatiotemporal slice, and records the media file identifier, start time, end time, speaker, and keyframe number.

[0166] The unified knowledge unit encapsulation unit is used to encapsulate text blocks, table blocks, image semantic proxies, audio segments, and video spatiotemporal slices into unified knowledge units; the i-th unified knowledge unit u i Represented as:

[0167]

[0168] In the above formula, i is the index of the unified knowledge unit, and i can take values ​​from 1 to N. u For any positive integer N, u The total number of unified knowledge units in the system; id i m is the unique identifier for the i-th unified knowledge unit; i The modal type can be one of TEXT, TABLE, IMAGE, AUDIO, or VIDEO; x iThis refers to the original content, a fragment of original data, or a reference to the original file; p i Semantic proxy text for keyword retrieval and vector retrieval; s i For structural information, text includes chapter paths, page numbers, and paragraph numbers; tables include table headers, primary keys, units, and row and column coordinates; images include region coordinates and object relationships; and audio / video includes speaker and keyframe information. i For business time or media time information, it can represent a single point in time or a time interval consisting of start and end times; i Source anchors include document name, file path, page number, table coordinates, image region, or media time interval; v i For version number, version status, and validity period; a i For access permission set; e i Let e ​​be the vector representation of the i-th unified knowledge unit. i Belonging to the d-dimensional real vector space R d d is the output dimension of the embedded model;

[0169] The hybrid index building unit consists of a keyword index subunit, a vector index subunit, and a metadata index subunit. The keyword index subunit builds an inverted index for semantic proxies, table headers, units, entities, and enterprise terms. The vector index subunit adds modal identifiers and structural summaries before the semantic proxies, generates vectors through an embedding model, and builds a vector index. The metadata index subunit builds filtering indexes for permissions, versions, validity periods, departments, document types, time ranges, page numbers, and media timestamps. During online recall, the system first performs metadata filtering, then performs keyword retrieval and vector retrieval in parallel, and calculates the initial recall score B of the i-th candidate unified knowledge unit using the following formula. i :

[0170]

[0171] In the above formula, i is the index of the candidate unified knowledge unit; B i B is the initial recall comprehensive score for the i-th candidate unified knowledge unit. i Located between 0 and 1; B i kw B is the keyword matching score for the i-th candidate unified knowledge unit. i vec B is the vector similarity score for the i-th candidate unified knowledge unit. i metaLet η1 be the metadata matching score for the i-th candidate unified knowledge unit. All three scores are normalized to 0 to 1 before weighting. η1, η2, and η3 are the weights of the keyword score, vector score, and metadata score, respectively, all of which are between 0 and 1 and satisfy η1 + η2 + η3 = 1. For numerical and statistical queries, increase the weight of η1 and the keywords corresponding to the table header, indicator name, and unit; for time-sensitive queries, increase the weight of η3 and the metadata corresponding to the version and time conditions.

[0172] See Figure 4 In the query structure module M2:

[0173] The context object localization unit is used to determine the object identifier and source anchor point corresponding to the indicated message from historical messages, historical evidence objects, and the chart or media object currently selected by the user; for example, "this image" is located as the selected dashboard screenshot in the current interface, "the previous table" is located as the table evidence referenced in the most recent Q&A, and "yesterday's meeting" is located as the audio and video range of the meeting one day before the query occurred; when there are multiple candidate objects and the highest matching confidence is lower than a preset threshold, the system retains multiple candidate objects for separate retrieval, or outputs a clarification request;

[0174] The terminology and spatiotemporal constraint standardization unit is used to map department abbreviations, product codes, indicator aliases and document abbreviations to standard names based on the enterprise terminology table, and to convert relative times such as "latest", "current", "this quarter" and "yesterday" into specific time points or time ranges, while extracting constraints such as department, region, version, permissions, required modality and answer format;

[0175] The fact slot decomposition unit is used to break down complex questions into fact slots that can be independently retrieved and verified according to the query intent. Each fact slot includes at least the slot name, target entity, time condition, expected data type, priority mode, and slot importance. For example, for the question "Compare the gross profit margin of this quarter with that of the previous quarter and explain the reasons for the decline", the system generates six fact slots: gross profit margin of this quarter, gross profit margin of the previous quarter, change, statistical scope, reasons for the decline, and source basis.

[0176] The structured query generation unit is used to encapsulate the above processing results into a structured query object Q; the structured query object Q includes at least a standard query q. std The query intent, standardized entities, constraints, preferred modalities, freshness requirement, fact slots, and answer structure are all specified. schema ; where q std To address the semantic self-containment problem arising from omitting entities and time, answerschema This is used to specify the conclusions, comparative data, reasons, conditions, and source formats that the answer should include.

[0177] See Figure 5 In the retrieval gating module M3:

[0178] The forced retrieval determination unit sets a forced retrieval flag F. q When a structured query involves internal company policies, business data, unpublished information, the latest or specified version, a specified source, page numbers, table coordinates, media timestamps, facts that must be answered based on images or audio / video, or falls under pre-defined high-risk business areas such as finance, compliance, and security, then F... q =1; otherwise, let F =1; q =0;F q When =1, regardless of whether the large language model can generate candidate answers, the system will enter the enterprise knowledge base retrieval path;

[0179] When F q When the answer score is 0, the knowledge sufficiency assessment unit generates candidate answers for internal evaluation only and calculates the answer confidence score C. q Fact slot completeness P q and uncertainty risk U q Then, the knowledge sufficiency K of query q is calculated according to the following formula. q :

[0180]

[0181] In the above formula, q represents the current structured query; K q K represents the model's knowledge sufficiency regarding query q. q The value is between 0 and 1, with a higher value indicating a higher probability of being able to answer correctly and completely without external evidence; C q P represents the confidence level of the internal candidate answers. q U represents the complete coverage of fact slots by internal candidate answers. q For the uncertainty risk of internal candidate answers, C q P q and U q All are between 0 and 1; 1-U q Used to convert uncertainty risk into certain contribution; λ1, λ2 and λ3 are the weights of answer confidence, fact slot completeness and certain contribution, respectively, all of which are between 0 and 1 and satisfy λ1+λ2+λ3=1;

[0182] The retrieval gating module M3 is based on F q and K q The retrieval decision D(q) is obtained by the following formula:

[0183]

[0184] In the above formula, D(q) represents the retrieval decision for query q, and D(q) can be 0 or 1; D(q) = 0 indicates that the enterprise knowledge base will not be searched, and the evidence organization and answer generation module M6 will generate an answer according to the direct answer constraint; D(q) = 1 indicates that the iterative retrieval module M4 will be entered; F q This is a forced search flag, and can be either 0 or 1; K q The calculated knowledge sufficiency; θ k θ is the knowledge sufficiency threshold. k Located between 0 and 1; the symbol "∧" represents logical AND. F ∈ F. q =0 and K q Not less than θ k When , D(q) = 0; in other cases, including F q =1 or K q Below θ k Let D(q) = 1 for all cases;

[0185] When D(q)=1, the retrieval plan generation unit generates an initial retrieval plan based on fact slots, priority modalities, time ranges, versions, permissions, and source requirements; numerical values, statistical definitions, and comparative data are preferentially allocated to the table index, institutional clauses and definitions are preferentially allocated to the text index, structures and trends are preferentially allocated to the image index, and meeting decisions, speech content, and operation processes are preferentially allocated to the audio and video index.

[0186] See Figure 6 In the iterative retrieval module M4:

[0187] The retrieval status maintenance unit maintains the retrieval status object R for the t-th round. t ;R t At least the following should be recorded: current round t, structured query object Q, fact slot set, and currently selected evidence set E. t The current set of candidate unified knowledge units C t Uncovered slots, conflict information, used search criteria, maximum search rounds T max Remaining executable budget b t and the current state; where t is a non-negative integer, T max The maximum number of search rounds is preset and T max Greater than zero, b t It can be represented by remaining search attempts, remaining computation time, remaining tokens, or a combination thereof. Search status transitions between PLAN, SEARCH, RERANK, CHECK, REFINE, DONE, and STOP.

[0188] The subquery planning and hybrid recall unit generates one or more subqueries for each uncovered fact slot in the PLAN state. Each subquery includes at least the target entity, time condition, keywords, priority modality, and metadata filtering conditions. In the SEARCH state, knowledge units are first filtered according to permissions, version, time, department, and document type. Then, keyword retrieval and vector retrieval are performed in parallel. Each subquery obtains the top K0 candidate unified knowledge units, where K0 is a positive integer. The candidate unified knowledge unit merging and filtering unit merges and deduplicates multiple candidate unified knowledge units according to unified knowledge unit identifier, source location, and content similarity, and filters candidate unified knowledge units with no access permissions, invalid versions, or resolution confidence below the minimum threshold.

[0189] The evidence gap and conflict diagnosis unit, in the CHECK state, determines one by one whether each fact slot is supported by evidence. For the j-th fact slot in the t-th round, when evidence that can directly and completely support that slot is obtained, let h be the evidence. j,t =1, when only partial information is obtained, let h j,t =0.5, when no evidence is obtained or there are unresolved key conflicts, let h = 0.5. j,t =0; Fact slot coverage Cov in round t t Calculate according to the following formula:

[0190]

[0191] In the above formula, t is the current retrieval round; j is the fact slot index; J is the total number of fact slots, where J is a positive integer; w j w represents the importance weight of the j-th fact slot. j A value greater than zero indicates that key conclusions, key values, statistical definitions, and sources can be set with a larger w. j h j,t The degree of support for the j-th fact slot by the evidence in round t, taking a value of 0, 0.5, or 1; Cov t For the fact slot coverage in round t, Cov t The value is between 0 and 1, with a larger value indicating a more complete coverage of the facts to be answered in the current evidence set;

[0192] The evidence gap and conflict diagnosis unit also generates a key conflict marker X. t When two or more pieces of evidence provide different values, different valid versions, or contradictory conclusions regarding the same entity, the same indicator, and the same time period, and these conclusions cannot be resolved based on source level, approval status, or version validity period, let X... t =1; otherwise let X =1; t =0. This unit also records the maximum prediction generation gain G from the newly added evidence in this round. tAnd the number of newly covered fact slots in this round; when G t Below the preset gain threshold θ g In such cases, the same search criteria should not be used repeatedly in the next round; instead, the search terms, modality, or search scope should be changed.

[0193] The state transition and supplementary retrieval unit determines the next state based on coverage, conflict flag, current round, and remaining budget, according to the following formula:

[0194]

[0195] In the above formula, State t+1 For the state in round t+1, choose one of DONE, REFINE, or STOP; Cov t Let θ be the actual slot coverage obtained in round t; c θ is the evidence coverage threshold. c Located between 0 and 1; X t The key conflict flag for round t is 0 or 1; t is the current search round; T max b is the maximum number of search rounds; t This represents the remaining executable budget after round t; the symbol "∧" represents logical AND, and the symbol "∨" represents logical OR; when Cov t Not less than θ c And X t When =0, enter the DONE state; when Cov... t Below θ c or X t =1, and t is less than T max And b t When the value is greater than 0, the system enters the REFINE state; otherwise, it enters the STOP state.

[0196] When entering the REFINE state, the state transition and supplementary retrieval unit generates new subqueries only around uncovered fact slots or conflict reasons; for example, if the indicator value is obtained but the reason for the decline is missing, only the business analysis report and meeting minutes are retrieved; when there is a conflict in the version of the system, filter conditions such as "latest effective version", "approved", and explicit effective date are added; when the table has values ​​but lacks units, the table header, footnotes, or adjacent table blocks are retrieved. When entering the DONE state, the final evidence set and covered fact slots are output; when entering the STOP state, confirmed facts, uncovered facts, and unresolved conflicts are output.

[0197] See Figure 7 In the gain reordering module M5:

[0198] The training samples and gain label construction units are used to construct the generation gain training labels for candidate unified knowledge units. Under the same generation model, the same prompt template, and the same decoding parameters, the system first generates a baseline answer A0 based on the query q and the current existing evidence set, and then generates an enhanced answer A after adding the i-th candidate unified knowledge unit to the existing evidence set. i The system calculates the answer quality Q(A) from four dimensions: factual accuracy, fact slot completeness, evidence consistency, and uncertainty.

[0199]

[0200] In the above formula, A represents the answer to be evaluated; Q(A) is the overall quality score of answer A, which is between 0 and 1; Acc(A) is the factual correctness score of answer A, Comp(A) is the completeness score of answer A to the fact slot, Cons(A) is the consistency score of answer A with the given evidence, and Unc(A) is the uncertainty score of answer A, all four indicators are between 0 and 1; 1-Unc(A) represents the certainty contribution; ω1, ω2, ω3 and ω4 are the weights of factual correctness, slot completeness, evidence consistency and certainty contribution, respectively, all four weights are between 0 and 1 and satisfy ω1+ω2+ω3+ω4=1;

[0201] The generation gain label ΔG of the i-th candidate unified knowledge unit i Calculate according to the following formula:

[0202]

[0203] In the above formula, i is the candidate unified knowledge unit index; A i A0 is the enhanced answer generated after adding the i-th candidate unified knowledge unit; A0 is the baseline answer generated without adding the candidate unified knowledge unit; Q(A i Q(A0) and Q(A0) are the enhanced answer quality and baseline answer quality obtained according to equation (7), respectively; ΔG i Let ΔG be the true generation gain label for the i-th candidate unified knowledge unit. i Located between -1 and 1; ΔG i A value greater than 0 indicates that the candidate unified knowledge unit can improve the quality of the answer, ΔG i A value close to 0 indicates that the candidate unified knowledge unit overlaps with or is only minimally helpful to existing evidence; ΔG i A value less than 0 indicates that the candidate unified knowledge unit may be irrelevant, outdated, contain conflicting information, or lead to a decrease in the quality of the answer;

[0204] The generation gain prediction unit employs a cross-coding reordering model, combining the standard query, query intent, fact slot, modality of the candidate unified knowledge unit, semantic proxy, structural information, version information, and source information of the candidate unified knowledge unit as joint input, and outputting the predicted generation gain of the i-th candidate unified knowledge unit. The model is trained using mean squared error loss.

[0205]

[0206] In the above formula, L gain To generate the training loss for the gain prediction model, L gain Greater than or equal to 0; M is the number of training samples, where M is an integer greater than 0; i is the index of the training sample; The generation gain of the model for the i-th training sample is preferably limited to between -1 and 1; ΔG i The calculated true generated gain labels; the goal of model training is to minimize L. gain This makes the predicted generation gain close to the actual generation gain;

[0207] During the online phase, the modality adaptation and comprehensive scoring unit calculates the modality adaptation degree Mod of the i-th candidate unified knowledge unit based on the query intent. i Furthermore, semantic relevance, prediction generation gain, modality fit, source and version validity, and degree of duplication with existing evidence are integrated into a comprehensive ranking score S. i :

[0208]

[0209] In the above formula, S i S is the comprehensive ranking score of the i-th candidate unified knowledge unit. i Rel is a real number used for the relative ranking of candidate unified knowledge units; i To determine the semantic relevance between the i-th candidate unified knowledge unit and the structured query, Mod i Val represents the modal fit between the i-th candidate unified knowledge unit and the query intent. i For the source reliability and version validity of the i-th candidate unified knowledge unit, Dup i Rel represents the degree of overlap between the i-th candidate unified knowledge unit and the currently selected evidence set. i Mod i Val i and Dup i Normalize to 0 to 1; The prediction generation gain for the i-th candidate unified knowledge unit is preferably located between -1 and 1; α, β, γ, κ, and δ are the non-negative weights of semantic relevance, prediction generation gain, modality fit, source version validity, and duplication penalty, respectively; since S i For relative sorting only, the sum of the five weights is not required to be 1; a negative sign is set before the repetition level item so that the higher the repetition level, the lower the overall sorting score.

[0210] The evidence selection unit is ranked according to a comprehensive score S. i The top N candidate unified knowledge units are selected from high to low, where N is an integer greater than zero. When the comprehensive scores are close, the candidate unified knowledge units that can add key fact slots, have valid sources, reliable parsing, and can accurately locate the original position are selected first. The ranking results, predicted generation gains, and suggested evidence sets are returned to the iterative retrieval module M4.

[0211] The iterative retrieval module M4 migrates between the states of PLAN, SEARCH, RERANK, CHECK, REFINE, DONE, and STOP, and calls the gain reordering module M5 in the RERANK state; the CHECK state determines the subsequent state based on the fact slot coverage, key conflict flags, retrieval rounds, and remaining budget.

[0212] See Figure 8 In the evidence organization and answer generation module M6:

[0213] The evidence object construction unit is used to assign an evidence number to each piece of evidence in the final evidence set and record the fact slot, modality type, source name, version, page number or timestamp, semantic content, and original location supported by the evidence; for tabular evidence, the evidence object retains the table header, primary key, unit, and cell coordinates; for image evidence, the evidence object retains the object relationship, key area, and OCR result; for audio and video evidence, the evidence object retains the speaker, transcription segment, start time, and end time.

[0214] The structured context assembly unit organizes and generates context in the order of "fact slot - evidence number - modality - source location - evidence content", rather than directly splicing candidate unified knowledge units according to the recall order. For multiple pieces of evidence for the same fact slot, the main evidence that directly supports the conclusion is arranged first, followed by the auxiliary evidence used for cross-validation; when the context length exceeds the input budget of the generative model, fact slots with higher importance and evidence with higher prediction generation gain are retained first.

[0215] The fact anchoring generation unit generates data based on the answers in the structured query object Q. schemaA prompt template should be created, requiring the generated model to answer only based on the given evidence, and to mark the corresponding evidence number after each key value, clause, reason, and conclusion; for the direct answer path where D(q)=0, only questions that do not involve corporate private facts, time-sensitive facts, designated sources, and high-risk business are allowed to be answered; when evidence is insufficient or conflicting, the generated model must clearly state the missing or conflicting items, and may not fill in corporate facts that are not supported by sources.

[0216] See Figure 9 In the answer verification and feedback control module M7:

[0217] The source permission version verification unit is used to check whether the evidence number cited in the initial answer exists, whether the current user has access rights, whether the source anchor is valid, and whether the corresponding document version is valid. The numerical semantic conflict verification unit is used to trace the numerical values, units, times, and calculation relationships in the answer back to table cells, OCR areas, or original text fragments to determine whether the evidence directly supports the corresponding conclusion and to check whether the same conclusion cites contradictory sources.

[0218] When the answer correction and review control unit detects missing references, inconsistent values, lack of direct support for conclusions, or conflicting key evidence, it first deletes, downgrades, or rewrites the relevant conclusions based on existing evidence; if the missing conclusion is a key fact and the remaining budget b is found in the retrieval status object... t If the remaining budget is greater than 0, the missing fact slot, conflict cause, and existing evidence status will be returned to the iterative retrieval module M4 for supplementary retrieval; if the remaining budget is 0, the confirmed facts will be retained and the insufficient evidence status will be output.

[0219] After verification, the final output unit outputs the user-facing answer, the evidence number corresponding to each key conclusion, the document name, page number, table coordinates, image area or audio / video timestamp, covered facts, uncovered facts, and explanations of any remaining conflicts.

[0220] Evidence organization, fact anchoring generation, and answer verification form a closed loop; when verification finds insufficient key evidence or unresolved conflicts and there is still budget, the answer verification and feedback control module M7 returns the missing fact slot and conflict cause to the iterative retrieval module M4; when there is no remaining budget, the confirmed facts and insufficient evidence status are output.

[0221] See Figure 10 This paper introduces a multimodal retrieval enhancement generation method for enterprise-level heterogeneous knowledge based on the aforementioned system. The method includes the following steps:

[0222] Step S100: Through the knowledge resource construction module M1, access the heterogeneous knowledge sources of the enterprise, register the document primary key, version and access permissions of the knowledge sources, perform structure-preserving parsing on knowledge sources of different modalities, and build a unified knowledge unit as well as keyword index, vector index and metadata index;

[0223] Step S200: Through the generation gain re-ranking module M5, training samples are constructed based on historical question-and-answer, manual test questions and business cases. The changes in answer quality before and after adding candidate unified knowledge units are compared to generate generation gain labels for candidate unified knowledge units. A generation gain prediction model for online ranking of candidate unified knowledge units is trained.

[0224] Step S300: By querying the structured module M2, perform referential resolution, terminology normalization, spatiotemporal constraint extraction, and fact slot decomposition on the current user's question, historical messages, historical evidence, and the chart or media object selected by the user, and output the structured query object.

[0225] In step S400, the retrieval gate module M3 determines whether to initiate an enterprise knowledge base retrieval based on the mandatory retrieval conditions and the model knowledge sufficiency calculation results. If the retrieval decision is not to retrieve, proceed to step S600. If the retrieval decision is to retrieve, generate a retrieval plan and proceed to step S500.

[0226] Step S500: Through the iterative retrieval module M4 and the generation gain reordering module M5, perform hybrid recall, candidate unified knowledge unit merging and filtering, generation gain prediction, modality-aware reordering, evidence coverage check, evidence conflict diagnosis and gap filling retrieval around the fact slot to be covered, until the completion condition is met or the stopping condition is reached.

[0227] In step S600, the final evidence is organized into a structured context that retains the source anchor point through the evidence organization and answer generation module M6 and the answer verification and feedback control module M7, and the fact-anchored answer is generated. The source, permissions, version, value, unit, time, semantic support relationship and evidence conflict in the answer are verified one by one. If the key evidence is insufficient and there is still a search budget, the process returns to step S500 to perform a supplementary search. Otherwise, the final answer and the corresponding evidence source are output.

[0228] Step S100 specifically includes the following steps:

[0229] Step S110: Receive enterprise knowledge sources such as PDFs, Word documents, web pages, Excel files, images, audio, and video through the resource access and version permission registration unit; obtain file identifiers, content summaries, version numbers, modification times, source paths, and access control information; generate document primary keys based on file identifiers, content summaries, version numbers, and modification times; if the document primary key and content summary are identical, it is determined to be a duplicate resource and duplicate entries are skipped; when a new version is detected, an inheritance or replacement relationship between the new and old versions is established, and expired versions are marked as non-default recalls; convert user, department, role, document, and field-level permissions into permission metadata.

[0230] Step S120: The structure-preserving parsing unit performs corresponding parsing based on the knowledge source modality: For PDFs, Word documents, web pages, and scanned documents, it identifies page layout areas such as titles, body text, lists, headers and footers, footnotes, images, and tables, restores the reading order, and segments the document by chapter, paragraph, list item, and semantic transition; for complex tables, it sequentially performs merged cell expansion, record direction recognition, row and column transposition when necessary, multi-level header compression, unit and primary key binding, multi-granular segmentation, and cell coordinate recording; for images, it performs OCR, region detection, and visual semantic description; for audio and video, it performs time-stamped speech recognition, speaker segmentation, scene transition detection, keyframe extraction, and audio-visual alignment. When the parsing confidence is lower than a preset threshold, it simultaneously retains simplified semantic proxy and the original image, original file reference, or original media time interval.

[0231] Step S130: By encapsulating text blocks, table blocks, image semantic proxies, audio segments, and video spatiotemporal slices into unified knowledge units through a unified knowledge unit encapsulation unit, and writing a unique identifier, modality type, original content or original file reference, semantic proxy, structural information, business time or media time, source anchor, version validity period, access permissions, and vector representation for each unified knowledge unit.

[0232] Step S140: By constructing a unit through a hybrid index, a keyword inverted index is built for the semantic agent, table header, unit, entity and enterprise terminology of the unified knowledge unit. Vectors are generated by embedding model and vector indexes are built. Metadata indexes are built for permissions, version, validity period, department, document type, time range, page number and media timestamp. The three types of indexes all point to the same unified knowledge unit identifier. During online recall, the initial recall score is calculated and the results from multiple paths are merged and deduplicated.

[0233] Step S200 specifically includes the following operation steps:

[0234] Step S210: Obtain training questions from enterprise historical Q&A, manual test questions, and business cases; form structured queries and fact slots through the query structure module M2; and recall candidate unified knowledge units in text, table, image, audio, and video modalities through the iterative retrieval module M4; each training sample includes at least a structured query, fact slot, current existing evidence, semantic proxy of the candidate unified knowledge unit, modality, structural information, source location, version information, and time information of the candidate unified knowledge unit; retain candidate unified knowledge units with high similarity but no answer, duplicate existing evidence, outdated version, or factual conflict as difficult negative samples;

[0235] Step S220: Under the same generative model, the same prompt template, and the same decoding parameters, generate a baseline answer A0 based on the training question and the existing evidence. Then, add the i-th candidate unified knowledge unit to the existing evidence to generate an enhanced answer A. i Calculate the baseline answer quality Q(A0) and the enhanced answer quality Q(A0) respectively. i Then calculate the generation gain label ΔG for the i-th candidate unified knowledge unit. i The correctness of facts is determined by the standard answer, knowledge base rules, or manual verification results; the completeness of fact slots is determined by the fact slots in the structured query; the consistency of evidence is determined by whether the candidate unified knowledge unit directly supports the answer conclusion; and the uncertainty is determined by fuzzy expressions, internal contradictions in the answer, and source conflicts.

[0236] Step S230: Combine the standard query, query intent, fact slot, modality of candidate unified knowledge unit, semantic proxy of candidate unified knowledge unit, structural information, source information and version information into the joint input of the generation gain prediction model, so that the model outputs the predicted generation gain of candidate unified knowledge unit; calculate the training loss and update the model parameters until the preset convergence condition is reached; select the model parameters with smaller generation gain prediction error and better ranking effect of candidate unified knowledge unit on the validation set, and write the selected model parameters into the model parameter storage area;

[0237] In step S240, the generated gain prediction model trained in step S230 is deployed to the generated gain reordering module M5 for online use in step S530. The manual review results, answer verification results, and newly added business data are used as new training samples, and incremental training or retraining is performed on the same generated gain prediction model according to a preset cycle. The updated model parameters replace the original model parameters in the model parameter storage area and are then used in subsequent steps S530.

[0238] Step S300 specifically includes the following operation steps:

[0239] Step S310: Using the context object positioning unit, determine the object identifier and source anchor point corresponding to the delivery of indicators such as "this chart", "previous table", "this product", and "yesterday's meeting" from historical messages, historical evidence objects and the chart or media object currently selected by the user; when there are multiple candidate objects and the highest matching confidence is lower than the preset threshold, retain multiple candidate objects for processing or output clarification requests separately;

[0240] Step S320: Through the terminology and spatiotemporal constraint standardization unit, the abbreviations of departments, product codes, indicator aliases and document abbreviations are mapped to standard names according to the enterprise terminology table. Relative times such as "latest", "current", "this quarter" and "yesterday" are converted into specific time points or time ranges, and constraints such as department, region, version, permission, required modality and answer format are extracted.

[0241] Step S330: Through the fact slot decomposition unit, the complex problem is broken down into fact slots that can be independently retrieved and verified according to the query intent; each fact slot includes at least the slot name, target entity, time condition, expected data type, priority modality and slot importance.

[0242] In step S340, the structured query generation unit encapsulates the standard query, query intent, standardized entity, constraint condition, priority mode, timeliness requirement, fact slot set and answer structure into a structured query object Q, and sends the structured query object to the retrieval gating module M3.

[0243] Step S400 specifically includes the following operation steps:

[0244] Step S410: The forced retrieval judgment unit determines whether the structured query involves internal corporate policies, corporate business data, unpublished information, the latest or specified version, a specified source, page numbers, table coordinates, media timestamps, facts that must be answered based on images or audio / video, or preset high-risk business such as finance, compliance, and security; if any condition is met, the forced retrieval flag F is activated. q =1, otherwise let F =1. q =0;

[0245] Step S420, when F q When =0, candidate answers are generated for internal evaluation only, and the confidence C of the answers is calculated respectively. q Fact slot completeness P q and uncertainty risk U q And calculate the knowledge sufficiency K. q When F q When =1, regardless of whether internal candidate answers can be generated, the forced retrieval state is directly retained;

[0246] Step S430, calculate the retrieval decision D(q); when F q =0 and K q Not lower than the knowledge sufficiency threshold θ k When D(q) is set to 0, proceed to step S600; otherwise, set D(q) to 1, generate the retrieval triggering reason, the fact slot to be covered, the priority mode, the retrieval range and the filtering conditions through the retrieval plan generation unit, initialize the retrieval state object R0 and proceed to step S500.

[0247] Step S500 specifically includes the following operation steps:

[0248] Step S510: Establish the retrieval state object R for the t-th round through the retrieval state maintenance unit. t Record the current round t, the structured query object Q, the set of fact slots, and the set of selected evidence E. t Candidate unified knowledge unit set C t Uncovered slots, conflict information, used search criteria, maximum search rounds T max Remaining executable budget b t And the current state; the initial state is set to PLAN;

[0249] Step S520: In the PLAN state, generate one or more subqueries for each uncovered fact slot. Each subquery includes at least the target entity, time condition, keywords, priority modality, and metadata filtering conditions. Then, enter the SEARCH state. First, filter unified knowledge units according to permissions, version, time, department, and document type. Then, perform keyword retrieval and vector retrieval in parallel and calculate the initial recall score. For each subquery, obtain the top K0 candidate unified knowledge units and merge and deduplicate them according to unified knowledge unit identifier, source location, and content similarity. Filter out candidate unified knowledge units with no access permissions, expired versions, or parsing confidence below the minimum threshold to form the t-th round candidate unified knowledge unit set C. t ;

[0250] Step S530: Combine the structured query object Q and the previously selected evidence set E. t-1 and the set of unified knowledge units C for candidates in round t t Input the generative gain re-ranking module M5, and output the predicted generative gain for each candidate unified knowledge unit from the generative gain prediction model trained in step S200; calculate the semantic relevance, modality fit, source and version validity, and degree of duplication with existing evidence for the candidate unified knowledge unit, and calculate the comprehensive ranking score S. i The top N candidate unified knowledge units are selected from high to low based on their comprehensive ranking scores, and priority is given to retaining candidate unified knowledge units that can add key fact slots, have valid sources, reliable parsing, and can accurately locate the original position.

[0251] Step S540: Enter the CHECK state and determine the support level h for the j-th fact slot in round t. j,t The complete evidence corresponds to h. j,t =1, partial evidence corresponds to h j,t =0.5, no evidence obtained or unresolved key conflicts corresponding to h j,t =0; Calculate the fact slot coverage Cov in round t. t When two or more pieces of evidence provide different values, valid versions, or contradictory conclusions regarding the same entity, the same indicator, and the same time period, and these cannot be resolved based on source level, approval status, or version validity period, then the key conflict indicator X is set. t =1, otherwise let X =1. t =0;

[0252] Step S550: Record the maximum prediction generation gain and the number of newly added covered fact slots in the newly added evidence in this round; when the maximum prediction generation gain is lower than the preset gain threshold θ g In such cases, the same search criteria must not be reused in the next round; instead, the search keywords, search modality, or search scope should be changed. When low gain occurs consecutively and no new fact slots are added, the remaining executable budget b will be used. t Set to 0; then determine the next state;

[0253] In step S560, when the next state is REFINE, generate a new subquery only around the uncovered fact slots or the reasons for evidence conflicts and return to step S520; when the next state is DONE, output the final evidence set, the covered fact slots and the source anchor and proceed to step S600; when the next state is STOP, output the confirmed facts, the uncovered facts and the unresolved conflicts, and proceed to step S600 in a conservative manner.

[0254] Step S600 specifically includes the following operation steps:

[0255] Step S610: Assign an evidence number to each piece of evidence in the final evidence set through the evidence object construction unit, and record the fact slot, modality type, source name, version, page number or timestamp, semantic content and original location supported by the evidence; table evidence retains the table header, primary key, unit and cell coordinates; image evidence retains the object relationship, key area and OCR result; audio and video evidence retains the speaker, transcription segment and start time and end time.

[0256] Step S620: The context is organized in the order of "fact slot - evidence number - modality - source location - evidence content" through the structured context assembly unit; under the same fact slot, the main evidence that directly supports the conclusion is arranged first, and then the auxiliary evidence used for cross-validation is arranged; a prompt template is generated according to the answer structure requirements in the structured query object Q, requiring the generated model to answer only based on the given evidence, and the corresponding evidence number is marked after each key value, clause, reason and conclusion, thereby generating the initial answer; when D(q)=0, direct answers are only performed for questions that do not involve enterprise private facts, time-sensitive facts, designated sources and high-risk businesses;

[0257] Step S630: The initial answer is broken down into several key conclusions and a conclusion-evidence number mapping is established. The source existence verification, access permission verification, version validity verification, numerical value and unit verification, time verification, semantic support verification, and evidence conflict verification are performed in sequence. For non-key conclusions that do not exist, have inconsistent numerical values, lack direct support, or have evidence conflicts, they are deleted, downgraded, or rewritten based on existing evidence.

[0258] Step S640, when the missing or conflicting conclusion pertains to key facts and the remaining executable budget b t When the value is greater than 0, the missing fact slot, conflict cause, and existing evidence status will be returned to step S500 to perform supplementary retrieval; when the remaining budget is 0 or there are no key facts that need to be supplemented, the retrieval feedback will be stopped.

[0259] Step S650 outputs the final answer for the user, the evidence number corresponding to each key conclusion, document name, page number, table coordinates, image area or audio / video timestamp, covered facts, uncovered facts, and explanations of any remaining conflicts; the manual review results and the verification status generated in this step are recorded as updated samples that can be used when executing step S240 in the future.

[0260] Parsing failure handling: When the confidence level of table structure recognition, image OCR or speech recognition is lower than the preset threshold, the knowledge resource construction module M1 simultaneously retains the original file, original image or original audio and video range and simplifies semantic proxy, and reduces the parsing reliability, without directly deleting the evidence;

[0261] Modality missing handling: When a query explicitly requires a certain modality but the shared knowledge storage does not contain that modality, the iterative retrieval module M4 outputs a modality gap; other modalities can be retrieved as supplementary evidence, but textual speculations must not be presented as non-existent visual or audio-visual facts;

[0262] Empty Recall Handling: When the initial recall is empty, the iterative retrieval module M4 sequentially expands the entity aliases, synonyms, and allowed time range, and changes the retrieval modality or index combination; if no candidate unified knowledge unit is still obtained, it enters the STOP state and outputs that insufficient evidence was not retrieved.

[0263] Handling of conflicting evidence: Prioritize evidence with updated validity, clear approval status, higher source level, and verifiability by other sources; if conflicting evidence still cannot be resolved, present the conflicting content and corresponding sources in parallel, and reduce the certainty of the conclusion.

[0264] Permissions and budget handling: Access permission filtering is performed before recall, and the final answer only shows the content and source that the current user is authorized to access; when the maximum retrieval rounds or computation budget are exhausted, confirmed conclusions, uncovered facts and unresolved conflicts are output based on the current fact slot coverage, and further calls are stopped;

[0265] Model or tool anomaly handling: When the embedded model, OCR tool, speech recognition tool, reordering model, or generation model fails to be invoked, a backup model of the same type is invoked first or the corresponding degradation processing is adopted; if the backup model is still unavailable, the current retrieval status and confirmed facts are retained, the reason for the anomaly, the incomplete steps, and the retry flag are output, and no conclusions that are not supported by evidence are generated.

[0266] The inventors conducted numerous experiments on the proposed solution, and the experimental results are as follows:

[0267] The inventors conducted experiments on the publicly available table-text joint question-answering dataset TAT-QA (https: / / github.com / NExTplusplus / TAT-QA), the publicly available document-image question-answering dataset DocVQA (https: / / www.docvqa.org / datasets / docvqa), the publicly available cross-modal question-answering dataset MultiModalQA (https: / / github.com / allenai / multimodalqa), and the private dataset EHMK built in a real-world production environment. EHMK is an enterprise heterogeneous knowledge anonymization dataset containing business reports, policy documents, Excel indicator tables, dashboard screenshots, flowcharts, meeting recordings, and business videos; it is not publicly available. Basic information about the datasets involved is shown in Table 2.

[0268] Table 1. Basic Information of the Dataset

[0269] TAT-QA 16552 2757 mixed contexts Text, Table DocVQA 50000 12,767 document images Text, images MultiModalQA 29918 29,918 question and answer samples Text, tables, images EHMK 4318 1627 enterprise knowledge documents Text, tables, images, audio, video

[0270] To address the core requirements of enterprise-level heterogeneous knowledge question answering, the experimental task was set as follows: Based on user questions, relevant unified knowledge units were retrieved from a multimodal knowledge base; combined reasoning was performed on content involving numerical values, clauses, reasons, versions, times, and sources; answers were generated; and key conclusions were bound to corresponding evidence locations. The experiment used the following metrics for evaluation: Recall@10 (top 10 evidence points), Normalized Diminished Cumulative Gain nDCG@10 (normalized loss cumulative gain), Answer Acc (answer accuracy), Answer F1 score, Evidence Precision EP (evidence accuracy), and Citation Location Accuracy CA (citation location accuracy). The closer each metric is to 1, the better the system performance.

[0271] The proposed solution is compared with single-round vector RAG, hybrid recall RAG, and semantic re-ranking RAG. Single-round vector RAG performs a single retrieval using only vector similarity; hybrid recall RAG uses keyword retrieval and vector retrieval in parallel; semantic re-ranking RAG ranks candidates based on the semantic relevance of the query to the candidate evidence, building upon hybrid recall; the proposed solution further employs retrieval gating, fact slot decomposition, iterative supplementary retrieval, generative gain re-ranking, and answer verification feedback mechanisms. Experimental results are shown in Table 3.

[0272] Table 2 Experimental results of different methods

[0273] TAT-QA Single-round vector RAG 0.7882 0.6814 0.5748 0.6437 0.7421 0.7215 Mixed Recall RAG 0.8364 0.7448 0.6219 0.6872 0.8075 0.7908 Semantic Reordering RAG 0.8721 0.7895 0.6574 0.7153 0.8468 0.8324 This invention 0.9163 0.8547 0.7062 0.7614 0.9149 0.9291 DocVQA Single-round vector RAG 0.7641 0.6517 0.6134 0.6998 0.7246 0.6842 Mixed Recall RAG 0.8167 0.7192 0.6588 0.7431 0.7843 0.7557 Semantic Reordering RAG 0.8502 0.7636 0.6911 0.7705 0.8204 0.8034 This invention 0.9048 0.8345 0.7519 0.8237 0.8956 0.9198 MultiModalQA Single-round vector RAG 0.6948 0.5831 0.4476 0.5242 0.6907 0.6417 Mixed Recall RAG 0.7506 0.6438 0.4947 0.5689 0.7469 0.7035 Semantic Reordering RAG 0.7961 0.6995 0.5384 0.6103 0.7868 0.7521 This invention 0.8642 0.7851 0.6149 0.6868 0.8785 0.8907 EHMK Single-round vector RAG 0.7163 0.5918 0.5361 0.5992 0.6987 0.6638 Mixed Recall RAG 0.7815 0.6684 0.6028 0.6611 0.7703 0.7476 Semantic Reordering RAG 0.8247 0.7229 0.6498 0.7027 0.8145 0.7952 This invention 0.9136 0.8488 0.7594 0.8079 0.9253 0.9491

[0274] Experimental results show that the proposed solution achieves optimal values ​​across all evaluation metrics on three public and private datasets. Compared to the suboptimal semantic reordering RAG method, on the TAT-QA, DocVQA, and MultiModalQA datasets, the proposed solution improves answer accuracy by 4.88, 6.08, and 7.65 percentage points, respectively, and F1 scores by 4.61, 5.32, and 7.65 percentage points, respectively. On the private dataset EHMK, the solution improves answer accuracy by 10.96 percentage points, F1 score by 10.52 percentage points, evidence accuracy by 11.08 percentage points, and citation localization accuracy by 15.39 percentage points.

[0275] The above results show that the present invention, by maintaining the structure and source information of text, tables, images, and audio-visual knowledge, using retrieval gating to control retrieval timing, performing supplementary retrieval around uncovered fact slots, and reordering according to the predicted gain of candidate evidence on answer quality, can reduce duplicate evidence, invalid versions, and conclusions lacking direct support, thereby improving the evidence recall effect, answer correctness, evidence support degree, and source location accuracy in complex question answering.

[0276] Experimental results demonstrate that the method proposed in this invention is feasible and effective.

[0277] The present invention has been described in detail above with reference to the embodiments. Those skilled in the art can make various modifications to the present invention based on the above description. Therefore, certain details in the embodiments should not be construed as limiting the present invention. The scope of protection of the present invention shall be defined by the appended claims.

Claims

1. A multimodal retrieval and enhancement generation system for enterprise-level heterogeneous knowledge, characterized in that: The system includes the following modules: Knowledge Resource Construction Module M1: This module is used to access heterogeneous knowledge sources within the enterprise, such as PDFs, Word documents, web pages, Excel files, images, audio, and video. It performs version registration, permission registration, and structure-preserving parsing on these heterogeneous knowledge sources, and establishes unified knowledge units and hybrid indexes for online retrieval. The module's inputs are enterprise knowledge source files, media streams, and version and access control information associated with the knowledge sources. The outputs are unified knowledge units, keyword indexes, vector indexes, and metadata indexes. The outputs are written to a shared knowledge storage area and serve as the basis for retrieval by the iterative retrieval module M4. This module includes a resource access and version permission registration unit, a structure-preserving parsing unit, a unified knowledge unit, an encapsulation unit, and a hybrid index construction unit. The query structuring module M2: This module converts the user's current question, historical dialogues, historical evidence, and user-selected charts or media objects into structured queries that can directly perform retrieval and evidence integrity judgment. The inputs to this module are the current user question, historical messages within a preset window, historical evidence objects, currently selected objects on the interface, and a glossary of enterprise terms. The output is a structured query object Q. This structured query object Q serves as the input to the retrieval gating module M3 and the iterative retrieval module M4. This module includes a context object location unit, a terminology and spatiotemporal constraint normalization unit, a fact slot decomposition unit, and a structured query generation unit. The retrieval gating module M3 is used to determine whether the current question requires access to the enterprise knowledge base and to generate a retrieval plan when necessary. Its inputs are the structured query object Q output by the query structuring module M2, the mandatory retrieval rules, and the knowledge sufficiency assessment results of internal candidate answers. Its outputs are the retrieval decision D(q), the retrieval trigger reason, the fact slots to be covered, the priority retrieval modality, and the filtering conditions. When D(q) = 1, the output of this module serves as the initial input to the iterative retrieval module M4; when D(q) = 0, the output serves as the direct answer constraint for the evidence organization and answer generation module M6. This module includes a mandatory retrieval determination unit, a knowledge sufficiency assessment unit, and a retrieval plan generation unit. Iterative Retrieval Module M4: This module performs one or more rounds of retrieval around the fact slots in the structured query, and determines whether to continue, complete, or stop the retrieval based on evidence coverage, evidence conflict, retrieval rounds, and remaining budget. The inputs to this module are the retrieval plan output by the retrieval gating module M3, the indexes and unified knowledge units in the shared knowledge storage area, and the gap information returned by the answer verification and feedback control module M7. The outputs are the current candidate unified knowledge unit set, the retrieval status object, and the final evidence set. The current candidate unified knowledge unit set is sent to the gain reordering module M5, and the final evidence set is sent to the evidence organization and answer generation module M6. This module includes a retrieval status maintenance unit, a subquery planning and hybrid recall unit, a candidate unified knowledge unit merging and filtering unit, an evidence gap and conflict diagnosis unit, and a state transition and supplementary retrieval unit. The Gain Re-ranking Module M5 is used to evaluate the actual gain on the quality of the final answer after the candidate unified knowledge units (CQUs) constructed by the Knowledge Resource Construction Module M1 and recalled by the Iterative Retrieval Module M4 are added to the current evidence set, and to rank the multimodal candidate CQUs accordingly. The inputs to this module are a structured query object Q, the currently selected evidence set, and the current candidate CQU set. The outputs are the predicted gain, overall ranking score, and suggested evidence set for each candidate CQU. This output is returned to the Iterative Retrieval Module M4, which uses it for evidence selection, fact slot coverage determination, and next state determination. This module includes a training sample and gain label construction unit, a gain prediction unit, a modality adaptation and overall scoring unit, and an evidence selection unit. Evidence Organization and Answer Generation Module M6: This module transforms the final evidence set into a structured context centered on fact slots and retaining source anchors, and generates initial answers that can bind key conclusions to evidence item by item. The inputs to this module are a structured query object Q, the final evidence set, and the answer structure requirements. The output is the initial answer and the conclusion-evidence mapping relationship. This output serves as the input to the answer verification and feedback control module M7. This module includes an evidence object construction unit, a structured context assembly unit, and a fact anchoring generation unit. Answer Verification and Feedback Control Module M7: This module verifies the source, access permissions, document version, numerical values, units, time, semantic support relationships, and evidence conflicts in the initial answer, and controls answer correction, supplementary retrieval, or final output based on the verification results. The inputs to this module are the initial answer, the conclusion-evidence mapping relationship, the final evidence set, and the retrieval status object. The outputs are the final answer, evidence source, uncovered facts, conflict explanation, and verification status. When key conclusions lack evidence but there is still retrieval budget, this module also outputs the missing slots and conflict reasons and returns to the iterative retrieval module M4. This module includes a source permission version verification unit, a numerical semantic conflict verification unit, an answer correction and re-check control unit, and a final output unit. During the online question-and-answer process, the structured query objects output by the query structure module M2 are used by the retrieval gating module M3 to form a retrieval decision. When a retrieval is needed, the iterative retrieval module M4 performs recall based on the hybrid index formed by the knowledge resource construction module M1, and completes the ranking of candidate unified knowledge units in each round of calling the gain generation and reordering module M5. After the retrieval is completed, the evidence organization and answer generation module M6 generates an initial answer based on the final evidence, and the answer verification and feedback control module M7 verifies the answer and conclusion-evidence mapping. When M7 finds that key evidence is missing or there are unresolved conflicts and there is still a retrieval budget, it returns the gap information to M4 to perform supplementary retrieval. When the verification is passed or the budget is exhausted, the final result is output, thus forming a closed loop of "retrieval - reordering - generation - verification - supplementary retrieval". The knowledge resource construction module M1 and the shared knowledge storage area constitute the knowledge construction link, while modules M2 to M7 constitute the online question-and-answer link. The online question-and-answer link selects either a direct answer path or an enterprise knowledge base retrieval path based on the retrieval decision, and feeds back the missing fact slots and conflict reasons to the iterative retrieval module M4 through the answer verification results.

2. The system according to claim 1, characterized in that: The knowledge resource construction module M1 processes enterprise knowledge sources in the order of "resource registration - modal parsing - unified knowledge unit encapsulation - hybrid index construction", so that the keyword index, vector index and metadata index all point to the same unified knowledge unit, and retain version, permission and source location information; In the knowledge resource construction module M1: The resource access and version permission registration unit consists of a connector adaptation subunit, a document primary key generation subunit, a version relationship subunit, and a permission mapping subunit. The connector adaptation subunit is used to obtain source files or media streams from file systems, document management platforms, object storage, databases, or media platforms. The document primary key generation subunit is used to generate document primary keys based on file identifiers, content summaries, version numbers, and modification times to identify duplicate files. The version relationship subunit records the inheritance, replacement, and expiration relationships between different versions and marks expired versions as non-default recalls. The permission mapping subunit converts user, department, role, document, and field-level access control rules into searchable permission metadata. The resource access and version permission registration unit outputs a resource registration object with document primary key, source path, version status, validity period, and access permissions. The structure preservation parsing unit includes a text layout parsing subunit, a complex table parsing subunit, an image parsing subunit, and an audio / video parsing subunit; The text layout parsing subunit is used to identify layout areas such as titles, body text, lists, headers, footers, footnotes, images, and tables. It restores the relationship between these areas according to the reading order, removes duplicate headers and footers, watermarks, and blank areas, and performs block segmentation according to chapters, paragraphs, list items, and semantic transitions. When a single semantic block exceeds the preset maximum length L, a secondary segmentation with overlapping windows is used, and the preceding block, succeeding block, chapter path, page number, and paragraph number are recorded. Here, L is a positive integer used to limit the maximum number of characters or the maximum number of words in a single text block. The complex table parsing subunit is used to sequentially perform merged cell expansion, table direction recognition, multi-level header compression, unit and primary key recognition, multi-granularity segmentation, and source location; merged cell expansion propagates the original value across rows or columns to the base cell it covers, while preserving the original coordinates of the merged area; Table orientation recognition determines the record orientation based on the text ratio, numerical ratio, repetition pattern, and data type distribution of the first row and column, and performs row and column transposition when necessary; multi-level header compression forms unique column names based on the parent-child hierarchy of the header; unit and primary key recognition binds statistical categories such as amount, ratio, quantity, date, region, and product with corresponding values. Multi-granular segmentation forms a complete table summary, sub-table blocks, and row blocks; source location records the table number, worksheet name, page number, row and column range, and cell coordinates. When the structure recognition confidence level is lower than a preset threshold, the OCR text, original table image, and original file reference are retained to avoid losing original evidence due to structure recognition failure; The image parsing subunit includes OCR processing, layout area detection, and visual semantic description. It is used to extract text, main objects, object relationships, coordinate axes, legends, key values, trends, and directly obtainable conclusions from flowcharts, equipment diagrams, dashboards, and screenshots, and to record image page numbers, area coordinates, and parsing confidence. The audio and video parsing subunit is used to perform timestamped speech recognition, speaker segmentation, scene transition detection, keyframe extraction, and audio-visual alignment. For audio, this subunit forms semantic segments with speakers and start and end times according to topic changes or silence intervals; for video, this subunit merges keyframe OCR results, visual descriptions, and speech transcriptions within the same time window into spatiotemporal slices, and records media file identifiers, start times, end times, speakers, and keyframe numbers. The unified knowledge unit encapsulation unit is used to encapsulate text blocks, table blocks, image semantic proxies, audio segments, and video spatiotemporal slices into unified knowledge units; the i-th unified knowledge unit u i Represented as: In the above formula, i is the index of the unified knowledge unit, and i can take values ​​from 1 to N. u For any positive integer N, u The total number of unified knowledge units in the system; id i m is the unique identifier for the i-th unified knowledge unit; i The modal type can be one of TEXT, TABLE, IMAGE, AUDIO, or VIDEO; x i This refers to the original content, a fragment of original data, or a reference to the original file; p i Semantic proxy text for keyword retrieval and vector retrieval; s i For structural information, text includes chapter paths, page numbers, and paragraph numbers; tables include table headers, primary keys, units, and row and column coordinates; images include region coordinates and object relationships; and audio and video include speaker and keyframe information. t i For business time or media time information, it can represent a single point in time or a time interval consisting of start and end times; i Source anchors include document name, file path, page number, table coordinates, image region, or media time interval; v i For version number, version status, and validity period; a i For access permission set; e i Let e ​​be the vector representation of the i-th unified knowledge unit. i Belonging to the d-dimensional real vector space R d d is the output dimension of the embedded model; The hybrid index building unit consists of a keyword index subunit, a vector index subunit, and a metadata index subunit. The keyword index subunit builds an inverted index for semantic proxies, table headers, units, entities, and enterprise terms. The vector index subunit adds modal identifiers and structural summaries before the semantic proxies, generates vectors through an embedding model, and builds a vector index. The metadata index subunit builds filtering indexes for permissions, versions, validity periods, departments, document types, time ranges, page numbers, and media timestamps. During online recall, the system first performs metadata filtering, then performs keyword retrieval and vector retrieval in parallel, and calculates the initial recall score B of the i-th candidate unified knowledge unit using the following formula. i : In the above formula, i is the index of the candidate unified knowledge unit; B i B is the initial recall comprehensive score for the i-th candidate unified knowledge unit. i Located between 0 and 1; B i kw B is the keyword matching score for the i-th candidate unified knowledge unit. i vec B is the vector similarity score for the i-th candidate unified knowledge unit. i meta The metadata matching score for the i-th candidate unified knowledge unit is normalized to 0 to 1 before weighting. η1, η2, and η3 are the weights of the keyword score, vector score, and metadata score, respectively, all of which are between 0 and 1 and satisfy η1 + η2 + η3 = 1. For numerical and statistical queries, increase the weight of η1 and the keywords corresponding to the table header, indicator name, and unit; for time-sensitive queries, increase the weight of η3 and the metadata corresponding to the version and time conditions.

3. The system according to claim 1, characterized in that: In the query structure module M2: The context object positioning unit is used to determine the object identifier and source anchor point corresponding to the target message from historical messages, historical evidence objects, and the currently selected chart or media object by the user. When there are multiple candidate objects and the highest matching confidence is lower than a preset threshold, the system retains multiple candidate objects for separate retrieval, or outputs a clarification request; The terminology and spatiotemporal constraint normalization unit is used to map department abbreviations, product codes, indicator aliases and document abbreviations to standard names based on the enterprise terminology table, and to convert relative times such as "latest", "current", "this quarter" and "yesterday" into specific time points or time ranges, while extracting constraints such as department, region, version, permissions, required modality and answer format; The fact slot decomposition unit is used to break down complex questions into fact slots that can be independently retrieved and verified according to the query intent; each fact slot includes at least the slot name, target entity, time condition, expected data type, priority modality, and slot importance. The structured query generation unit is used to encapsulate the above processing results into a structured query object Q; A structured query object Q must include at least the standard query q. std The query intent, standardized entities, constraints, preferred modalities, freshness requirement, fact slots, and answer structure are all specified. schema ; where q std To address the semantic self-containment problem arising from omitting entities and time, answer schema This is used to specify the conclusions, comparative data, reasons, conditions, and source formats that the answer should include.

4. The system according to claim 1, characterized in that: In the retrieval gating module M3: The forced retrieval determination unit sets a forced retrieval flag F. q When a structured query involves internal company policies, business data, unpublished information, the latest or specified version, a specified source, page numbers, table coordinates, media timestamps, facts that must be answered based on images or audio / video, or high-risk business matters such as finance, compliance, and security, then F is allowed to... q =1; otherwise, let F =1; q =0;F q When =1, regardless of whether the large language model can generate candidate answers, the system will enter the enterprise knowledge base retrieval path; When F q When the answer score is 0, the knowledge sufficiency assessment unit generates candidate answers for internal evaluation only and calculates the answer confidence score C. q Fact slot completeness P q and uncertainty risk U q Then, the knowledge sufficiency K of query q is calculated according to the following formula. q : In the above formula, q represents the current structured query; K q K represents the model's knowledge sufficiency regarding query q. q The value is between 0 and 1, with a higher value indicating a higher probability of being able to answer correctly and completely without external evidence; C q P represents the confidence level of the internal candidate answers. q U represents the complete coverage of fact slots by internal candidate answers. q For the uncertainty risk of internal candidate answers, C q P q and U q All are between 0 and 1; 1-U q Used to convert uncertainty risk into certain contribution; λ1, λ2 and λ3 are the weights of answer confidence, fact slot completeness and certain contribution, respectively, all of which are between 0 and 1 and satisfy λ1+λ2+λ3=1; The retrieval gating module M3 is based on F q and K q The retrieval decision D(q) is obtained by the following formula: In the above formula, D(q) represents the retrieval decision for query q, and D(q) can be 0 or 1; D(q) = 0 indicates that the enterprise knowledge base will not be searched, and the evidence organization and answer generation module M6 will generate an answer according to the direct answer constraint; D(q) = 1 indicates that the iterative retrieval module M4 will be entered; F q This is a forced search flag, and can be either 0 or 1; K q The calculated knowledge sufficiency; θ k θ is the knowledge sufficiency threshold. k Located between 0 and 1; the symbol "∧" represents logical AND. F ∈ F. q =0 and K q Not less than θ k When , D(q) = 0; in other cases, including F q =1 or K q Below θ k Let D(q) = 1 for all cases; When D(q)=1, the retrieval plan generation unit generates an initial retrieval plan based on fact slots, priority modalities, time ranges, versions, permissions, and source requirements; numerical values, statistical definitions, and comparative data are preferentially allocated to the table index, institutional clauses and definitions are preferentially allocated to the text index, structures and trends are preferentially allocated to the image index, and meeting decisions, speech content, and operation processes are preferentially allocated to the audio and video index.

5. The system according to claim 1, characterized in that: In the iterative retrieval module M4: The retrieval status maintenance unit maintains the retrieval status object R for the t-th round. t ;R t At least the following should be recorded: current round t, structured query object Q, fact slot set, and currently selected evidence set E. t The current set of candidate unified knowledge units C t Uncovered slots, conflict information, used search criteria, maximum search rounds T max Remaining executable budget b t and the current state; where t is a non-negative integer, T max The maximum number of search rounds is preset and T max Greater than zero, b t It can be represented by remaining search attempts, remaining computation time, remaining tokens, or a combination thereof. Search status transitions between PLAN, SEARCH, RERANK, CHECK, REFINE, DONE, and STOP. The subquery planning and hybrid recall unit generates one or more subqueries for each uncovered fact slot in the PLAN state. Each subquery includes at least the target entity, time condition, keywords, priority modality, and metadata filtering conditions. In the SEARCH state, knowledge units are first filtered according to permissions, version, time, department, and document type. Then, keyword retrieval and vector retrieval are performed in parallel. Each subquery obtains the top K0 candidate unified knowledge units, where K0 is a positive integer. The candidate unified knowledge unit merging and filtering unit merges and deduplicates multiple candidate unified knowledge units according to unified knowledge unit identifier, source location, and content similarity, and filters candidate unified knowledge units with no access permissions, invalid versions, or resolution confidence below the minimum threshold. The evidence gap and conflict diagnosis unit, in the CHECK state, determines one by one whether each fact slot is supported by evidence. For the j-th fact slot in the t-th round, when evidence that can directly and completely support that slot is obtained, let h be the evidence. j,t =1, when only partial information is obtained, let h j,t =0.5, when no evidence is obtained or there are unresolved key conflicts, let h = 0.

5. j,t =0; Fact slot coverage Cov in round t t Calculate according to the following formula: In the above formula, t is the current retrieval round; j is the fact slot index; J is the total number of fact slots, where J is a positive integer; w j w represents the importance weight of the j-th fact slot. j A value greater than zero indicates that key conclusions, key values, statistical definitions, and sources can be set with a larger w. j h j,t The degree of support for the j-th fact slot by the evidence in round t, taking a value of 0, 0.5, or 1; Cov t For the fact slot coverage in round t, Cov t The value is between 0 and 1, with a larger value indicating a more complete coverage of the facts to be answered in the current evidence set; The evidence gap and conflict diagnosis unit also generates a key conflict marker X. t When two or more pieces of evidence provide different values, different valid versions, or contradictory conclusions regarding the same entity, the same indicator, and the same time period, and these conclusions cannot be resolved based on source level, approval status, or version validity period, let X... t =1; otherwise let X =1; t =0. This unit also records the maximum prediction generation gain G from the newly added evidence in this round. t And the number of newly covered fact slots in this round; when G t Below the preset gain threshold θ g In such cases, the same search criteria must not be reused in the next round; instead, the search terms, modality, or search scope should be changed. The state transition and supplementary retrieval unit determines the next state based on coverage, conflict flag, current round, and remaining budget, according to the following formula: In the above formula, State t+1 For the state in round t+1, choose one of DONE, REFINE, or STOP; Cov t Let θ be the actual slot coverage obtained in round t; c θ is the evidence coverage threshold. c Located between 0 and 1; X t The key conflict flag for round t is 0 or 1; t is the current search round; T max b is the maximum number of search rounds; t This represents the remaining executable budget after round t; the symbol "∧" represents logical AND, and the symbol "∨" represents logical OR; when Cov t Not less than θ c And X t When =0, enter the DONE state; when Cov... t Below θ c or X t =1, and t is less than T max And b t When the value is greater than 0, the system enters the REFINE state; otherwise, it enters the STOP state. When entering the REFINE state, the state transition and supplementary retrieval unit generates new subqueries only around uncovered fact slots or conflict reasons; when the indicator value has been obtained but the reason for the decline is missing, only the business analysis report and meeting minutes are retrieved; when there is a conflict between system versions, filter conditions such as "latest effective version", "approved", and explicit effective date are added; when the table has values ​​but lacks units, the table header, footnotes, or adjacent table blocks are retrieved; when entering the DONE state, the final evidence set and covered fact slots are output; when entering the STOP state, confirmed facts, uncovered facts, and unresolved conflicts are output.

6. The system according to claim 1, characterized in that: In the gain reordering module M5: The training samples and gain label construction units are used to construct the generation gain training labels for candidate unified knowledge units. Under the same generation model, the same prompt template, and the same decoding parameters, the system first generates a baseline answer A0 based on the query q and the current existing evidence set, and then generates an enhanced answer A after adding the i-th candidate unified knowledge unit to the existing evidence set. i The system calculates the answer quality Q(A) from four dimensions: factual accuracy, fact slot completeness, evidence consistency, and uncertainty. In the above formula, A represents the answer to be evaluated; Q(A) is the overall quality score of answer A, which is between 0 and 1; Acc(A) is the factual correctness score of answer A, Comp(A) is the completeness score of answer A to the fact slot, Cons(A) is the consistency score of answer A with the given evidence, and Unc(A) is the uncertainty score of answer A, all four indicators are between 0 and 1; 1-Unc(A) represents the certainty contribution; ω1, ω2, ω3 and ω4 are the weights of factual correctness, slot completeness, evidence consistency and certainty contribution, respectively, all four weights are between 0 and 1 and satisfy ω1+ω2+ω3+ω4=1; The generation gain label ΔG of the i-th candidate unified knowledge unit i Calculate according to the following formula: In the above formula, i is the index of the candidate unified knowledge unit; A i A0 is the enhanced answer generated after adding the i-th candidate unified knowledge unit; A0 is the baseline answer generated without adding the candidate unified knowledge unit; Q(A i Q(A0) and Q(A0) are the enhanced answer quality and baseline answer quality obtained according to equation (7), respectively; ΔG i Let ΔG be the true generation gain label for the i-th candidate unified knowledge unit. i Located between -1 and 1; ΔG i A value greater than 0 indicates that the candidate unified knowledge unit can improve the quality of the answer, ΔG i A value close to 0 indicates that the candidate unified knowledge unit overlaps with or is only minimally helpful to existing evidence; ΔG i A value less than 0 indicates that the candidate unified knowledge unit may be irrelevant, outdated, contain conflicting information, or lead to a decrease in the quality of the answer; The generation gain prediction unit employs a cross-coding reordering model, combining the standard query, query intent, fact slot, modality of the candidate unified knowledge unit, semantic proxy, structural information, version information, and source information of the candidate unified knowledge unit as joint input, and outputting the predicted generation gain of the i-th candidate unified knowledge unit. The model is trained using mean squared error loss. In the above formula, L gain To generate the training loss for the gain prediction model, L gain Greater than or equal to 0; M is the number of training samples, where M is a positive integer; i is the index of the training sample; #imgpt9# is the generation gain predicted by the model for the i-th training sample, preferably limited to between -1 and 1; ΔG i The calculated actual generated gain label; The goal of model training is to minimize L gain This makes the predicted generation gain close to the actual generation gain; During the online phase, the modality adaptation and comprehensive scoring unit calculates the modality adaptation degree Mod of the i-th candidate unified knowledge unit based on the query intent. i Furthermore, semantic relevance, prediction generation gain, modality fit, source and version validity, and degree of duplication with existing evidence are integrated into a comprehensive ranking score S. i : In the above formula, S i S is the comprehensive ranking score of the i-th candidate unified knowledge unit. i is a real number used for the relative ranking among candidate unified knowledge units; Rel i To determine the semantic relevance between the i-th candidate unified knowledge unit and the structured query, Mod i Val represents the modal fit between the i-th candidate unified knowledge unit and the query intent. i For the source reliability and version validity of the i-th candidate unified knowledge unit, Dup i Rel represents the degree of overlap between the i-th candidate unified knowledge unit and the currently selected evidence set. i Mod i Val i and Dup i All values ​​are normalized to 0 to 1; #imgpt11# represents the prediction generation gain of the i-th candidate unified knowledge unit, preferably between -1 and 1; α, β, γ, κ, and δ are the non-negative weights of semantic relevance, prediction generation gain, modality fit, source version validity, and duplication penalty, respectively; since S i For relative sorting only, the sum of the five weights is not required to be 1; a negative sign is set before the repetition level item so that the higher the repetition level, the lower the overall sorting score. The evidence selection unit is ranked according to a comprehensive score S. i The top N candidate unified knowledge units are selected from high to low, where N is an integer greater than zero. When the comprehensive scores are close, the candidate unified knowledge units that can add key fact slots, have valid sources, reliable parsing, and can accurately locate the original position are selected first. The ranking results, predicted generation gains, and suggested evidence sets are returned to the iterative retrieval module M4. The iterative retrieval module M4 migrates between the states of PLAN, SEARCH, RERANK, CHECK, REFINE, DONE, and STOP, and calls the gain reordering module M5 in the RERANK state; the CHECK state determines the subsequent state based on the fact slot coverage, key conflict flags, retrieval rounds, and remaining budget.

7. The system according to claim 1, characterized in that: In the evidence organization and answer generation module M6: The evidence object construction unit is used to assign an evidence number to each piece of evidence in the final evidence set and record the fact slot, modality type, source name, version, page number or timestamp, semantic content, and original location supported by the evidence; for tabular evidence, the evidence object retains the table header, primary key, unit, and cell coordinates; for image evidence, the evidence object retains the object relationship, key area, and OCR result; for audio and video evidence, the evidence object retains the speaker, transcription segment, start time, and end time. The structured context assembly unit organizes and generates context in the order of "fact slot - evidence number - modality - source location - evidence content", rather than directly splicing candidate unified knowledge units according to the recall order. For multiple pieces of evidence for the same fact slot, the main evidence that directly supports the conclusion is arranged first, followed by the auxiliary evidence used for cross-validation; when the context length exceeds the input budget of the generative model, fact slots with higher importance and evidence with higher prediction generation gain are retained first. The fact anchoring generation unit generates data based on the answers in the structured query object Q. schema A prompt template should be created, requiring the generated model to answer only based on the given evidence, and to mark the corresponding evidence number after each key value, clause, reason, and conclusion; for the direct answer path where D(q)=0, only questions that do not involve corporate private facts, time-sensitive facts, designated sources, and high-risk business are allowed to be answered; when evidence is insufficient or conflicting, the generated model must clearly state the missing or conflicting items, and may not fill in corporate facts that are not supported by sources.

8. The system according to claim 1, characterized in that: In the answer verification and feedback control module M7: The source permission version verification unit is used to check whether the evidence number cited in the initial answer exists, whether the current user has access rights, whether the source anchor is valid, and whether the corresponding document version is valid. The numerical semantic conflict verification unit is used to trace the numerical values, units, times, and calculation relationships in the answer back to table cells, OCR areas, or original text fragments to determine whether the evidence directly supports the corresponding conclusion and to check whether the same conclusion cites contradictory sources. When the answer correction and review control unit discovers that a reference is missing, the values ​​are inconsistent, the conclusion lacks direct support, or the key evidence is conflicting, it first deletes, downgrades, or rewrites the relevant conclusions based on existing evidence. If the missing conclusion is a key fact and the remaining budget b is retrieved from the state object. t If the remaining budget is greater than 0, the missing fact slot, conflict cause, and existing evidence status will be returned to the iterative retrieval module M4 for supplementary retrieval; if the remaining budget is 0, the confirmed facts will be retained and the insufficient evidence status will be output. After verification, the final output unit outputs the user-facing answer, the evidence number corresponding to each key conclusion, the document name, page number, table coordinates, image area or audio / video timestamp, covered facts, uncovered facts, and explanations of any remaining conflicts. Evidence organization, fact anchoring generation, and answer verification form a closed loop; when verification finds insufficient key evidence or unresolved conflicts and there is still budget, the answer verification and feedback control module M7 returns the missing fact slot and conflict cause to the iterative retrieval module M4; when there is no remaining budget, the confirmed facts and insufficient evidence status are output.

9. A multimodal retrieval enhancement generation method for enterprise-level heterogeneous knowledge based on the system described in claim 1, characterized in that: The method includes the following steps: Step S100: Through the knowledge resource construction module M1, access the heterogeneous knowledge sources of the enterprise, register the document primary key, version and access permissions of the knowledge sources, perform structure-preserving parsing on knowledge sources of different modalities, and build a unified knowledge unit as well as keyword index, vector index and metadata index; Step S200: Through the generation gain re-ranking module M5, training samples are constructed based on historical question answers, manual test questions and business cases. The changes in answer quality before and after adding candidate unified knowledge units are compared to generate generation gain labels for candidate unified knowledge units. A generation gain prediction model for online ranking of candidate unified knowledge units is trained. Step S300: By querying the structured module M2, perform referential resolution, terminology normalization, spatiotemporal constraint extraction, and fact slot decomposition on the current user's question, historical messages, historical evidence, and the chart or media object selected by the user, and output the structured query object. In step S400, the retrieval gate module M3 determines whether to initiate an enterprise knowledge base retrieval based on the mandatory retrieval conditions and the model knowledge sufficiency calculation results. If the retrieval decision is not to retrieve, proceed to step S600. If the retrieval decision is to retrieve, generate a retrieval plan and proceed to step S500. Step S500: Through the iterative retrieval module M4 and the generation gain reordering module M5, perform hybrid recall, candidate unified knowledge unit merging and filtering, generation gain prediction, modality-aware reordering, evidence coverage check, evidence conflict diagnosis and gap filling retrieval around the fact slot to be covered, until the completion condition is met or the stopping condition is reached. In step S600, the final evidence is organized into a structured context that retains the source anchor point through the evidence organization and answer generation module M6 and the answer verification and feedback control module M7, and the fact-anchored answer is generated. The source, permissions, version, value, unit, time, semantic support relationship and evidence conflict in the answer are verified one by one. If the key evidence is insufficient and there is still a search budget, the process returns to step S500 to perform a supplementary search. Otherwise, the final answer and the corresponding evidence source are output.

10. The method according to claim 9, characterized in that: Step S100 specifically includes the following steps: Step S110: Receive enterprise knowledge sources such as PDFs, Word documents, web pages, Excel files, images, audio, and video through the resource access and version permission registration unit; obtain file identifiers, content summaries, version numbers, modification times, source paths, and access control information; generate document primary keys based on file identifiers, content summaries, version numbers, and modification times; if the document primary key and content summary are identical, it is determined to be a duplicate resource and duplicate entries are skipped; when a new version is detected, an inheritance or replacement relationship between the new and old versions is established, and expired versions are marked as non-default recalls; convert user, department, role, document, and field-level permissions into permission metadata. Step S120: The structure-preserving parsing unit performs corresponding parsing based on the knowledge source modality: For PDF, Word, web pages, and scanned documents, it identifies page areas such as titles, body text, lists, headers and footers, footnotes, images, and tables, restores the reading order, and divides the document into blocks according to chapters, paragraphs, list items, and semantic transitions; For complex tables, it sequentially performs merged cell expansion, record direction recognition, row and column transposition when necessary, multi-level header compression, unit and primary key binding, multi-granular segmentation, and cell coordinate recording. For images, OCR, region detection, and visual semantic description are performed; for audio and video, time-stamped speech recognition, speaker segmentation, scene transition detection, keyframe extraction, and audio-visual alignment are performed. When the resolution confidence is lower than a preset threshold, simplified semantic proxy and the original image, original file reference, or original media time interval are preserved. Step S130: By encapsulating text blocks, table blocks, image semantic proxies, audio segments, and video spatiotemporal slices into unified knowledge units through a unified knowledge unit encapsulation unit, and writing a unique identifier, modality type, original content or original file reference, semantic proxy, structural information, business time or media time, source anchor, version validity period, access permissions, and vector representation for each unified knowledge unit. Step S140: By constructing a unit through a hybrid index, a keyword inverted index is built for the semantic agent, table header, unit, entity and enterprise terminology of the unified knowledge unit. Vectors are generated by embedding model and vector indexes are built. Metadata indexes are built for permissions, version, validity period, department, document type, time range, page number and media timestamp. The three types of indexes all point to the same unified knowledge unit identifier. During online recall, the initial recall score is calculated and multiple results are merged and deduplicated.

11. The method according to claim 9, characterized in that: Step S200 specifically includes the following operation steps: Step S210: Obtain training questions from enterprise historical Q&A, manual test questions, and business cases; form structured queries and fact slots through the query structure module M2; and recall candidate unified knowledge units in text, table, image, audio, and video modalities through the iterative retrieval module M4; each training sample includes at least a structured query, fact slot, current existing evidence, semantic proxy of the candidate unified knowledge unit, modality, structural information, source location, version information, and time information of the candidate unified knowledge unit; retain candidate unified knowledge units with high similarity but no answer, duplicate existing evidence, outdated version, or factual conflict as difficult negative samples; Step S220: Under the same generative model, the same prompt template, and the same decoding parameters, generate a baseline answer A0 based on the training question and the existing evidence. Then, add the i-th candidate unified knowledge unit to the existing evidence to generate an enhanced answer A. i Calculate the baseline answer quality Q(A0) and the enhanced answer quality Q(A0) respectively. i Then calculate the generation gain label ΔG for the i-th candidate unified knowledge unit. i The correctness of facts is determined by the standard answer, knowledge base rules, or manual verification results; the completeness of fact slots is determined by the fact slots in the structured query; the consistency of evidence is determined by whether the candidate unified knowledge unit directly supports the answer conclusion; and the uncertainty is determined by fuzzy expressions, internal contradictions in the answer, and source conflicts. Step S230: Combine the standard query, query intent, fact slot, modality of candidate unified knowledge unit, semantic proxy of candidate unified knowledge unit, structural information, source information and version information into the joint input of the generation gain prediction model, so that the model outputs the predicted generation gain of candidate unified knowledge unit; calculate the training loss and update the model parameters until the preset convergence condition is reached; select the model parameters with smaller generation gain prediction error and better ranking effect of candidate unified knowledge unit on the validation set, and write the selected model parameters into the model parameter storage area; In step S240, the generated gain prediction model trained in step S230 is deployed to the generated gain reordering module M5 for online use in step S530. The manual review results, answer verification results, and newly added business data are used as new training samples, and incremental training or retraining is performed on the same generated gain prediction model according to a preset cycle. The updated model parameters replace the original model parameters in the model parameter storage area and are then used in subsequent steps S530.

12. The method according to claim 9, characterized in that: Step S300 specifically includes the following operation steps: Step S310: Using the context object positioning unit, determine the corresponding object identifier and source anchor point for indicators such as "this chart", "previous table", "this product", and "yesterday's meeting" from historical messages, historical evidence objects, and the chart or media object currently selected by the user; When there are multiple candidate objects and the highest matching confidence is lower than a preset threshold, multiple candidate objects are retained for separate processing or a clarification request is output. Step S320: Through the terminology and spatiotemporal constraint standardization unit, the abbreviations of departments, product codes, indicator aliases and document abbreviations are mapped to standard names according to the enterprise terminology table. Relative times such as "latest", "current", "this quarter" and "yesterday" are converted into specific time points or time ranges, and constraints such as department, region, version, permissions, required modality and answer format are extracted. Step S330: Through the fact slot decomposition unit, the complex problem is broken down into fact slots that can be independently retrieved and verified according to the query intent; each fact slot includes at least the slot name, target entity, time condition, expected data type, priority modality and slot importance. In step S340, the structured query generation unit encapsulates the standard query, query intent, standardized entity, constraint condition, priority mode, timeliness requirement, fact slot set and answer structure into a structured query object Q, and sends the structured query object to the retrieval gating module M3.

13. The method according to claim 9, characterized in that: Step S400 specifically includes the following operation steps: Step S410: The forced retrieval judgment unit determines whether the structured query involves internal enterprise policies, business data, unpublished information, the latest or specified version, a specified source, page number, table coordinates, media timestamps, facts that must be answered based on images or audio / video, or preset high-risk business such as finance, compliance, and security; if any condition is met, the forced retrieval flag F is activated. q =1, otherwise let F q =0; Step S420, when F q When =0, candidate answers are generated for internal evaluation only, and the confidence C of the answers is calculated respectively. q Fact slot completeness P q and uncertainty risk U q And calculate the knowledge sufficiency K. q When F q When =1, regardless of whether internal candidate answers can be generated, the forced retrieval state is directly retained; Step S430, calculate the retrieval decision D(q); when F q =0 and K q Not lower than the knowledge sufficiency threshold θ k When D(q) is set to 0, proceed to step S600; otherwise, set D(q) to 1, generate the retrieval triggering reason, the fact slot to be covered, the priority mode, the retrieval range and the filtering conditions through the retrieval plan generation unit, initialize the retrieval state object R0 and proceed to step S500.

14. The method according to claim 9, characterized in that: Step S500 specifically includes the following operation steps: Step S510: Establish the retrieval state object R for the t-th round through the retrieval state maintenance unit. t Record the current round t, the structured query object Q, the set of fact slots, and the set of selected evidence E. t Candidate unified knowledge unit set C t Uncovered slots, conflict information, used search criteria, maximum search rounds T max Remaining executable budget b t And the current state; the initial state is set to PLAN; Step S520: In the PLAN state, generate one or more subqueries for each uncovered fact slot. Each subquery includes at least the target entity, time condition, keywords, priority modality, and metadata filtering conditions. Then, enter the SEARCH state. First, filter unified knowledge units according to permissions, version, time, department, and document type. Then, perform keyword retrieval and vector retrieval in parallel and calculate the initial recall score. For each subquery, obtain the top K0 candidate unified knowledge units and merge and deduplicate them according to unified knowledge unit identifier, source location, and content similarity. Filter out candidate unified knowledge units with no access permissions, expired versions, or parsing confidence below the minimum threshold to form the t-th round candidate unified knowledge unit set C. t ; Step S530: Combine the structured query object Q and the previously selected evidence set E. t-1 and the set of unified knowledge units C for candidates in round t t Input the generative gain re-ranking module M5, and output the predicted generative gain for each candidate unified knowledge unit from the generative gain prediction model trained in step S200; calculate the semantic relevance, modality fit, source and version validity, and degree of duplication with existing evidence for the candidate unified knowledge unit, and calculate the comprehensive ranking score S. i The top N candidate unified knowledge units are selected from high to low based on their comprehensive ranking scores, and priority is given to retaining candidate unified knowledge units that can add key fact slots, have valid sources, reliable parsing, and can accurately locate the original position. Step S540: Enter the CHECK state and determine the support level h for the j-th fact slot in round t. j,t The complete evidence corresponds to h. j,t =1, partial evidence corresponds to h j,t =0.5, no evidence obtained or unresolved key conflicts corresponding to h j,t =0; Calculate the fact slot coverage Cov in round t. t When two or more pieces of evidence provide different values, valid versions, or contradictory conclusions regarding the same entity, the same indicator, and the same time period, and these cannot be resolved based on source level, approval status, or version validity period, then the key conflict indicator X is set. t =1, otherwise let X =1. t =0; Step S550: Record the maximum prediction generation gain and the number of newly added covered fact slots in the newly added evidence in this round; when the maximum prediction generation gain is lower than the preset gain threshold θ g In such cases, the same search criteria must not be reused in the next round; instead, the search keywords, search modality, or search scope should be changed. When low gain occurs consecutively and no new fact slots are added, the remaining executable budget b will be used. t Set to 0; Then determine the next state; In step S560, when the next state is REFINE, generate a new subquery only around the uncovered fact slots or the reasons for evidence conflicts and return to step S520; when the next state is DONE, output the final evidence set, the covered fact slots and the source anchor and proceed to step S600; when the next state is STOP, output the confirmed facts, the uncovered facts and the unresolved conflicts, and proceed to step S600 in a conservative manner.

15. The method according to claim 9, characterized in that: Step S600 specifically includes the following operation steps: Step S610: Assign an evidence number to each piece of evidence in the final evidence set through the evidence object construction unit, and record the fact slot, modality type, source name, version, page number or timestamp, semantic content and original location supported by the evidence; table evidence retains the table header, primary key, unit and cell coordinates; image evidence retains the object relationship, key area and OCR result; audio and video evidence retains the speaker, transcription segment and start time and end time. Step S620: The context is organized in the order of "fact slot - evidence number - modality - source location - evidence content" through the structured context assembly unit; under the same fact slot, the main evidence that directly supports the conclusion is arranged first, and then the auxiliary evidence used for cross-validation is arranged. Generate a prompt template based on the answer structure requirements in the structured query object Q. The generated model is required to answer only based on the given evidence and to mark the corresponding evidence number after each key value, clause, reason and conclusion, thereby generating the initial answer. When D(q)=0, direct answers are only performed for questions that do not involve private facts of the enterprise, time-sensitive facts, designated sources, or high-risk business. Step S630: The initial answer is broken down into several key conclusions and a conclusion-evidence number mapping is established. The source existence verification, access permission verification, version validity verification, numerical value and unit verification, time verification, semantic support verification, and evidence conflict verification are performed in sequence. For non-key conclusions that do not exist, have inconsistent numerical values, lack direct support, or have evidence conflicts, they are deleted, downgraded, or rewritten based on existing evidence. Step S640, when the missing or conflicting conclusion pertains to key facts and the remaining executable budget b t When the value is greater than 0, the missing fact slot, conflict cause, and existing evidence status will be returned to step S500 to perform supplementary retrieval; when the remaining budget is 0 or there are no key facts that need to be supplemented, the retrieval feedback will be stopped. Step S650 outputs the final answer for the user, the evidence number corresponding to each key conclusion, document name, page number, table coordinates, image area or audio / video timestamp, covered facts, uncovered facts, and explanations of any remaining conflicts; the manual review results and the verification status generated in this step are recorded as updated samples that can be used when executing step S240 in the future.

16. The method according to claim 9, characterized in that: Parsing failure handling: When the confidence level of table structure recognition, image OCR or speech recognition is lower than the preset threshold, the knowledge resource construction module M1 simultaneously retains the original file, original image or original audio and video range and simplifies semantic proxy, and reduces the parsing reliability, without directly deleting the evidence; Modality missing handling: When a query explicitly requires a certain modality but the shared knowledge storage does not contain that modality, the iterative retrieval module M4 outputs a modality missing value; other modalities can be retrieved as supplementary evidence, but textual speculations must not be presented as non-existent visual or audio-visual facts; Empty Recall Handling: When the initial recall is empty, the iterative retrieval module M4 sequentially expands the entity aliases, synonyms, and allowed time range, and changes the retrieval modality or index combination; if no candidate unified knowledge unit is still obtained, it enters the STOP state and outputs that insufficient evidence was not retrieved. Evidence conflict handling: Prioritize evidence with updated validity, clear approval status, higher source level, and verifiability by other sources; if conflict still cannot be resolved, present the conflicting content and corresponding sources side by side, and reduce the certainty of the conclusion; Permissions and budget handling: Access permission filtering is performed before recall, and the final answer only shows the content and source that the current user is authorized to access; when the maximum retrieval rounds or computation budget are exhausted, confirmed conclusions, uncovered facts and unresolved conflicts are output based on the current fact slot coverage, and further calls are stopped; Model or tool anomaly handling: When the embedded model, OCR tool, speech recognition tool, reordering model, or generation model fails to be invoked, a backup model of the same type is invoked first or the corresponding degradation processing is adopted; if the backup model is still unavailable, the current retrieval status and confirmed facts are retained, the reason for the anomaly, the incomplete steps, and the retry flag are output, and no conclusions that are not supported by evidence are generated.

Images