Intelligent question answering and knowledge retrieval method and system based on multi-modal fusion

Abstract

The application discloses a multi-modal fusion intelligent question answering and knowledge retrieval method and system, and the method comprises the following steps: a multi-modal data index model facing heterogeneous knowledge sources is established, text, image, table, chart, audio and video content are mapped through a unified semantic embedding space to generate a cross-modal index set; after receiving a query request, a multi-channel retriever is used to perform semantic matching and structural matching on the cross-modal index set to obtain candidate evidence segments; based on an evidence granularity decomposition strategy, the text sentences, table units, chart data points and multimedia frame contents in the candidate evidence segments are divided into minimum evidence units, and a semantic consistency graph is established between the units. Through multi-modal fusion and spatiotemporal consistency constraints, the application realizes high-trust and traceable generation of question answering results, and significantly improves the retrieval accuracy and explanation transparency in complex knowledge scenarios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimodal fusion intelligent question answering and knowledge retrieval method and system. Background Technology

[0002] With the development of pre-trained large models and retrieval-enhanced generation, multimodal question answering and knowledge retrieval systems targeting heterogeneous knowledge sources both inside and outside enterprises are rapidly emerging. Existing technologies mostly employ multimodal embedding and cross-modal alignment, combined with vector databases to achieve unified retrieval and generation of content such as images, tables, charts, and audio / video.

[0003] However, while existing technologies are relatively mature in semantic relevance modeling, vector retrieval, and multi-retrieval fusion, they still expose two hidden defects in high-risk scenarios, directly affecting credibility and reproducibility. First, insufficient granularity and traceability of evidence: Evidence is generally presented in paragraphs, pages, or entire images as units, making it difficult to accurately map each fact, value, and entity in the response to table cells, chart data points, their units, and footnotes. There is also a lack of automatic detection and adjudication of cross-modal contradictions, making it difficult for users to obtain verifiable evidence chains and coverage indicators. Second, weak spatiotemporal and version consistency: Insufficient attention is paid to timestamps and version identifiers during indexing and reordering. The reasoning stage lacks constraints regarding the same time point and version, easily leading to the mixing of text and charts from different versions of reports, video footage and subtitles, and webpage snapshots and offline images, forming seemingly reasonable but factually invalid conclusions. Based on these defects, practice urgently needs two capabilities: establishing a one-to-one correspondence and visual backtracking of the minimum sufficient evidence set across modalities, and introducing time and version consistency constraints and proof mechanisms at each stage of retrieval and generation. Summary of the Invention

[0004] To address the aforementioned problems, embodiments of the present invention provide a multimodal fusion-based intelligent question answering and knowledge retrieval method, the method comprising:

[0005] Establish a multimodal data indexing model for heterogeneous knowledge sources, and generate a cross-modal index set by feature mapping of text, images, tables, charts, audio and video content through a unified semantic embedding space;

[0006] After receiving a query request, a multi-channel retrieval tool is used to perform semantic and structural matching on the cross-modal index set to obtain candidate evidence fragments.

[0007] Based on the evidence granularity decomposition strategy, the text statements, table units, chart data points and multimedia frame content in the candidate evidence fragments are divided into the smallest evidence units, and a semantic consistency graph is established between each unit.

[0008] The cross-modal association determination module performs evidence aggregation and contradiction adjudication on the semantic consistency graph to generate a minimum sufficient set of evidence that satisfies logical constraints.

[0009] During the generation phase, the inference engine performs traceable question-and-answer generation based on the minimum sufficient evidence set, and generates a fact-evidence mapping table and a cross-modal visual backtracking path.

[0010] Throughout the entire retrieval and generation process, a timestamp and version identification constraint mechanism is introduced to perform spatiotemporal consistency verification and version locking on the evidence source, ensuring that the output content is consistent with the original data source in terms of time and version dimensions.

[0011] Furthermore, the multimodal data indexing model includes a semantic embedding sub-model and a structural embedding sub-model. The semantic embedding sub-model uses a cross-modal alignment network to achieve a unified semantic representation of text, images, and table titles. The structural embedding sub-model describes the relationships between table rows and columns, chart coordinates, and time axis based on a hierarchical graph structure encoding method. The two are concatenated by features to form a trainable cross-modal embedding matrix, which is used to construct the cross-modal index set.

[0012] Furthermore, the multi-channel retrieval device includes a semantic retrieval channel, a structural retrieval channel, and a contextual rearrangement channel. The semantic retrieval channel uses a cross-modal similarity metric function to calculate the embedding distance between text and visual features. The structural retrieval channel performs relationship matching based on table structure and chart coordinate constraints. The contextual rearrangement channel uses the query intent vector to perform relevance rearrangement and confidence weighting on the outputs of the first two channels, thereby generating the final candidate evidence fragment set.

[0013] Furthermore, the evidence granularity decomposition strategy includes a hierarchical segmentation unit and a semantic anchor detection unit; the hierarchical segmentation unit, based on document structure identification, visual segmentation, and audio / video frame sequence feature extraction algorithms, subdivides documents and multimedia content into sentence-level, element-level, and frame-level evidence units; the semantic anchor detection unit labels key concepts, values, and reference nodes in each unit according to entity recognition models and relation extraction models, thereby achieving a one-to-one correspondence between cross-modal units.

[0014] Furthermore, the cross-modal association determination module includes a semantic consistency detection unit, a numerical deviation analysis unit, and a contradiction adjudication unit; the semantic consistency detection unit calculates the semantic similarity between different modal evidences through a semantic matching network; the numerical deviation analysis unit establishes a numerical comparison relationship between table units and chart data points; the contradiction adjudication unit performs a weighted adjudication based on the evidence source weight, time recentity, and modal confidence, and outputs the minimum sufficient evidence set with the highest consistency.

[0015] Furthermore, the timestamp and version identifier constraint mechanism includes consistency constraints in the indexing phase and consistency constraints in the generation phase. The consistency constraints in the indexing phase record the collection time and version source of each data item through a time hash identifier. The consistency constraints in the generation phase perform a secondary verification of the time hash and version identifier of the evidence when the inference engine performs question and answer generation, ensuring that the source of the cited evidence is completely consistent in terms of time and version.

[0016] Furthermore, when generating traceable question-and-answer results, the inference engine establishes a fact-evidence mapping table, which includes fact descriptions, evidence unit identifiers, modality types, and corresponding semantic anchor point positions. At the same time, it generates a cross-modal visual backtracking path, displaying the position of the evidence corresponding to the fact in the original data source in a graphic and textual format, thereby enabling verifiable display on the user side.

[0017] Multimodal fusion-based intelligent question answering and knowledge retrieval methods, including:

[0018] An evidence completion mechanism based on reasoning feedback is introduced, which includes:

[0019] When constructing the fact-evidence mapping table, the inference engine performs anchor point completeness verification on the fact descriptions; when it detects that entity anchor points, time anchor points, or numerical anchor points are not covered in the existing evidence set, it generates corresponding supplementary query instructions.

[0020] The supplementary query instruction is submitted to the multichannel search engine, which performs a supplementary search in the cross-modal index set according to the instruction to obtain supplementary evidence units;

[0021] The supplementary evidence unit undergoes semantic consistency verification and time version verification via the consistency determination module. Only when the verification passes will the supplementary evidence be added to the current evidence set.

[0022] The inference engine reconstructs the inference chain based on the updated evidence set. Once all anchor points are covered and a logical closure is formed, a traceable question-and-answer result is output.

[0023] A multimodal fusion intelligent question answering and knowledge retrieval system, comprising:

[0024] The heterogeneous index module establishes a multimodal data index model for heterogeneous knowledge sources, and performs feature mapping on text, images, tables, charts, audio and video content through a unified semantic embedding space to generate a cross-modal index set.

[0025] The semantic retrieval module, after receiving a query request, uses a multi-channel retrieval device to perform semantic matching and structural matching on the cross-modal index set to obtain candidate evidence fragments;

[0026] The granularity decomposition module, based on the evidence granularity decomposition strategy, divides the text statements, table units, chart data points and multimedia frame content in the candidate evidence fragments into the smallest evidence units, and establishes a semantic consistency graph between each unit.

[0027] The association adjudication module performs evidence aggregation and contradiction adjudication on the semantic consistency graph through the cross-modal association judgment module, generating a minimum sufficient evidence set that satisfies logical constraints;

[0028] The traceable generation module, in the generation phase, the inference engine performs traceable question-and-answer generation based on the minimum sufficient evidence set, and at the same time generates a fact-evidence mapping table and a cross-modal visual backtracking path;

[0029] The spatiotemporal consistency module introduces a timestamp and version identifier constraint mechanism throughout the entire retrieval and generation process. It performs spatiotemporal consistency verification and version locking on the evidence source to ensure that the output content is consistent with the original data source in terms of time and version dimensions.

[0030] The technical effects and advantages of the multimodal fusion intelligent question answering and knowledge retrieval method provided by this invention are as follows:

[0031] This invention achieves highly reliable and traceable generation of question-and-answer results through multimodal fusion and spatiotemporal consistency constraints, significantly improving retrieval accuracy and interpretive transparency in complex knowledge scenarios. By constructing a unified semantic embedding and structural embedding model, this invention achieves highly consistent indexing and retrieval of heterogeneous modal content, enhancing the effect of cross-modal information fusion; employing evidence granularity decomposition and semantic anchoring mechanisms, it ensures that each fact in the question-and-answer results corresponds to a specific evidence unit, enhancing verifiability; leveraging cross-modal association judgment and contradiction adjudication mechanisms, it effectively identifies and eliminates semantic deviations between evidence from different modalities, improving conclusion consistency; introducing timestamp and version identification constraints, it ensures the consistency of evidence in the time and version dimensions, preventing reasoning distortion caused by historical misuse; and by generating a fact-evidence mapping table and a visualized backtracking path, it provides users with verifiable and interpretable multimodal question-and-answer results, enhancing system transparency and credibility. Attached Figure Description

[0032] Figure 1 This is a flowchart of the multimodal fusion intelligent question answering and knowledge retrieval method in Example 1;

[0033] Figure 2 This is a flowchart of the evidence completion mechanism based on reasoning feedback in Example 2;

[0034] Figure 3 This is a schematic diagram of the connection of the multimodal fusion intelligent question answering and knowledge retrieval system in Example 2. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Example 1:

[0037] Please see Figure 1 As shown, embodiments of the present invention provide a multimodal fusion-based intelligent question answering and knowledge retrieval method, including the following steps:

[0038] Establish a multimodal data indexing model for heterogeneous knowledge sources, and generate a cross-modal index set by feature mapping of text, images, tables, charts, audio and video content through a unified semantic embedding space;

[0039] After receiving a query request, a multi-channel retrieval tool is used to perform semantic and structural matching on the cross-modal index set to obtain candidate evidence fragments.

[0040] Based on the evidence granularity decomposition strategy, the text statements, table units, chart data points and multimedia frame content in the candidate evidence fragments are divided into the smallest evidence units, and a semantic consistency graph is established between each unit.

[0041] The cross-modal association determination module performs evidence aggregation and contradiction adjudication on the semantic consistency graph to generate a minimum sufficient set of evidence that satisfies logical constraints.

[0042] During the generation phase, the inference engine performs traceable question-and-answer generation based on the minimum sufficient evidence set, and generates a fact-evidence mapping table and a cross-modal visual backtracking path.

[0043] Throughout the entire retrieval and generation process, a timestamp and version identification constraint mechanism is introduced to perform spatiotemporal consistency verification and version locking on the evidence source, ensuring that the output content is consistent with the original data source in terms of time and version dimensions.

[0044] The multimodal data indexing model includes a semantic embedding sub-model and a structural embedding sub-model. The semantic embedding sub-model uses a cross-modal alignment network to achieve a unified semantic representation of text, images, and table titles. The structural embedding sub-model describes the relationship between table rows and columns, chart coordinates, and time axis based on a hierarchical graph structure encoding method. The two are concatenated by features to form a trainable cross-modal embedding matrix, which is used to construct the cross-modal index set.

[0045] The cross-modal alignment network in the semantic embedding sub-model introduces a shared semantic encoding layer during the training phase. By aligning the contextual relationships of semantic fragments such as text descriptions, image captions, and table titles, it achieves a unified distribution of semantic vectors. For example, when the knowledge source contains a chart displaying "Annual Sales Growth," the corresponding title text "Sales Growth by Year," and the descriptive paragraph "the sales increased continuously in recent years," the model projects these three elements to similar positions in the semantic space through the cross-modal alignment network, thus ensuring semantic consistency across different modalities. In this process, the semantic embedding sub-model does not merely extract single-modal features but also uses semantic alignment loss constraints to align text vectors with the vectors of images or table titles within the same reference frame. This type of alignment network can employ a bidirectional semantic encoder structure, aligning text and images in the forward direction and text and table titles in the reverse direction, thereby enhancing the robustness of semantic associations. In this way, the model can handle complex scenarios such as mixed text and image layouts, table annotations, and multilingual titles, providing a unified semantic representation foundation for subsequent indexing.

[0046] The structure embedding sub-model employs a hierarchical graph structure encoding approach, explicitly modeling structural information as a combination of nodes and edges. Each node represents a data unit, such as a table cell, a chart coordinate point, or a time series node. Edges are used to characterize the semantic and structural dependencies between nodes. For example, in a table, the logical hierarchy between rows and the statistical relationships between columns are abstracted as edges between nodes. In a chart, the correspondence between the horizontal and vertical axes and data points is also incorporated into the structure graph model for encoding. During the encoding stage, the structure embedding sub-model maps node features and edge features into low-dimensional vector representations and forms a global structural representation through aggregation functions. This global representation not only preserves the hierarchical logic of the table but also retains the evolutionary relationships of the chart coordinates and time series, thus supporting semantic retrieval of complex structures during the indexing stage. For example, when retrieving "energy consumption trend in a certain year," the model can accurately locate the corresponding curve segment in the chart based on the relationship between the time axis and the numerical axis, rather than relying solely on text descriptions.

[0047] Each row of the cross-modal embedding matrix corresponds to a cross-modal data unit, and each column corresponds to a feature dimension, containing joint encoding of semantic and structural information. To avoid feature bias between different modalities, a unified normalization and attention reweighting mechanism is used after concatenation to ensure the spatial balance of features of each modality. The final cross-modal embedding matrix is ​​stored in the index engine as the basis for building a cross-modal index set to support subsequent semantic retrieval, evidence decomposition, and question answering tasks.

[0048] The multi-channel retrieval system includes a semantic retrieval channel, a structural retrieval channel, and a contextual reordering channel. The semantic retrieval channel uses a cross-modal similarity metric function to calculate the embedding distance between text and visual features. The structural retrieval channel performs relationship matching based on table structure and chart coordinate constraints. The contextual reordering channel uses the query intent vector to reorder the relevance and confidence weights of the outputs of the first two channels, thereby generating the final set of candidate evidence fragments.

[0049] The operation methods of the semantic retrieval channel include:

[0050] First, semantic vectors corresponding to text, image descriptions, and table titles are read from the cross-modal index set. The input query is then parsed to generate a query intent vector. Subsequently, a similarity metric function is used to calculate the distance or similarity score between the query intent vector and each semantic vector in the index. To ensure the comparability of semantic scores across different modalities, the semantic retrieval channel performs distribution calibration on the scores, such as normalization based on intra-channel statistics, and labels index items with higher scores as semantic candidates. The semantic retrieval channel emphasizes coverage of the semantic correspondence between natural language queries and text image titles, and is suitable for finding descriptive evidence and image-text annotation evidence.

[0051] The operation methods of the structure retrieval channel include:

[0052] First, the structure embedding vectors in the cross-modal index are loaded into the structure retrieval sub-index, and a structure index (including row and column positions, table header links, and cell type labels) is created for table cells; a coordinate index (including legend associations, axis labels, and data point labels) is created for charts; and a time index (including time point labels and series intervals) is created for time series. During the retrieval phase, a structured retrieval template is extracted based on the parsed query intent, such as "retrieve numerical curves by time range" or "retrieve table cells by row and column." Constraint matching is performed on the structure sub-index, and a structure score is generated. By comparing coordinate relationships, row and column dependencies, and series trends, the structure retrieval channel can accurately locate table cells and chart fragments, thereby compensating for the shortcomings of semantic retrieval in fine-grained numerical positioning.

[0053] The methods for executing context-reordered channels include:

[0054] The system collects the raw scores, modality types, and source contexts (such as document paragraph positions, chart titles, and table row numbers) for the same candidate option from both channels. Secondly, it calculates the contextual fit of the candidate option based on the query intent, for example, determining whether the paragraph containing the candidate option overlaps with the keywords of the query intent and whether the timestamps of the candidate data points fall within the time frame specified in the query. Thirdly, it applies a set of trainable rearrangement rules or a lightweight model to correct the raw scores according to the contextual fit and output the final confidence score. The context rearrangement channel also handles the first round of conflict detection: when the same query returns contradictory information in different modalities, the context rearrangement channel marks the conflict records based on the source context and confidence hints for further adjudication by the subsequent cross-modal decision module.

[0055] The synergistic integration of the three channels adopts the following mechanism:

[0056] The raw scores output from each channel are first standardized within the channel. Then, the confidence scores calculated by the context-rearranged channels are used as fusion weights and weighted together with the semantic and structural scores to form the comprehensive score of each candidate evidence unit. For candidates whose comprehensive scores reach the threshold, the retrieval engine performs deduplication and fragment merging: if two candidate fragments correspond to different expressions of the same position in the cross-modal index set (e.g., a table cell and a graph data point represent the same value), the retrieval engine merges them into a single candidate evidence unit and records the multimodal source information to maintain evidence coverage and traceability. Finally, the retrieval engine outputs a set of candidate evidence fragments arranged in descending order of comprehensive scores. Each fragment carries a source identifier, modality type, original channel score, and rearranged confidence score, which are used for subsequent evidence granularity decomposition and cross-modal contradiction adjudication.

[0057] For example:

[0058] If a user queries "the reasons for the decline in sales of product A in the past year", the semantic search channel may select paragraphs and chart titles containing the description of "decline in sales" as high semantic candidates; the structural search channel can locate table cells in reports and data curve segments in charts corresponding to the year; the contextual reordering channel will evaluate whether the chart timeline matches the table year and give priority to candidates that are consistent in time and have high contextual fit; if there is a numerical deviation between the table cell and the chart data, the deviation will be recorded in the contextual reordering stage and passed to the cross-modal decision module for further adjudication, ensuring that the search results are both semantically relevant and structurally correct.

[0059] The evidence granularity decomposition strategy includes hierarchical segmentation units and semantic anchor detection units. The hierarchical segmentation units are based on document structure identification, visual segmentation, and audio / video frame sequence feature extraction algorithms to subdivide document and multimedia content into sentence-level, element-level, and frame-level evidence units. The semantic anchor detection units label the key concepts, values, and reference nodes in each unit according to entity recognition models and relation extraction models, thereby achieving a one-to-one correspondence between cross-modal units.

[0060] The implementation of hierarchical segmentation units follows the principle of parallel parsing of document prior structure and multimedia content features. For text and semi-structured documents, structural parsing is performed first: a document layout recognition process is used to identify headers and footers, paragraph boundaries, table areas, and annotation areas. The identified table areas are further parsed using a table parsing process to extract table headers, row and column indices, and cell boundaries, thereby generating several table cells. For images and charts, a visual segmentation process is applied to segment the image into several candidate regions, and visual elements such as legends, coordinate axes, and data curves are identified. For images containing text, text recognition is performed first, and the recognition results are written back as text cells for subsequent semantic processing. For audio and video materials, keyframe extraction and speech-to-text processing are performed first, and then the video is segmented into frame-level units according to the time sequence. Inter-frame change detection is applied to the frame sequence to identify and query relevant time intervals.

[0061] The semantic anchor detection unit's task is to label each segmentation unit with semantic tags and numerical anchors that can be used for cross-modal correspondence. Its implementation includes four sub-processes: entity recognition, relation extraction, numerical standardization, and reference node recognition. Entity recognition operates on text units and OCR-extracted text and graphics, identifying names of people, organizations, products, places, time expressions, and technical terms, and assigning a unique identifier to each entity. Relation extraction identifies semantic relationships between entities at the sentence and paragraph levels, such as "X causes Y" and "X grows in year T," and records the relation triples as anchor links. Numerical standardization normalizes the units and formats of numerical values ​​in table units and chart data points, such as mapping "thousand" and "k" to a unified expression, recording the original units as annotations, and generating numerical-unit type anchors. Reference node recognition locates footnotes, chart descriptions, and table annotations, and establishes pointing relationships between these references and their corresponding main units, i.e., anchors.

[0062] After establishing anchor points, anchor point matching is performed on units from different modalities to establish a one-to-one correspondence. The matching adopts a multi-dimensional anchor point vector comparison method: entity identifier, numerical anchor point, time identifier, and location identifier are combined into an anchor point signature. The signature comparison is used to determine whether units from different modalities express the same fact. For example, in a report, a table unit is identified as "2019 Product A Sales: 1,200,000" and a data point in a chart is labeled "2019" and corresponds to the same legend. The table unit and the chart data point are matched as the same evidence object through anchor point signature matching (time, product entity, numerical range). The two are merged into a single minimum evidence unit and multi-source proof information is recorded. For fuzzy matching or cases where the values ​​are similar but not completely the same, the semantic anchor point detection unit generates a matching confidence score and retains the original difference information for subsequent cross-modal conflict adjudication module processing.

[0063] For example:

[0064] In the scenario of retrieving company annual reports, the system divides the paragraphs of the annual report text into sentence-level units, tables into table units, and charts into graphic fragments and extracts legends and coordinates. Subsequently, the semantic anchor detection unit identifies entities and values ​​such as "profit," "2019," and "amount." The "Profit 2019: 2.5M" in the table unit and the "2019" data at the chart point are matched as the same evidence unit after numerical standardization. This evidence unit is labeled with source identifiers (document name, page number, or video timestamp), anchor signatures, and matching confidence, and written into the cross-modal index set for subsequent consistency determination and question answer generation.

[0065] The cross-modal association determination module includes a semantic consistency detection unit, a numerical deviation analysis unit, and a contradiction adjudication unit. The semantic consistency detection unit calculates the semantic similarity between evidence from different modalities through a semantic matching network. The numerical deviation analysis unit establishes a numerical comparison relationship between table cells and chart data points. The contradiction adjudication unit performs a weighted adjudication based on the evidence source weight, time relevance, and modal confidence, and outputs the minimum sufficient evidence set with the highest consistency.

[0066] The operation method of the semantic consistency detection unit includes:

[0067] Each unit is loaded with its semantic vector in the cross-modal embedding matrix. The semantic representation of the query and candidates is expanded by combining the unit's context window, such as adjacent sentences in the same paragraph, paragraphs of charts and diagrams, or text a few seconds before and after a video. Then, a semantic similarity score is output using a specially designed similarity function, which can be cosine similarity or a similarity measure with a calibration term. To enhance the robustness of the judgment, the semantic consistency detection unit also adopts multi-scale semantic comparison: comparing the semantic consistency between whole sentences and segments, as well as comparing the matching between local entities, such as subject entities, time words, and key terms. These sub-item scores are combined into the final semantic consistency score. The unit also outputs explanatory information, such as which entities or keywords contribute the most to the similarity, so that the subsequent adjudication unit can provide auditable judgment reasons.

[0068] The operation method of the numerical deviation analysis unit includes:

[0069] All candidate values ​​are standardized, and then a numerical comparison relationship is established: for a set of candidates considered semantically matched, the absolute difference, relative error, and trend consistency in the case of time series are calculated, such as whether they change in the same direction or whether there is a significant lag; in addition, this unit introduces an uncertainty measure, that is, the original confidence level of each numerical source is recorded. The original confidence level comes from the OCR / extraction steps, the confidence label of the source document, etc., and the measurement error range or confidence interval of the numerical value is reflected in the comparison results; for the deviations found, the unit will mark the deviation type, such as rounding error, unit conversion error, source registration error or inconsistency, and provide quantitative evidence for subsequent adjudication, such as "the values ​​in the table and chart differ by 2% and the time is aligned" or "the numerical values ​​have significant deviations and the source confidence is high".

[0070] The operation method of the conflict resolution unit includes:

[0071] The semantic score, numerical bias, source confidence and modal confidence are calibrated to allow them to participate in the calculation of the same weighted model.

[0072] Weights can be based on rules, such as preset source trust levels or time priority principles in laws or corporate standards, or determined through supervised learning methods, that is, using manually verified samples or historical error correction records as labels for optimization during the training phase.

[0073] For each candidate set that expresses the same fact, a weighted composite score is calculated, and the minimum number of evidence units are selected from high to low based on the composite score so that the combined set satisfies the criterion of "sufficiently supporting the query assertion", such as covering key entities, time and numerical anchors.

[0074] When faced with intermodal conflicts, such as a text asserting "increase" while a numerical value shows a decrease, the adjudication unit introduces multiple rules: it prioritizes source weight and time relevance; if the conflict still cannot be resolved, it checks whether the numerical deviation can be caused by units or explanatory annotations; if it can be explained, it tends to merge and record the reason; if the conflict cannot be resolved by automatic rules, it marks the instance as "requiring human intervention" and outputs an evidence difference report.

[0075] The adjudication unit generates an adjudication report for each automatic adjudication, explaining the source of the selected evidence, the semantic score and numerical deviation of each piece of evidence, the weight values, the final score, and the reasons for the adjudication, so as to facilitate review or user tracking.

[0076] Through the synergy of the above three units, this embodiment can balance semantic and numerical consistency judgments, and comprehensively consider source credibility and time sensitivity during adjudication, thereby arriving at conclusions that are both statistically significant and interpretable; for example:

[0077] When searching for "whether Company X's net profit increased in 2019", the semantic consistency detection unit compares the main text paragraphs, chart titles, and table column names, indicating that they all point to "net profit - 2019". The numerical deviation analysis unit performs unit standardization and relative error calculation on the "net profit: 2,500,000" given in the table and the "2.45M" at the chart point, judging it as a small numerical difference and recording the possible source, such as the chart data being rounded. The contradiction adjudication unit, if it identifies that the two are time-aligned and the source is credible, merges the two into a final evidence item with high confidence, and records in the adjudication report: original value, standardized value, size of difference, weight of each source, and final comprehensive score. If there is a statement in the text that contradicts the two, such as claiming "decline", the system will make an automatic adjudication or mark it as a contradictory case that requires manual verification based on the source weight and numerical evidence tendency.

[0078] The timestamp and version identifier constraint mechanism includes consistency constraints in the indexing phase and consistency constraints in the generation phase. The consistency constraints in the indexing phase record the collection time and version source of each data item through time hash identifiers. The consistency constraints in the generation phase perform secondary verification on the time hash and version identifier of the evidence when the inference engine performs question and answer generation to ensure that the source of the cited evidence is completely consistent in terms of time and version.

[0079] The consistency constraint guarantee during the indexing phase involves attaching a time hash identifier to each collected or generated cross-modal data item. This identifier is a fingerprint-like identifier generated by combining several elements according to deterministic rules, mainly including, but not limited to: data collection time, data source identifier (a unique identifier of the source system or document), displayed version number or version identifier provided by the source, content fingerprinting or verification of the data content, and detection of whether the content has changed between two collections. In implementation, the system calculates and writes the time hash identifier at the final stage of the data entry pipeline and stores this identifier along with the data's metadata in the metadata area of ​​the cross-modal index set. Key implementation points include:

[0080] To avoid ambiguity in subsequent judgments, the construction of time hash identifiers follows a fixed field order and normalization strategy. For example, a unified time zone and format are used for time, a unique ID from the registry is used for source identification, and the original source version field or incremental sequence number during crawling is used for version. Before combination, each field is normalized, and then a content fingerprint is generated as part of the final hash. The content fingerprint can use checksums and irreversible digests of the content text and binary to quickly determine whether the content from the same source has undergone substantial changes. For images or complex tables, the fingerprint can be derived from the normalized representation after structured extraction. In distributed collection and indexing scenarios, a unified version registration and time synchronization strategy is adopted, such as a unified clock domain or synchronization server. An idempotent write strategy is used to deal with write conflicts and duplicate collections to ensure that the same data item does not generate multiple mutually exclusive time hashes.

[0081] Methods for implementing consistency constraints during the generation phase include:

[0082] After the retrieval module outputs candidate evidence fragments, the generation module again requests the latest time hash identifier from the index using the unique location identifier of the candidate fragment, and strictly compares it with the time hash returned with the candidate fragment during the retrieval. The unique location identifier can be, for example, document ID + offset / table coordinates / video time segment. If all hashes match, it proves that no version update occurred between retrieval and generation, and the generation process can continue to use this evidence. If inconsistencies exist, such as the time hash of the corresponding position in the index having changed, the system can handle it according to a preset strategy: first, attempt to revert to a historical snapshot at the time of retrieval; if the index retains historical versions; second, mark the candidate evidence as updated and trigger a fast re-retrieval to obtain the latest context; third, if it is impossible to... When verifying source consistency, the evidence is rejected from automatic citation and marked as requiring manual verification. This strategy ensures that the generated results will not cite different versions of evidence that have not been verified twice. The generation module records the time hash identifier of the cited evidence and the index snapshot ID used for verification in the final output as traceable credentials for easy auditing and reproduction. To avoid time-consuming full verification for each generation, this embodiment designs a layered verification system, including: strong consistency verification for high-priority or high-risk scenarios (strong verification path), and asynchronous or near real-time verification and background correction for low-risk or latency-sensitive scenarios (weak verification path). The verification level and consistency status of the evidence are clearly marked in the generated results.

[0083] For example:

[0084] During the indexing phase, the system collects a table from a company's annual report, recording the collection time as "2024-04-10T10:15" and the source identifier as "AnnualReport_v1.2". It also generates a content fingerprint for the table content, synthesizes a time hash A, and writes it into the index. When a user initiates a query and retrieves this table cell as candidate evidence, the retrieval tool returns time hash A along with the candidate. If the annual report is revised and published during the generation phase, with the source updated to "AnnualReport_v1.3" and the content slightly changed, the time hash of the corresponding item in the index changes to B. During secondary verification, the generation module will find that A≠B. Therefore, it will either choose to backtrack to the snapshot at the time of retrieval to reproduce the context at that time, or trigger a re-retrieval and mark the generated output with a processing description stating "The cited evidence was updated before generation; the latest version / backtracked version has been used," ensuring the time consistency and traceability of the output.

[0085] When generating traceable question-and-answer results, the inference engine establishes a fact-evidence mapping table, which includes fact descriptions, evidence unit identifiers, modality types, and corresponding semantic anchor point positions. At the same time, it generates a cross-modal visual backtracking path, which displays the position of the evidence corresponding to the fact in the original data source in a graphic and textual format, enabling verifiable display on the user side.

[0086] The fact-evidence mapping table uses rows to represent single fact mapping entries. Each entry includes at least: fact description, evidence unit identifier, modality type, semantic anchor position, time hash and version identifier, confidence level and source weight, and a summary of association reasons. The generation method of the fact-evidence mapping table includes: selecting the minimum set of evidence units to support the fact from the final evidence set output by the cross-modal association determination module; for each evidence unit, reading its semantic anchor position and modality metadata from the index based on the evidence unit identifier to generate coordinates and descriptions for front-end positioning; and including metadata such as time hash and version identifier, original channel score and rearranged confidence level for post-verification; and automatically synthesizing a short text as a summary of association reasons by combining the output of semantic consistency detection and numerical deviation analysis, explaining how the evidence supports the fact, such as indicating which entities, times and values ​​match.

[0087] For example:

[0088] A user asks, "Did Company X's revenue increase in 2019?" The inference engine generates the assertion "Revenue increased in 2019 compared to the previous year" and records three evidence units in the fact-evidence mapping table: a text paragraph (page number 10, sentence offset position), a table unit (row 3, column 2), and a chart data point (data point index 7). The mapping table records the semantic anchor position (character offset of the text, row and column coordinates of the table, index of the chart point), time hash (snapshot identifier when the index was entered into the database), and confidence level for each item. The interface displays a summary of facts and reasons on the left and three pieces of evidence side by side on the right, each with a "Jump to view the original text" button. When the user clicks on a table unit, the third row and second column will be highlighted in the embedded table view, and the original value and source time hash will be displayed, thus quickly verifying the reasoning basis.

[0089] Example 2:

[0090] like Figure 2 As shown, this embodiment further improves upon the design of Embodiment 1. The difference lies in the fact that, in the actual operation of Embodiment 1, it was found that when the query content involves cross-document communication and includes multimodal evidence association, the candidate evidence set obtained from a single retrieval suffers from insufficient coverage and a disconnect from logical association. This results in the inference engine lacking key evidence when generating traceable question-and-answer results, failing to guarantee the closure of the inference chain and the credibility of the conclusions, and failing to provide complete evidence support in complex question scenarios. Based on this, the multimodal fusion intelligent question-and-answer and knowledge retrieval method also includes:

[0091] An evidence completion mechanism based on reasoning feedback is introduced, which includes:

[0092] When constructing the fact-evidence mapping table, the inference engine performs anchor point completeness verification on the fact descriptions; when it detects that entity anchor points, time anchor points, or numerical anchor points are not covered in the existing evidence set, it generates corresponding supplementary query instructions.

[0093] The supplementary query instruction is submitted to the multichannel search engine, which performs a supplementary search in the cross-modal index set according to the instruction to obtain supplementary evidence units;

[0094] The supplementary evidence unit undergoes semantic consistency verification and time version verification via the consistency determination module. Only when the verification passes will the supplementary evidence be added to the current evidence set.

[0095] The inference engine reconstructs the inference chain based on the updated evidence set. Once all anchor points are covered and a logical closure is formed, a traceable question-and-answer result is output.

[0096] Through the above design, evidence can be continuously supplemented and rigorously screened during the reasoning process, ensuring that under the questioning conditions of complex cross-modal associations and a wide range of knowledge, the question-and-answer output can achieve complete factual expression and traceable evidence links.

[0097] Example 3:

[0098] like Figure 3 As shown, based on the same inventive concept as the mine personnel positioning and environmental parameter linkage early warning method in the foregoing embodiments, this application provides a mine personnel positioning and environmental parameter linkage early warning method system. The system and method embodiments in this application are based on the same inventive concept. The system includes:

[0099] Heterogeneous Index Module: The heterogeneous index module establishes a multimodal data indexing model for heterogeneous knowledge sources, and performs feature mapping on text, images, tables, charts, audio and video content through a unified semantic embedding space to generate a cross-modal index set;

[0100] Semantic retrieval module: After receiving a query request, the semantic retrieval module uses a multi-channel retrieval device to perform semantic and structural matching on the cross-modal index set to obtain candidate evidence fragments;

[0101] Granularity decomposition module: Based on the evidence granularity decomposition strategy, the granularity decomposition module divides the text statements, table units, chart data points and multimedia frame content in the candidate evidence fragments into the smallest evidence units, and establishes a semantic consistency graph between each unit.

[0102] The association adjudication module performs evidence aggregation and contradiction adjudication on the semantic consistency graph through the cross-modal association judgment module, generating a minimum sufficient set of evidence that satisfies logical constraints;

[0103] Traceable Generation Module: During the generation phase, the inference engine performs traceable question-and-answer generation based on the minimum sufficient evidence set, and generates a fact-evidence mapping table and a cross-modal visual backtracking path.

[0104] Spatiotemporal Consistency Module: Throughout the entire retrieval and generation process, the spatiotemporal consistency module introduces a timestamp and version identifier constraint mechanism to perform spatiotemporal consistency verification and version locking on the evidence source, ensuring that the output content is consistent with the original data source in both time and version dimensions.

[0105] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0106] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present application, based on the technical solution and concept of the present application, should be covered within the scope of protection of the present application.

Claims

1. A multimodal fusion-based intelligent question answering and knowledge retrieval method, characterized in that the method... include: Establish a multimodal data indexing model for heterogeneous knowledge sources, and generate a cross-modal index set by feature mapping of text, images, tables, charts, audio and video content through a unified semantic embedding space; After receiving a query request, a multi-channel retrieval tool is used to perform semantic and structural matching on the cross-modal index set to obtain candidate evidence fragments. Based on the evidence granularity decomposition strategy, the text sentences, table units, chart data points, and multimedia frame content in the candidate evidence fragments are divided into the smallest evidence units, and a semantic consistency graph is established between each unit. The evidence granularity decomposition strategy includes a hierarchical segmentation unit and a semantic anchor detection unit. The hierarchical segmentation unit is based on document structure identification, visual segmentation, and audio and video frame sequence feature extraction algorithms to subdivide the document and multimedia content into sentence-level, element-level, and frame-level evidence units. The semantic anchor detection unit labels the key concepts, values, and reference nodes in each unit according to the entity recognition model and the relation extraction model, thereby realizing a one-to-one correspondence between cross-modal units. The cross-modal association determination module performs evidence aggregation and contradiction adjudication on the semantic consistency graph to generate a minimum sufficient set of evidence that satisfies logical constraints. During the generation phase, the inference engine performs traceable question-and-answer generation based on the minimum sufficient evidence set, and generates a fact-evidence mapping table and a cross-modal visual backtracking path. Throughout the entire retrieval and generation process, a timestamp and version identification constraint mechanism is introduced to perform spatiotemporal consistency verification and version locking on the evidence source, ensuring that the output content is consistent with the original data source in terms of time and version dimensions.

2. The multimodal fusion intelligent question answering and knowledge retrieval method according to claim 1, characterized in that, The multimodal data indexing model includes a semantic embedding sub-model and a structural embedding sub-model. The semantic embedding sub-model uses a cross-modal alignment network to achieve a unified semantic representation of text, images, and table titles. The structural embedding sub-model describes the relationships between table rows and columns, chart coordinates, and time axis based on a hierarchical graph structure encoding method. The two are concatenated by features to form a trainable cross-modal embedding matrix, which is used to construct the cross-modal index set.

3. The multimodal fusion intelligent question answering and knowledge retrieval method according to claim 1, characterized in that, The multi-channel retrieval system includes a semantic retrieval channel, a structural retrieval channel, and a contextual rearrangement channel. The semantic retrieval channel uses a cross-modal similarity metric function to calculate the embedding distance between text and visual features. The structural retrieval channel performs relationship matching based on table structure and chart coordinate constraints. The contextual rearrangement channel uses the query intent vector to perform relevance rearrangement and confidence weighting on the outputs of the first two channels, thereby generating the final set of candidate evidence fragments.

4. The multimodal fusion intelligent question answering and knowledge retrieval method according to claim 1, characterized in that, The cross-modal association determination module includes a semantic consistency detection unit, a numerical deviation analysis unit, and a contradiction adjudication unit. The semantic consistency detection unit calculates the semantic similarity between different modal evidences through a semantic matching network; the numerical deviation analysis unit establishes a numerical comparison relationship between table cells and chart data points; and the contradiction adjudication unit performs a weighted adjudication based on the evidence source weight, time recentity, and modal confidence, and outputs the minimum sufficient evidence set with the highest consistency.

5. The multimodal fusion intelligent question answering and knowledge retrieval method according to claim 1, characterized in that, The timestamp and version identifier constraint mechanism includes consistency constraints in the indexing stage and consistency constraints in the generation stage. The consistency constraints in the indexing stage record the collection time and version source of each data item through time hash identifiers. The consistency constraints in the generation stage perform secondary verification on the time hash and version identifier of the evidence when the inference engine performs question and answer generation to ensure that the source of the cited evidence is completely consistent in terms of time and version.

6. The multimodal fusion intelligent question answering and knowledge retrieval method according to claim 1, characterized in that, When generating traceable question-and-answer results, the inference engine establishes a fact-evidence mapping table, which includes fact descriptions, evidence unit identifiers, modality types, and corresponding semantic anchor point positions. At the same time, it generates a cross-modal visual backtracking path, which displays the position of the evidence corresponding to the fact in the original data source in a graphic and textual format, enabling verifiable display on the user side.

7. The multimodal fusion intelligent question answering and knowledge retrieval method according to claim 1, characterized in that, The method also includes: An evidence completion mechanism based on reasoning feedback is introduced, which includes: When constructing the fact-evidence mapping table, the inference engine performs anchor point completeness verification on the fact descriptions; When it is detected that an entity anchor, time anchor, or numerical anchor is not covered in the existing evidence set, a corresponding supplementary query instruction is generated. The supplementary query instruction is submitted to the multichannel search engine, which performs a supplementary search in the cross-modal index set according to the instruction to obtain supplementary evidence units; The supplementary evidence unit undergoes semantic consistency verification and time version verification via the consistency determination module. Only when the verification passes will the supplementary evidence be added to the current evidence set. The inference engine reconstructs the inference chain based on the updated evidence set. Once all anchor points are covered and a logical closure is formed, a traceable question-and-answer result is output.

8. A multimodal fusion intelligent question-answering and knowledge retrieval system, characterized in that the system... include: The heterogeneous index module establishes a multimodal data index model for heterogeneous knowledge sources, and performs feature mapping on text, images, tables, charts, audio and video content through a unified semantic embedding space to generate a cross-modal index set. The semantic retrieval module, after receiving a query request, uses a multi-channel retrieval device to perform semantic matching and structural matching on the cross-modal index set to obtain candidate evidence fragments; The granularity decomposition module, based on an evidence granularity decomposition strategy, divides text sentences, table units, chart data points, and multimedia frame content in candidate evidence fragments into the smallest evidence units and establishes a semantic consistency graph between each unit. The evidence granularity decomposition strategy includes a hierarchical segmentation unit and a semantic anchor detection unit. The hierarchical segmentation unit, based on document structure identification, visual segmentation, and audio / video frame sequence feature extraction algorithms, subdivides documents and multimedia content into sentence-level, element-level, and frame-level evidence units. The semantic anchor detection unit, based on entity recognition and relation extraction models, labels key concepts, values, and reference nodes in each unit, thereby achieving a one-to-one correspondence between cross-modal units. The association adjudication module performs evidence aggregation and contradiction adjudication on the semantic consistency graph through the cross-modal association judgment module, generating a minimum sufficient evidence set that satisfies logical constraints; The traceable generation module, in the generation phase, the inference engine performs traceable question-and-answer generation based on the minimum sufficient evidence set, and at the same time generates a fact-evidence mapping table and a cross-modal visual backtracking path; The spatiotemporal consistency module introduces a timestamp and version identifier constraint mechanism throughout the entire retrieval and generation process. It performs spatiotemporal consistency verification and version locking on the evidence source to ensure that the output content is consistent with the original data source in terms of time and version dimensions.

Images