A domain knowledge fusion and semantic question answering system based on text and multi-modal large model

Abstract

This invention provides a domain knowledge fusion and semantic question answering system based on text and a multimodal large model, relating to the field of electronic digital data processing. It includes a multimodal data intelligent preprocessing and feature extraction module, a domain knowledge fusion and semantic enhancement module, an intelligent reasoning and answer generation module, and a quality monitoring and adaptive optimization module. The multimodal data intelligent preprocessing and feature extraction module is responsible for unified access, deep feature extraction, and preliminary alignment of heterogeneous data sources. The domain knowledge fusion and semantic enhancement module is responsible for constructing a domain knowledge base and achieving deep fusion of knowledge and multimodal features. The intelligent reasoning and answer generation module is responsible for question decomposition, semantic reasoning, and answer generation. The quality monitoring and adaptive optimization module is responsible for consistency detection, confidence assessment, and system adaptive optimization. This system can significantly improve the accuracy and reliability of domain knowledge question answering systems.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, specifically to a domain knowledge fusion and semantic question answering system based on text and multimodal large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large models have demonstrated powerful capabilities in processing complex information tasks. Traditional question-answering systems mainly rely on single-modal data, making it difficult to fully utilize multiple information sources such as text, images, and tables, resulting in limited accuracy in specialized domains. Integrating domain knowledge with multimodal features to enhance semantic understanding and reasoning capabilities has become an important research direction for intelligent question-answering systems.

[0003] Existing multimodal question answering technologies mainly fall into three categories. The first category is based on early fusion, which fuses raw data from different modalities during the feature extraction stage, generating a unified feature representation through a multimodal encoder. Typical examples include VisualBERT and VL-BERT, which concatenate image region features and text token features at the input layer and then perform joint encoding using a Transformer. The advantage of this type of method is its ability to capture cross-modal interactions at an early stage, but it suffers from high computational complexity and stringent preprocessing requirements for different modal data; poor quality data from one modality can negatively impact overall performance. The second category is based on late fusion, which processes each modality separately and fuses the data only in the later decision-making stage. For example, it uses independent image and text encoders to extract features, then generates the final answer through simple concatenation, weighting, or voting mechanisms. The advantage of this type of method is that each modality is processed relatively independently, but cross-modal interactions are insufficient, making it difficult to capture complex semantic relationships between modalities. The third category is based on hybrid fusion, which combines the advantages of early and late fusion, performing cross-modal interactions in the intermediate layer of feature extraction. Representative methods such as LXMERT and UNITER employ specialized cross-attention modules to exchange information between modalities multiple times during the encoding process. These methods typically outperform the previous two types in terms of performance, but their fusion mechanism is mainly based on the soft association of attention weights, which fails to adequately utilize structured knowledge and lacks the ability to adaptively adjust the importance of modalities during the fusion process.

[0004] US Patent 11915123B2 discloses a method for fusing multimodal data using recurrent neural networks. This patent establishes data feeds for different modalities, encodes textual data and time-series data into vector sequences, and then uses a recurrent neural network to process the encoded multimodal information for fusion and inference. While this patent proposes a basic framework for multimodal data fusion, it suffers from the following shortcomings: The multimodal fusion uses a fixed network structure, lacking an adaptive adjustment mechanism for the importance of different modalities, and cannot dynamically optimize the fusion strategy based on specific tasks and data characteristics; in terms of knowledge utilization, it lacks an integration mechanism with external knowledge sources, relying solely on implicit knowledge learned internally by the model, which limits its ability to handle question-answering tasks requiring domain-specific knowledge; the inference process lacks explicit logical reasoning chain modeling, making it difficult to handle complex problems requiring multi-step reasoning; and confidence assessment mainly relies on model output probabilities, without establishing a comprehensive mechanism for evaluating inference quality, making it unable to effectively identify low-confidence inference results.

[0005] Chinese patent CN118689997B discloses a multimodal question-answering explanation method. This method performs logical reasoning based on target visual content and a target question to generate an initial answer and an initial explanation text describing the logical reasoning process. It then iteratively optimizes from the initial answer and explanation text to obtain the target answer and target explanation text. In each iteration, an evaluation value representing the semantic relevance of the answer and explanation text for that iteration is predicted. If the evaluation value exceeds a set threshold, the answer and explanation text for that iteration are used as the target answer and target explanation text; otherwise, logical reasoning is performed again based on the evaluation value and the explanation text for that iteration to generate the answer and explanation text for the next iteration. This patent explores iterative optimization and interpretability in the reasoning process, but it has the following shortcomings: multimodal feature processing adopts conventional encoding and fusion methods, without differentiated alignment and fusion of features at different levels of abstraction, resulting in insufficient modeling depth for cross-modal semantic associations; knowledge utilization lacks a dynamic mechanism, failing to adaptively adjust the scope and quality of knowledge according to the progress of reasoning, leading to potential redundancy or omissions in knowledge recall; iterative optimization is mainly based on a single semantic relevance evaluation index, without comprehensively considering multiple dimensions such as the rigor of reasoning logic, the sufficiency of evidence, and the certainty of conclusions, resulting in an insufficiently comprehensive evaluation of reasoning quality; the integration of the reasoning process with the knowledge graph is not close enough, failing to fully utilize the structured information of the knowledge graph to guide the selection of reasoning paths and the generation of answers. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings by proposing a domain knowledge fusion and semantic question answering system based on text and a multimodal large model.

[0007] The present invention adopts the following technical solution:

[0008] A domain knowledge fusion and semantic question answering system based on text and multimodal large models includes a multimodal data intelligent preprocessing and feature extraction module, a domain knowledge fusion and semantic enhancement module, an intelligent reasoning and answer generation module, and a quality monitoring and adaptive optimization module.

[0009] The multimodal data intelligent preprocessing and feature extraction module is responsible for unified access, deep feature extraction and preliminary alignment of heterogeneous data sources; the domain knowledge fusion and semantic enhancement module is responsible for building a domain knowledge base and realizing deep fusion of knowledge and multimodal features; the intelligent reasoning and answer generation module is responsible for performing problem decomposition, semantic reasoning and answer generation; and the quality monitoring and adaptive optimization module is responsible for consistency detection, confidence assessment and system adaptive optimization.

[0010] The multimodal data intelligent preprocessing and feature extraction module includes a heterogeneous data source access and identification unit, a submodal deep feature extraction unit, and a cross-modal preliminary alignment and normalization unit. The heterogeneous data source access and identification unit is used to automatically receive and identify the modality type, format specification, and quality level of the data. The submodal deep feature extraction unit is used to extract feature vector sequences specific to each modality. The cross-modal preliminary alignment and normalization unit is used to map the features of different modalities to a unified dimension representation space.

[0011] The domain knowledge fusion and semantic enhancement module includes a domain knowledge base construction and indexing unit, a problem-aware knowledge precise retrieval unit, and a multimodal feature and knowledge deep fusion unit. The domain knowledge base construction and indexing unit is used to establish an efficient knowledge indexing mechanism that supports semantic retrieval and relational reasoning. The problem-aware knowledge precise retrieval unit is used to parse the problem intent and generate a knowledge demand graph. The multimodal feature and knowledge deep fusion unit is used to effectively inject domain knowledge into the multimodal feature representation.

[0012] The intelligent reasoning and answer generation module includes a problem decomposition and reasoning path planning unit, a knowledge-enhanced semantic reasoning unit, and a multi-strategy answer generation and integration unit. The problem decomposition and reasoning path planning unit is used to output a structured reasoning plan to guide the subsequent reasoning process. The knowledge-enhanced semantic reasoning unit performs logical reasoning based on the fused semantic representation to generate intermediate conclusions with complete reasoning chains. The multi-strategy answer generation and integration unit generates multiple candidate answers from different reasoning paths and generates the final answer through an intelligent integration strategy.

[0013] The quality monitoring and adaptive optimization module includes a multimodal consistency detection and conflict resolution unit, an inference result verification and confidence assessment unit, and a user feedback-driven system adaptive optimization unit. The multimodal consistency detection and conflict resolution unit automatically detects semantic conflicts between data of different modalities. The inference result verification and confidence assessment unit is used to verify the inference results and calculate the comprehensive confidence score. The user feedback-driven system adaptive optimization unit is used to dynamically update the knowledge retrieval strategy, inference path selection, and generation control parameters.

[0014] Furthermore, the cross-modal preliminary alignment and normalization unit includes a dimension mapping processor, a semantic alignment processor, and a feature normalization processor. The dimension mapping processor maps feature vectors of different modalities to a unified representation space of the same dimension. The semantic alignment processor is used to learn the semantic correspondence between different modalities to achieve coarse-grained semantic alignment. The feature normalization processor performs numerical range normalization and distribution standardization on the aligned feature vectors.

[0015] The semantic alignment processor calculates the hierarchical weighted semantic distance D(f1, f2) between two different modal features according to the following formula:

[0016] ;

[0017] Where f1 and f2 represent feature vectors of two different modalities, and L is the total level of the feature vectors. This represents the i-th layer feature in the feature vector f1. w represents the i-th layer feature in feature vector f2. i Let be the alignment weight of the i-th layer. To smooth out the penalty coefficient.

[0018] Furthermore, the problem-aware knowledge retrieval unit includes a problem intent parsing processor, a knowledge demand planning processor, and a multi-hop knowledge retrieval processor. The problem intent parsing processor performs deep semantic analysis on the user-input problem to generate a structured problem representation to guide the knowledge retrieval process. The knowledge demand planning processor plans the scope of knowledge to be recalled and the depth of reasoning based on the problem intent analysis results. The multi-hop knowledge retrieval processor performs multi-hop relationship reasoning and path search starting from the seed nodes in the knowledge demand graph, recalling the most relevant knowledge subgraphs and filtering out irrelevant knowledge.

[0019] For node k in the knowledge graph i The multi-hop knowledge retrieval processor calculates the retention probability of a node in the t-th inference stage according to the following formula:

[0020] ;

[0021] Where q represents the user question, H t This represents the entire reasoning history from the beginning to the current stage. It is the Sigmoid activation function. For learnable weight vectors, N(k) represents the five-dimensional feature vector of a node. i ) represents node k i The set of neighboring nodes, The influence coefficient of the neighbor For the previous stage node k j The retention probability.

[0022] Furthermore, the multimodal feature and knowledge deep fusion unit includes a semantic anchor discovery processor, a cross-attention fusion processor, and a unified representation generation processor. The semantic anchor discovery processor automatically identifies the semantic correspondence between multimodal features and recalled knowledge entities, and establishes a cross-modal semantic anchor mapping by calculating the similarity between feature vectors and knowledge embedding vectors. The cross-attention fusion processor realizes bidirectional enhancement of multimodal features and knowledge representation, achieving deep interaction and mutual enhancement of features and knowledge at the semantic level. The unified representation generation processor integrates the fused multimodal features and knowledge information into a unified semantic representation vector.

[0023] The cross-attention fusion processor first constructs an interaction tensor T. Where M is the number of modalities, N is the feature sequence length, K is the number of knowledge nodes recalled, and the tensor elements are defined according to the following formula:

[0024] ;

[0025] in, Let k represent the nth eigenvector of mode m. k W is the embedding vector of knowledge node k. fus The fusion weight matrix is ​​learnable;

[0026] The cross-attention fusion processor performs Tucker decomposition on the interaction tensor T according to the following formula:

[0027] ;

[0028] in, For the core tensor, , , It consists of three factor matrices, with R1, R2, and R3 being the compressed dimensions. Representing tensor-matrix multiplication along the i-th dimension, Tucker decomposition decomposes the high-dimensional interaction tensor into a low-rank core structure and three pattern matrices, capturing "modal patterns", "feature patterns" and "knowledge patterns" respectively.

[0029] Based on the decomposition results, the unified representation generation processor calculates the knowledge-enhanced unified semantic representation according to the following formula:

[0030] ;

[0031] Where K = [k1, ..., k K ] represents the knowledge matrix.

[0032] Furthermore, the reasoning result verification and confidence assessment unit includes a logical consistency verification processor, an evidence sufficiency check processor, and a comprehensive confidence assessment processor. The logical consistency verification processor checks whether the reasoning result conforms to formal logic rules and domain knowledge constraints. The evidence sufficiency check processor assesses whether the evidence supporting the reasoning conclusion is sufficient. The comprehensive confidence assessment processor is used to calculate the comprehensive confidence of the reasoning result and quantify the sources of uncertainty.

[0033] For the inference chain C = {s1, s2, ..., s...} A The comprehensive confidence assessment processor constructs a probability distribution of the result for each inference step according to the following formula:

[0034] ;

[0035] Where A is the length of the inference chain, and y is the candidate result. e is the summation variable that iterates through all candidate results y h is the embedding vector of the candidate results. i For the fusion representation of the i-th step of reasoning, For smoothing parameters, Let represent the probability of obtaining the result y in the i-th step, given the known reasoning history of the previous i-1 steps. Let be the similarity between the candidate result y and the mode m in step i. Let m be the weight exponent of mode m in step i;

[0036] The logical consistency verification processor calculates the consistency H(C) of the inference process according to the following formula:

[0037] ;

[0038] in, Let D represent the Shannon entropy at step i. KL () represents the KL divergence between adjacent reasoning steps. This is the consistency weighting coefficient;

[0039] The comprehensive confidence assessment processor models the probability distribution described above and calculates the comprehensive confidence level C according to the following formula. final (C):

[0040] ;

[0041] in, Let represent the optimal candidate result for step i.

[0042] The beneficial effects achieved by this invention are:

[0043] This system achieves precise alignment of multimodal features at different semantic levels through a hierarchical adaptive alignment mechanism. It captures high-order statistical relationships between features using covariance matrix similarity, thus preserving fine-grained semantic information and improving the accuracy of cross-modal feature matching. Employing a reasoning-driven dynamic knowledge pruning mechanism, it achieves real-time evaluation of knowledge node quality through five-dimensional node feature vectors and neighbor suppression factors. Combined with a dynamic threshold strategy, it adaptively adjusts the knowledge scope based on reasoning progress, significantly improving knowledge utilization efficiency and reasoning accuracy. By constructing an interaction tensor with modality, feature sequence, and knowledge node as three independent dimensions and extracting core interaction patterns through Tucker decomposition, it overcomes the limitations of existing two-dimensional fusion frameworks, achieving deep semantic fusion of multimodal features and knowledge, and enhancing the system's ability to express complex semantic relationships. Using a structured entropy multi-factor confidence evaluation mechanism, it captures the uncertainty of the reasoning process through probability distribution modeling and comprehensively considers the optimal path probability, reasoning chain consistency, and knowledge quality to calculate the comprehensive confidence score, enabling a more accurate assessment of the reliability of the reasoning results.

[0044] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the overall structural framework of the present invention;

[0046] Figure 2 This is a schematic diagram of the multimodal data intelligent preprocessing and feature extraction module of the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the structure of the knowledge fusion and semantic enhancement module in this invention.

[0048] Figure 4 This is a schematic diagram of the intelligent reasoning and answer generation module of the present invention;

[0049] Figure 5 This is a schematic diagram of the quality monitoring and adaptive optimization module of the present invention;

[0050] Figure 6 This is a schematic diagram comparing the accuracy of the present invention with other methods on different datasets;

[0051] Figure 7 This diagram illustrates the correlation between the confidence levels of the present invention and other methods. Detailed Implementation

[0052] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention. Furthermore, the accompanying drawings of the present invention are for simple illustrative purposes only and are not depictions of actual dimensions; this is stated beforehand. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention.

[0053] Example 1.

[0054] A domain knowledge fusion and semantic question answering system based on text and multimodal large models, combining... Figure 1 It includes a multimodal data intelligent preprocessing and feature extraction module, a domain knowledge fusion and semantic enhancement module, an intelligent reasoning and answer generation module, and a quality monitoring and adaptive optimization module;

[0055] The multimodal data intelligent preprocessing and feature extraction module is responsible for unified access, deep feature extraction and preliminary alignment of heterogeneous data sources; the domain knowledge fusion and semantic enhancement module is responsible for building a domain knowledge base and realizing deep fusion of knowledge and multimodal features; the intelligent reasoning and answer generation module is responsible for performing problem decomposition, semantic reasoning and answer generation; and the quality monitoring and adaptive optimization module, as a guarantee mechanism throughout the entire process, is responsible for consistency detection, confidence assessment and system adaptive optimization.

[0056] Combination Figure 2The multimodal data intelligent preprocessing and feature extraction module includes a heterogeneous data source access and recognition unit, a submodal deep feature extraction unit, and a cross-modal preliminary alignment and normalization unit. The heterogeneous data source access and recognition unit accesses data from different modalities such as text, images, tables, and audio through a unified interface, automatically identifies the modality type, format specifications, and quality level of the data, adds metadata tags to each type of data, and outputs a raw data stream with complete descriptive information. The submodal deep feature extraction unit calls corresponding dedicated encoders for different modalities, including text encoders, visual encoders, and table encoders, to extract multi-level feature representations from low-level features to high-level semantic features, and outputs feature vector sequences specific to each modality. The cross-modal preliminary alignment and normalization unit maps the features of different modalities to a unified dimension representation space based on unsupervised learning methods, performs coarse-grained semantic alignment operations between modalities, lays the foundation for subsequent fine-grained fusion, and outputs a normalized multimodal feature set.

[0057] Combination Figure 3 The domain knowledge fusion and semantic enhancement module includes a domain knowledge base construction and indexing unit, a problem-aware knowledge precision retrieval unit, and a multimodal feature and knowledge deep fusion unit. The domain knowledge base construction and indexing unit is responsible for constructing a hierarchical domain knowledge graph containing a general layer, a domain layer, and a task layer, establishing an efficient knowledge indexing mechanism that supports semantic retrieval and relational reasoning, and providing incremental knowledge updates and version management functions. The problem-aware knowledge precision retrieval unit generates a knowledge demand graph by parsing the problem intent, performs multi-hop reasoning to retrieve the minimum necessary knowledge subgraph, effectively avoids interference from irrelevant knowledge in the reasoning process, and achieves accurate knowledge positioning and efficient retrieval. The multimodal feature and knowledge deep fusion unit automatically discovers the correspondence between multimodal features and knowledge entities, achieves bidirectional enhancement of features and knowledge through a cross-attention mechanism, effectively injects domain knowledge into the multimodal feature representation, and outputs a unified semantic representation with knowledge enhancement for subsequent reasoning.

[0058] Combination Figure 4The intelligent reasoning and answer generation module includes a problem decomposition and reasoning path planning unit, a knowledge-enhanced semantic reasoning unit, and a multi-strategy answer generation and integration unit. The problem decomposition and reasoning path planning unit automatically decomposes complex problems into a sequence of sub-problems that can be reasoned independently, plans a corresponding reasoning path for each sub-problem, including strategies such as single-hop reasoning, multi-hop reasoning, and parallel reasoning, and outputs a structured reasoning plan to guide the subsequent reasoning process. The knowledge-enhanced semantic reasoning unit performs logical reasoning based on the fused semantic representation, uses the attention mechanism of the large model for cross-modal knowledge reference and reasoning, generates intermediate conclusions with complete reasoning chains, and ensures the traceability of the reasoning process. The multi-strategy answer generation and integration unit generates multiple candidate answers from different reasoning paths, comprehensively evaluates and ranks the candidate answers based on dimensions such as confidence, completeness, and relevance, generates the final answer through intelligent integration strategies, and supports multiple granularity answer presentation formats such as summary, detailed, and report formats.

[0059] Combination Figure 5 The quality monitoring and adaptive optimization module includes a multimodal consistency detection and conflict resolution unit, an inference result verification and confidence assessment unit, and a user feedback-driven system adaptive optimization unit. The multimodal consistency detection and conflict resolution unit automatically detects semantic conflicts between different modal data, such as inconsistencies between text descriptions and image content. It resolves conflicts based on multiple dimensions such as confidence, timeliness, and authority, and outputs a consistency verification report and correction suggestions to ensure the coordination and unity of multimodal information. The inference result verification and confidence assessment unit verifies the inference results from multiple dimensions such as logical consistency, evidence sufficiency, and domain rule compliance. It calculates a comprehensive confidence score and labels the sources of uncertainty. For low-confidence results, it triggers a re-inference or manual intervention mechanism to ensure the reliability of the answer quality. The user feedback-driven system adaptive optimization unit captures user satisfaction evaluations, correction opinions, and supplementary information for the system answers. It dynamically updates the knowledge retrieval strategy, inference path selection, and generation control parameters through a reinforcement learning mechanism to achieve continuous iterative improvement and adaptive optimization of system performance.

[0060] The heterogeneous data source access and identification unit includes a data source adaptation processor, a modality type identification processor, and a metadata annotation processor. The data source adaptation processor is responsible for providing a unified data access interface, supporting the reading of data from various data sources such as local file systems, cloud storage, databases, and API interfaces, and converting data from different sources into the system's internal standard data format. The modality type identification processor automatically determines whether the data belongs to text, image, table, audio, or other modal types by analyzing information such as the data's file extension, MIME type, data structure characteristics, and content characteristics, and assigns a corresponding modality tag to each data object. The metadata annotation processor extracts and adds metadata information to the accessed data, including descriptive information such as data source, creation time, file size, format version, and quality score, and generates a complete data description file for use in subsequent processing stages.

[0061] The modality-specific deep feature extraction unit includes an encoder selection processor, a multi-level feature extraction processor, and a feature vector generation processor. The encoder selection processor automatically selects the corresponding dedicated encoder based on the modality type of the data, including a pre-trained language model encoder for text, a convolutional neural network encoder for images, and a structured data encoder for tables, ensuring that each modality obtains the most suitable feature extraction method. The multi-level feature extraction processor performs stepwise feature extraction from shallow to deep layers on the input data. For image modalities, it extracts low-level visual features such as edges and textures, as well as high-level semantic features such as objects and scenes. For text modalities, it extracts shallow features such as lexical and syntactic features, as well as deep features such as semantics and themes. The feature vector generation processor integrates the extracted multi-level features into a fixed-dimensional feature vector sequence. For sequence data, it generates temporal feature vectors; for overall data, it generates global feature vectors, outputting a modality-specific feature representation in a unified format.

[0062] The cross-modal preliminary alignment and normalization unit includes a dimension mapping processor, a semantic alignment processor, and a feature normalization processor. The dimension mapping processor maps feature vectors from different modalities to a unified representation space of the same dimension through linear transformation or nonlinear neural networks, ensuring that each modal feature has a comparable vector dimension. The semantic alignment processor is used to learn the semantic correspondence between different modalities, bringing semantically similar cross-modal features closer together in the representation space to achieve coarse-grained semantic alignment. The feature normalization processor performs numerical range normalization and distribution standardization on the aligned feature vectors to eliminate differences in numerical scale and statistical distribution of different modal features, and outputs a normalized multimodal feature set for use by the subsequent fusion module.

[0063] The semantic alignment processor calculates the hierarchical weighted semantic distance between two different modality features according to the following formula:

[0064] ;

[0065] Where f1 and f2 represent feature vectors of two different modalities, and L is the total level of the feature vectors. This represents the i-th layer feature in the feature vector f1. w represents the i-th layer feature in feature vector f2. i Let be the alignment weight of the i-th layer. To smooth out the penalty coefficient;

[0066] Each layer of features in the feature vector contains a sequence of features, rather than a single number;

[0067] The alignment weight is calculated according to the following formula:

[0068] ;

[0069] in, This represents the importance coefficient of the hierarchy. The consistency coefficient, This represents the semantic consistency index of two feature vectors at the i-th layer.

[0070] The semantic consistency index is calculated according to the following formula:

[0071] ;

[0072] in, Let tr() represent the covariance matrix of the i-th layer features in the feature vector f1, and tr() be the trace of the matrix.

[0073] The domain knowledge base construction and indexing unit includes a knowledge graph construction processor, a hierarchical knowledge organization processor, and a knowledge index generation processor. The knowledge graph construction processor extracts domain entities, concepts, attributes, and relationships from multiple sources such as domain literature, expert rules, and structured databases. It uses technologies such as entity recognition, relationship extraction, and attribute extraction to construct a domain knowledge graph containing triples. The hierarchical knowledge organization processor organizes the constructed knowledge graph into three layers according to the level of abstraction: a general knowledge layer, a domain knowledge layer, and a task knowledge layer. The general knowledge layer contains cross-domain basic concepts, the domain knowledge layer contains professional knowledge of a specific domain, and the task knowledge layer contains rules and templates for specific tasks. The knowledge index generation processor establishes various index structures for the knowledge graph, including inverted indexes based on entity names, vector indexes based on semantic embedding, and relation indexes based on relation types. It supports fast knowledge retrieval and relation reasoning and provides knowledge version management and incremental update mechanisms.

[0074] The problem-aware knowledge retrieval unit includes a problem intent parsing processor, a knowledge requirement planning processor, and a multi-hop knowledge retrieval processor. The problem intent parsing processor performs deep semantic analysis on the user-input question, identifying elements such as question type, key entities, expected answer type, and reasoning depth, and generates a structured question representation to guide the knowledge retrieval process. The knowledge requirement planning processor, based on the problem intent analysis results, locates relevant seed entities and concepts in the domain knowledge graph, plans the knowledge scope and reasoning depth to be retrieved, and generates a knowledge requirement graph to clarify the minimum necessary knowledge set required to answer the question. The multi-hop knowledge retrieval processor starts from the seed nodes in the knowledge requirement graph, performs multi-hop relationship reasoning and path search, calculates the relevance score of each knowledge node to the question, and comprehensively considers factors such as semantic similarity, graph distance, and node confidence to retrieve the most relevant knowledge subgraph and filter irrelevant knowledge, effectively avoiding knowledge pollution problems.

[0075] The multi-hop knowledge retrieval processor starts from seed nodes in the knowledge demand graph, performs multi-hop relationship reasoning and path search, calculates the relevance score of each knowledge node to the question, and recalls the most relevant knowledge subgraph. For node k in the knowledge graph... i The multi-hop knowledge retrieval processor calculates the retention probability of a node in the t-th inference stage according to the following formula:

[0076] ;

[0077] Where q represents the user question, H t This represents the entire reasoning history from the beginning to the current stage. It is the Sigmoid activation function. For learnable weight vectors, N(k) represents the five-dimensional feature vector of a node. i ) represents node k i The set of neighboring nodes, The influence coefficient of the neighbor For the previous stage node k j The probability of retention;

[0078] The five dimensions of the node's five-dimensional feature vector are semantic similarity to the question, minimum graph distance to the used knowledge, indicator function indicating whether it is directly related to the used knowledge, proportion of neighboring nodes in the candidate set, and PageRank score with time decay.

[0079] When the node's retention probability is less than the dynamic threshold When the node is pruned, the dynamic threshold is dynamically adjusted according to the following formula:

[0080] ;

[0081] in, As the initial threshold, The maximum threshold, The growth rate coefficient, This marks a turning point in growth.

[0082] The multimodal feature and knowledge deep fusion unit includes a semantic anchor discovery processor, a cross-attention fusion processor, and a unified representation generation processor. The semantic anchor discovery processor automatically identifies the semantic correspondence between multimodal features and recalled knowledge entities. By calculating the similarity between feature vectors and knowledge embedding vectors, it determines which visual regions correspond to which knowledge entities and which text fragments are associated with which knowledge concepts, and establishes a cross-modal semantic anchor mapping. The cross-attention fusion processor realizes bidirectional enhancement of multimodal features and knowledge representations. Through the cross-attention mechanism, multimodal features focus on relevant knowledge content, while knowledge representation focuses on supporting multimodal evidence, realizing deep interaction and mutual enhancement of features and knowledge at the semantic level. The unified representation generation processor integrates the fused multimodal features and knowledge information into a unified semantic representation vector. This representation contains both the perceptual information of the original multimodal data and the prior information of domain knowledge, outputting a knowledge-enhanced unified semantic representation for use by the inference module.

[0083] The cross-attention fusion processor first constructs an interaction tensor T. Where M is the number of modalities, N is the length of the feature sequence, K is the number of knowledge nodes, and the tensor elements are defined according to the following formula:

[0084] ;

[0085] in, Let k represent the nth eigenvector of mode m. k W is the embedding vector of knowledge node k. fus The fusion weight matrix is ​​learnable;

[0086] The cross-attention fusion processor performs Tucker decomposition on the interaction tensor T according to the following formula:

[0087] ;

[0088] in, For the core tensor, , , It consists of three factor matrices, with R1, R2, and R3 being the compressed dimensions. Representing tensor-matrix multiplication along the i-th dimension, Tucker decomposition decomposes the high-dimensional interaction tensor into a low-rank core structure and three pattern matrices, capturing "modal patterns," "feature patterns," and "knowledge patterns," respectively. Based on the decomposition results, the unified representation generation processor calculates the knowledge-enhanced unified semantic representation according to the following formula:

[0089] ;

[0090] Where K = [k1, ..., k K ] represents the knowledge matrix;

[0091] The problem decomposition and reasoning path planning unit includes a problem complexity assessment processor, a sub-problem decomposition processor, and a reasoning path planning processor. The problem complexity assessment processor analyzes the complexity of the problem by evaluating factors such as the number of entities involved, the hierarchy of relationships, the number of reasoning hops, and the depth of domain knowledge to determine whether the problem needs to be decomposed and the granularity of the decomposition. For simple problems, it directly performs reasoning, while for complex problems, it initiates the decomposition process. The sub-problem decomposition processor decomposes the complex problem into a series of logically related sub-problems, identifies the dependencies and execution order between sub-problems, ensures that the sub-problems are relatively independent and can be reasoned independently, and ensures that the answers to all sub-problems can be combined to form the complete answer to the original problem. The reasoning path planning processor plans a specific reasoning path for each sub-problem, determines whether to use single-hop direct reasoning, multi-hop chain reasoning, or parallel branch reasoning, allocates corresponding knowledge resources and computing resources, and generates a structured reasoning execution plan to guide the subsequent reasoning process.

[0092] The knowledge-enhanced semantic reasoning unit includes a reasoning execution processor, a cross-modal knowledge reference processor, and a reasoning chain generation processor. The reasoning execution processor, based on knowledge-enhanced unified semantic representation and reasoning path planning, invokes the reasoning capabilities of the large model to perform various reasoning operations such as logical deduction, inductive reasoning, and analogical reasoning. It selects appropriate reasoning strategies according to the question type and knowledge characteristics. The cross-modal knowledge reference processor utilizes the attention mechanism within the large model to dynamically reference relevant cross-modal knowledge during the reasoning process, including textual evidence, visual cues, and structured knowledge, ensuring that the reasoning steps are supported by sufficient multimodal evidence and improving the reliability and interpretability of the reasoning results. The reasoning chain generation processor records each reasoning step, the knowledge used, and the intermediate conclusions obtained during the reasoning process, constructing a complete reasoning chain from question to answer. It labels the confidence level and evidence source for each reasoning step and outputs intermediate conclusions with detailed reasoning chains for use by the answer generation module.

[0093] The multi-strategy answer generation and integration unit includes a candidate answer generation processor, an answer evaluation and ranking processor, and an answer integration and output processor. The candidate answer generation processor generates multiple candidate answers based on different reasoning paths and strategies. For the same question, it may generate diverse answer schemes from different angles, granularities, and emphases to ensure the comprehensiveness and robustness of the answers. The answer evaluation and ranking processor performs a multi-dimensional comprehensive evaluation of all candidate answers, calculating the confidence score, completeness score, relevance score, and fluency score of each answer. It comprehensively considers factors such as the logic of the reasoning chain, the sufficiency of evidence, and the accuracy of expression, and ranks the candidate answers and selects the optimal answer. The answer integration and output processor integrates the optimal answer into the corresponding output format according to user needs and question characteristics. It supports summary answers that provide concise conclusions, detailed answers that include reasoning processes and evidence, and report-style answers that provide structured analysis reports. It also adds meta-information such as reasoning chains, knowledge sources, and confidence levels to enhance the interpretability of the answers.

[0094] The multimodal consistency detection and conflict resolution unit includes a cross-modal conflict detection processor, a confidence calculation processor, and a conflict resolution strategy processor. The cross-modal conflict detection processor compares the semantic information transmitted by different modal data, identifies semantic inconsistencies or logical conflicts between text descriptions and image content, and between tabular data and text statements, and discovers potential conflict points through semantic similarity calculation and logical rule checks. The confidence calculation processor calculates a confidence score for the information of each modality, comprehensively considering multiple factors such as the authority of the data source, the timeliness of the data, the completeness of the information, and the accuracy of historical verification, to provide a quantitative basis for the credibility of conflict resolution. The conflict resolution strategy processor determines the conflict resolution scheme based on the confidence score and domain rules, giving priority to the information of high-confidence modalities, and performing weighted fusion or labeling modalities with similar confidence levels as uncertain states for modalities, generating a consistency verification report and correction suggestions to ensure the coordination and consistency of the output information.

[0095] The reasoning result verification and confidence assessment unit includes a logical consistency verification processor, an evidence sufficiency check processor, and a comprehensive confidence assessment processor. The logical consistency verification processor checks whether the reasoning result conforms to formal logic rules and domain knowledge constraints, verifies whether the logical relationship between each step in the reasoning chain is reasonable, detects whether there are problems such as circular reasoning, contradictory conclusions, or logical jumps, and ensures the logical rigor of the reasoning process. The evidence sufficiency check processor assesses whether the evidence supporting the reasoning conclusion is sufficient, checks whether there is enough multimodal evidence to support each reasoning step, evaluates the quality and coverage of the evidence, and identifies weak links with insufficient or singular evidence. The comprehensive confidence assessment processor is used to calculate the comprehensive confidence of the reasoning result and quantify the source of uncertainty. For results below the confidence threshold, a re-reasoning mechanism is triggered or marked as requiring manual intervention, and a verification report with confidence labeling and uncertainty description is output.

[0096] For the inference chain C = {s1, s2, ..., s...} A The comprehensive confidence assessment processor constructs a probability distribution of the result for each inference step according to the following formula:

[0097] ;

[0098] Where A is the length of the inference chain, and y is the candidate result. e is the summation variable that iterates through all candidate results y h is the embedding vector of the candidate results. i For the fusion representation of the i-th step of reasoning, For smoothing parameters, Let represent the probability of obtaining the result y in the i-th step, given the known reasoning history of the previous i-1 steps. Let be the similarity between the candidate result y and the mode m in step i. Let m be the weight exponent of mode m in step i;

[0099] The logical consistency verification processor calculates the consistency H(C) of the inference process according to the following formula:

[0100] ;

[0101] in, Let D represent the Shannon entropy at step i. KL () represents the KL divergence between adjacent reasoning steps. This is the consistency weighting coefficient;

[0102] The comprehensive confidence assessment processor models the probability distribution described above and calculates the comprehensive confidence level C according to the following formula. final (C):

[0103] ;

[0104] in, Let represent the optimal candidate result at step i;

[0105] The user feedback-driven system adaptive optimization unit includes a feedback information acquisition processor, a parameter update strategy processor, and a performance monitoring and evaluation processor. The feedback information acquisition processor captures various types of user feedback to the system output, including explicit feedback such as satisfaction ratings, answer correctness labels, correction suggestions, supplementary information, and usage preferences, as well as implicit feedback such as click behavior, dwell time, and repeated questions. The feedback information is structured and stored as training samples. Based on the collected feedback information, the parameter update strategy processor dynamically updates system parameters through methods such as reinforcement learning, online learning, or transfer learning. These parameters include the relevance weights of knowledge retrieval, strategy parameters for inference path selection, and control parameters for answer generation, enabling the system to continuously optimize performance based on user feedback. The performance monitoring and evaluation processor continuously monitors the system's performance on key indicators such as answer accuracy, response speed, and user satisfaction, evaluates the performance changes brought about by parameter updates, conducts A / B testing to verify the optimization effect, and ensures that the system performance steadily improves without performance degradation during iterative optimization.

[0106] To verify the technical effectiveness of this invention, a set of comparative experiments were designed to evaluate the performance of the method of this invention against existing mainstream methods on three publicly available multimodal question-answering datasets. The datasets used in the experiments included the VQA v2 dataset, containing 204,721 images and 1,105,904 question-answer pairs, primarily used to test visual question-answering capabilities; the OK-VQA dataset, containing 14,031 question-answering samples requiring external knowledge, specifically designed to test knowledge-enhanced question-answering performance; and the self-built domain-specific question-answering dataset MedQA-MM, containing 8,500 multimodal question-answering samples from the medical field, used to verify the application effectiveness in professional domains.

[0107] The experimental comparison methods included four baseline models. The first was the ViLBERT model, which uses a two-stream Transformer architecture for visual-language fusion, representing an early fusion method. The second was the LXMERT model, which enhances cross-modal interaction through a cross-attention mechanism, representing a hybrid fusion method. The third was the MCAN model, which uses a multi-level collaborative attention network, representing an attention enhancement method. The fourth was the KAT model, which combines knowledge graphs and Transformers for knowledge-enhanced question answering, representing a knowledge enhancement method. The method of this invention, referred to as the method in this paper, fully implements all the technical innovations of Equations 1 to 4.

[0108] The experimental procedures were conducted according to standard protocols. First, data preprocessing was performed, dividing the dataset into 70% training, 15% validation, and 15% test sets. Images were uniformly adjusted to a 224×224 resolution, and text was processed using the WordPiece word segmenter with a maximum sequence length of 128. Then, model training was conducted, using the AdamW optimizer for all models with a learning rate of 5×10⁻⁻. 5 The batch size was 32, and training was performed on four NVIDIA A100 GPUs for 50 epochs using a cosine annealing learning rate scheduling strategy. Next, in the inference evaluation phase, the performance of all models was evaluated on the test set. For problems requiring multi-hop inference, the maximum number of inference steps was set to 5, and the confidence threshold was set to 0.75; answers below this threshold were marked as low-confidence results. Finally, an ablation study phase was conducted, in which the various technical innovations of this invention were progressively removed to verify the contribution of each innovation to the overall performance.

[0109] The experimental evaluation metrics adopted standard metrics in the multimodal question-answering domain. Accuracy represents the proportion of correct answers to the total number of questions, used to measure the overall question-answering accuracy. Confidence correlation uses the Pearson correlation coefficient, measuring the correlation between the model's output confidence and the actual correctness; a value closer to 1 indicates a more accurate confidence assessment. The data was processed to obtain... Figure 6 and Figure 7 .

[0110] Example 2.

[0111] This embodiment provides a specific implementation plan for a domain knowledge fusion and semantic question answering system based on text and a multimodal large model. This system is applied to medical image diagnosis assistance scenarios, integrating patient electronic medical record text data, medical image data, laboratory test form data, and a medical knowledge base to achieve intelligent question answering and assisted diagnosis for complex cases. The system includes a multimodal data intelligent preprocessing and feature extraction module, a domain knowledge fusion and semantic enhancement module, an intelligent reasoning and answer generation module, and a quality monitoring and adaptive optimization module. The multimodal data intelligent preprocessing and feature extraction module uses BERT-Large-Uncased as the text encoder to process medical record descriptions, a ResNet-152 pre-trained model as the visual encoder to extract CT image features, and a TabNet network architecture to process test form data. All encoders have undergone secondary processing on a dataset containing 1.2 million medical samples. The pre-training, domain knowledge fusion, and semantic enhancement modules constructed a three-layer medical knowledge graph containing 58,000 medical entities and 230,000 disease-symptom-treatment relationships. The general layer contains 12,000 entities of basic medical knowledge, the domain layer contains 38,000 entities of professional disease knowledge from various departments, and the task layer contains 8,000 rule nodes of diagnostic processes and medication rules. The intelligent reasoning and answer generation module uses GPT-4-Turbo as the inference engine and is configured with a maximum context length of 32,768 tokens. The quality monitoring and adaptive optimization module sets the confidence threshold to 0.82. After three months of actual deployment in the radiology department of a tertiary hospital, the system processed 15,200 diagnostic consultation requests. The accuracy of diagnostic suggestions increased from 74.3% to 89.7%, the average generation time decreased from 4.2 minutes to 1.8 seconds, and the doctor satisfaction score increased from 3.6 points to 4.5 points out of 5.

[0112] The multimodal data intelligent preprocessing and feature extraction module receives heterogeneous data from the hospital information system, including XML format medical record text provided by the electronic medical record system, DICOM format medical images output by the PACS system, HL7 format test reports generated by the LIS system, and JPG format symptom photos uploaded by patients via an app. The data source adaptation processor is configured with dedicated adapters for the above four data sources. The DICOM adapter automatically extracts metadata information such as window width and window level and converts the image data into a standard PNG format image sequence. The HL7 adapter parses the OBX paragraph in the report, extracts the test item names and numerical results, and converts them into structured JSON objects. The modality type recognition processor has a built-in rule base containing recognition rules for 128 file formats and a ResNet-18-based binary classification neural network model for handling unknown formats. The data recognition accuracy reached 99.2%. The metadata annotation processor generated a metadata description file containing 32 fields for each data object. Key fields included the data unique identifier UUID, the hospital department code of the data source, the model and parameters of the acquisition device, the acquisition timestamp accurate to milliseconds, the data quality score using a continuous score from 0 to 100, the patient anonymized ID encrypted with SHA-256, and the associated medical record number and examination request number. The quality scoring mechanism evaluated six indicators for image data, including whether the resolution was greater than 512×512 pixels, whether the contrast was within the recommended range, and whether there was artifact interference. For text data, it evaluated four indicators, including completeness, standardization, and readability. For tabular data, it evaluated three indicators, including data missing rate and numerical reasonableness. The quality score was calculated comprehensively, and a manual review process was triggered when the score was below 60.

[0113] The multimodal depth feature extraction unit utilizes 17 customized encoder configurations tailored for different modalities of data, specifically designed for the medical field. For the largest portion of the data, CT image data, the Med3D encoder is employed. This encoder, based on the 3DResNet-50 architecture, can simultaneously process the three-dimensional spatial information of CT sequences. The multi-level feature extraction processor is configured to extract feature representations at five levels: Level 1 extracts edge gradient features and texture statistics to generate a 512-dimensional feature vector; Level 2 extracts morphological features of organs and tissues to generate a 1024-dimensional feature vector; Level 3 extracts local features of the lesion region to generate a 1024-dimensional feature vector; Level 4 extracts features related to the relationship between the lesion and surrounding tissues to generate a 768-dimensional feature vector; and Level 5 extracts overall features... The system generates a 256-dimensional feature vector from semantic features. For text modalities, the Bio-BERT encoder extracts four levels of features, from 128-dimensional character level to 768-dimensional paragraph level. For tabular data, the LabNet encoder uses the TabNet architecture to extract three levels of features, from 64-dimensional individual indicators to 256-dimensional comprehensive evaluation. The five-layer features of CT images are fused into a 2048-dimensional global feature vector through weighted averaging, while retaining the local features of each layer, totaling 3584 dimensions for fine-grained analysis. For text, the four-layer features are fused into a 1536-dimensional global feature vector through pooling. For tables, the three-layer features are directly concatenated to form a 576-dimensional feature vector. All feature vectors are normalized using LayerNorm before output to ensure numerical stability.

[0114] The cross-modal initial alignment and normalization unit is responsible for mapping heterogeneous features from different modalities to a unified semantic space. The dimensionality mapping processor adopts a two-stage mapping strategy. In the first stage, a linear projection layer is used to uniformly map the features of each modality to a 2048-dimensional intermediate representation space. All weight matrices are initialized using Xavier and trained end-to-end on 120,000 medical multimodal sample pairs. In the second stage, a three-layer MLP neural network is used for nonlinear mapping refinement. Each layer contains 2048 neurons with the GELU activation function and a dropout rate of 0.1. The semantic alignment processor adopts a contrastive learning-based alignment method instead of the covariance similarity method in the principle as a variant implementation. An alignment training set containing 80,000 pairs of medical image-medical record report pairs and 35,000 pairs of test data-diagnostic conclusion pairs is constructed. The improved InfoNCE contrastive loss function is used for training with a batch size of 128. Training uses the AdamW optimizer with an initial learning rate of 3×10⁻. 4After 50 epochs of training using cosine annealing scheduling, the alignment accuracy on the validation set reached 91.3%. The feature normalization processor performed two-step normalization on the aligned features. The first step was L2 norm normalization, which standardized the magnitude of the feature vector to 1. The second step was Z-score normalization on each dimension of the feature vector, which made the mean of the feature distribution 0 and the standard deviation 1. In t-SNE visualization, the average Euclidean distance between CT image features and text features of the same disease category in the dimensionality-reduced two-dimensional space was 0.23, while the average distance between different disease categories was 1.47. The inter-class and intra-class distance ratio reached 6.4 times, which verified the effectiveness of the alignment effect.

[0115] The domain knowledge fusion and semantic enhancement module constructs and maintains a multi-layered knowledge graph in the medical field. The knowledge graph construction processor acquires knowledge from three main sources: First, it extracts abstracts from 1.2 million English medical papers from the PubMed medical literature database, using the ScispaCy toolkit for medical named entity recognition, identifying 2.1 million entity mentions across 12 medical entity categories. Then, it uses a BERT-based relation classification model to identify 5.8 million relation triples across 18 relation types. Second, it extracts 38,000 structured rules from 1,200 clinical practice guidelines and expert consensus documents through a combination of manual annotation and semi-automatic extraction. Third, it mines disease-symptom-test indicator association patterns from historical data in hospital information systems across 500,000 medical records. 120,000 high-confidence co-occurrence patterns were discovered. After deduplication and conflict resolution, all knowledge was transformed into a medical knowledge graph containing 58,000 entity nodes and 850,000 relation edges. A hierarchical knowledge organization processor divided the knowledge graph into three layers based on abstraction: a general knowledge layer containing 12,000 nodes of basic knowledge across disease domains, such as human anatomy and basic physiological parameters; a domain knowledge layer divided into 15 subgraphs according to clinical departments, each containing professional disease knowledge for that department, totaling 38,000 nodes; and a task knowledge layer containing 8,000 process nodes of process rules and decision trees for specific diagnostic tasks. A knowledge index generation processor constructed four types of index structures for the knowledge graph, with an inverted index based on entity names using Elasticsearch. Version 8.5 has an index size of 8.2GB and an average query response time of 12 milliseconds. The FAISS vector index based on knowledge embedding vectors uses the HNSW algorithm to build all entities and uses the TransE model to train and generate 256-dimensional embedding vectors. When recalling the top 100 similar entities, the query time is 8 milliseconds and the recall accuracy is 87.4%. The relation index based on relation type uses an adjacency list structure for storage and supports one-hop relation queries with O(1) time complexity. The pre-computation index based on subgraphs pre-computes and caches disease subgraphs with high frequency queries. The cache hit rate reaches 43%, which can reduce the average query time from 45 milliseconds to 6 milliseconds. The knowledge graph supports an incremental update mechanism, adding 200 to 300 knowledge triples every week.

[0116] The problem-aware knowledge retrieval unit intelligently retrieves the most relevant knowledge subgraphs based on the clinical questions input by doctors. The problem intent parsing processor performs deep analysis of the problem text. First, it uses the PKUSEG Chinese word segmentation tool to segment the problem. Then, it uses a medical named entity recognition model to identify medical entities such as diseases, symptoms, and examination items in the problem. This recognition model, based on the BERT-CRF architecture, was trained on 50,000 self-built labeled data sets, achieving an F1 score of 94.2% for entity recognition. Next, a RoBERTa-based problem classification model was used to classify the problem into one of eight categories with a classification accuracy of 89.7%. Then, based on... Based on the question category and identified entities, a structured question representation is generated, including fields such as core disease entities, a list of accompanying symptom entities, a list of completed examination items, and the expected answer type. The knowledge requirement planning processor locates seed entity nodes in the knowledge graph based on the question parsing results and then plans the knowledge retrieval scope based on medical reasoning rules. The planned knowledge requirement graph contains three levels: Level 1 consists of 18 direct relationship nodes for the seed entities; Level 2 consists of 67 indirect relationship nodes reachable in two hops; and Level 3 consists of 12 diagnostic and treatment rule nodes, totaling 97 target knowledge nodes. The multi-hop knowledge retrieval processor starts from the seed nodes and executes... Multi-hop retrieval employs a breadth-first search strategy, expanding layer by layer. Relevance calculation uses a modified scoring method instead of the retention probability calculation in the original principle. Specifically, the relevance score equals a weighted sum of three factors: the first is the cosine similarity between the node's semantic embedding and the question embedding, multiplied by a weight of 0.5; the second is the inverse of the shortest path length from the node to the selected knowledge set, multiplied by a weight of 0.3; and the third is the node's PageRank importance score, multiplied by a weight of 0.2. For each hop, the top 30 nodes with the highest relevance scores are added to the candidate set. After three hops of expansion, the candidate set contains 94 nodes. Then, pruning is performed according to a dynamic threshold mechanism. The threshold for the first hop is 0.35, the threshold for the second hop is 0.42, and the threshold for the third hop is 0.50. Nodes with relevance scores lower than the corresponding thresholds are pruned. The final knowledge subgraph contains 62 nodes and 178 edges. Compared with the planned 97 target nodes, the recall rate is 64%, and redundant nodes account for only 8%. The statistical results of knowledge recall show that the average number of knowledge nodes recalled per question is 58.3, which is 51.4% lower than the traditional fixed K-value recall method that recalls an average of 120 nodes. The accuracy of answer generation increases from 83.6% to 89.7%, an increase of 6.1 percentage points, which fully verifies the effectiveness of the dynamic pruning mechanism.

[0117] The multimodal feature and knowledge deep fusion unit achieves deep fusion of multimodal feature representation and recalled knowledge. The semantic anchor discovery processor automatically establishes the mapping relationship between multimodal features and knowledge entities. For CT image features, the Faster-RCNN model is used to locate key anatomical structures and lesion regions in the images, and can detect 28 organs and 15 abnormal manifestations with a detection accuracy of 0.834 mAP. For text features, the named entity recognition model is used to extract entity references, and then the text references are linked to standard entities in the knowledge graph through an entity linking algorithm. The entity linking adopts a BERT-based disambiguation model with a linking accuracy of 92.1%. The tabular data is directly linked through a predefined standard code mapping table for test items, achieving a mapping coverage of 96.3%. After semantic anchor point discovery, an average of 23 anchor point mappings are established per CT image, 17 anchor point mappings per medical record text, and 8 anchor point mappings per test report. The cross-attention fusion processor uses a modified fusion architecture to replace the Tucker tensor decomposition method in the principle, specifically implementing a bidirectional Transformer attention mechanism. First, the multimodal feature vector sequence and the knowledge entity embedding vector sequence are concatenated to form a mixed sequence of length N+K, where N is the number of multimodal features (average 48 feature vectors) and K is the recall. The average number of knowledge nodes is 58 entity embedding vectors. The concatenated hybrid sequence is input into a 6-layer Transformer encoder for deep interaction. Each layer contains 8 attention heads, each with a dimension of 256. The feedforward network has a dimension of 2048, with a total of 120 million parameters. After processing by the 6-layer Transformer, the representation of each feature vector incorporates the semantic information of relevant knowledge, and the representation of each knowledge embedding vector also incorporates multimodal information supporting evidence, achieving bidirectional enhancement of features and knowledge. The unified representation generation processor aggregates the fused feature vector sequence to generate a unified semantic representation. The aggregation strategy adopts a hierarchical pooling method. The proposed method performs max pooling on image feature vectors, average pooling on text feature vectors, and weighted average pooling on table feature vectors. Then, it concatenates the three global representation vectors with the weighted average vector of the knowledge entity embedding vector. Finally, it maps the concatenated vector to a 2048-dimensional unified semantic representation through a two-layer fully connected network. On the validation set, the accuracy of diagnostic reasoning using the knowledge-enhanced unified semantic representation is 89.7%, an improvement of 13.3 percentage points compared to the baseline method using only multimodal features without knowledge integration (76.4%), and an improvement of 20.8 percentage points compared to the method using only knowledge without multimodal feature integration (68.9%).

[0118] The intelligent reasoning and answer generation module performs clinical reasoning and generates diagnostic suggestions based on a knowledge-enhanced unified semantic representation. The problem decomposition and reasoning path planning unit evaluates problem complexity using a random forest classifier trained on 5000 labeled samples with 15-dimensional features, categorizing complexity into simple, medium, and complex levels. For medium and complex problems, a dependency-based decomposition method is employed, using HanLP to analyze the core and modifying components of the problem, decomposing it into multiple sub-problems. The reasoning path planning processor generates a reasoning execution plan based on the dependencies between sub-problems, using a directed acyclic graph (DAG) to represent the reasoning path, stored in JSON format. The knowledge-enhanced semantic reasoning unit executes the actual reasoning process based on a GPT-4-Turbo large model. The reasoning execution processor constructs the large model's prompts (Prompts) using a structured format, with the total length of the prompts controlled within 8000 tokens. The large model's generation parameters are set to temperature 0.3 and top_p 0.9. The `max_tokens` parameter is 2048. Statistical analysis reveals that the reasoning process references an average of 3.2 times of image evidence, 4.7 times of textual evidence, 2.1 times of tabular evidence, and 5.8 times of knowledge base evidence. The referenced knowledge covers 76% of the key nodes in the recalled knowledge subgraph. The reasoning chain generation processor parses the output of the large model, extracting the reasoning steps and intermediate conclusions. The average length of the reasoning chain is 4.8 steps, with a standard deviation of 1.6 steps. The multi-strategy answer generation and integration unit employs four strategies to generate candidate answers: direct generation based on the reasoning chain, generation based on knowledge templates, generation based on large model continuation, and generation based on multi-expert integration. The answer evaluation and ranking processor performs multi-dimensional evaluation of candidate answers, including five dimensions: confidence, completeness, relevance, fluency, and professionalism. Confidence evaluation uses a modified calculation method instead of the structured entropy method in the principle; specifically, it uses a calibrated logistic regression model to predict the probability of the correct answer within 10,000 labeled data. The training input features on the sample include 12 dimensions such as inference chain length, number of cited evidence, and knowledge coverage. The error between the calibrated scores and the actual accuracy rate is within 5%. The evaluation scores of each dimension are normalized to between 0 and 1, and then weighted and summed according to the following weights: confidence 40%, completeness 25%, relevance 15%, fluency 10%, and professionalism 10% to obtain the comprehensive score. Statistics show that the probability of the template-generated answer being selected as the best is 45%. The answer integration and output processor integrates the best answer into the corresponding output format according to user preference settings and adds meta-information annotations at the end of the answer, including inference chain visualization, knowledge source list, evidence citation index, and comprehensive confidence score. Answers with a confidence score below 0.82 are specially marked to remind users to treat them with caution. The average time for answer generation is 1.8 seconds, of which knowledge recall takes 0.4 seconds, feature fusion takes 0.3 seconds, large model inference takes 0.9 seconds, and answer evaluation takes 0.2 seconds. Compared with the traditional method of 4.2 seconds, it is shortened by 57%.

[0119] The quality monitoring and adaptive optimization module performs comprehensive quality checks and verification on the system's output. The multimodal consistency detection and conflict resolution unit checks the semantic consistency between different modalities of data. The cross-modal conflict detection processor uses a rule-based and model-based detection method. The rule base contains 128 medical common sense rules, covering 73% of common conflict situations. For complex conflicts, a BERT-based text entailment model is used for detection. After training on 8000 pairs of labeled samples, the model's accuracy rate is 88.6%. The detection results show that 189 cases of multimodal conflicts were found in 15200 diagnostic requests, accounting for 1.24%, including 106 cases of image-text inconsistency, 58 cases of examination-text inconsistency, and 25 cases of image-examination inconsistency. The confidence score calculation processor calculates a confidence score for each modality based on four factors: the authority of the data source, the timeliness of the data, the completeness of the information, and historical accuracy. The weights for these factors are 0.3, 0.2, 0.2, and 0.3, respectively. Statistics show that the average confidence score for CT image data is 0.92, the average confidence score for medical record text data is 0.87, the average confidence score for laboratory report data is 0.95, and the average confidence score for patient self-report data is 0.58. The conflict resolution strategy processor executes a conflict resolution strategy based on the confidence score. If the confidence score difference is greater than 0.2, the information from the high-confidence modality is accepted (75%). If the confidence score difference is between 0.05 and 0.2, weighted fusion is performed (16%). If the confidence score difference is less than 0.05... Five cases triggered manual review, accounting for 9%. The average response time for manual review was 2.5 hours, and the accuracy rate after review was 100%. The reasoning result verification and confidence assessment unit rigorously verified and evaluated the diagnostic suggestions output by the reasoning. The logical consistency verification processor checked the logical rigor of the reasoning chain, using a logic rule verification method based on medical ontology. The medical ontology contains 3200 logical constraints, including necessary and sufficient conditions for disease diagnosis. The verification results showed that logical problems were detected in 342 out of 15200 diagnostic requests, accounting for 2.25%, including 18 cases of circular reasoning, 67 cases of contradictory conclusions, and 257 cases of logical jumps. The evidence sufficiency check processor evaluated whether the evidence support for the reasoning conclusion was sufficient. Statistical analysis showed that on average, each reasoning... The process references 2.8 pieces of evidence, with 73% of the steps meeting the sufficiency of evidence requirements. The comprehensive confidence assessment processor uses a modified assessment method to replace the structured entropy method in the principle. Specifically, it is implemented as a multi-factor linear regression model. The model input features include 15-dimensional features such as logical consistency score and sufficiency of evidence score. The model is trained on 12,000 labeled samples and outputs a comprehensive confidence score between 0 and 1. The mean squared error of the model is 0.012. For results with a confidence score below the threshold of 0.82, the re-inference process is automatically initiated. On the test set, the proportion of results with a confidence score below the threshold is 8.3%. After re-inference, the confidence score increases to above the threshold in 42% of cases. The proportion requiring human intervention is 4.8%. The diagnostic accuracy after human intervention reaches 96%.7%, the user feedback-driven system adaptive optimization unit is responsible for collecting user feedback and optimizing system parameters. The feedback information acquisition processor collects user feedback through multiple channels, including explicit and implicit feedback. During the three-month deployment period, it collected feedback information from 15,200 diagnostic requests. Among them, 12,800 (84.2%) provided satisfaction ratings, 8,600 (56.6%) indicated diagnostic correctness, and 2,100 (13.8%) provided text feedback. The parameter update strategy processor optimizes the system's decision-making strategy parameters based on the feedback data using reinforcement learning methods. It models the reasoning process as a Markov Decision Process (MDP) and uses the Proximal Policy Optimization (PPO) algorithm for policy learning. After training on 8,600 labeled samples for 1,000 epochs, the average reward of the knowledge recall strategy increased from the initial 0.32 to 0.67, the average reward of the reasoning path selection strategy increased from 0.28 to 0.58, and the average reward of the answer generation strategy increased from 0.35 to 0.72. At the same time, it updates the entity weights of the knowledge graph and During the optimization process, the weights of 320 knowledge nodes were increased by an average of 0.18. 127 knowledge nodes were marked as suspicious and awaiting review. After manual review, it was found that 83 of these nodes (65.4%) did indeed have outdated or erroneous knowledge. These nodes were updated or deleted. Performance monitoring and evaluation continuously monitored the system's key performance indicators (KPIs) and set performance thresholds. Alarms were triggered when the KPIs fell below the thresholds. After parameter updates, A / B testing was conducted to verify the optimization effect. The results showed that the diagnostic accuracy rate of the experimental group was 91.2%, an improvement of 1.5 percentage points compared to the control group's 89.7%. The user satisfaction score of the experimental group was 4.6, an improvement of 0.1 points compared to the control group's 4.5. A paired t-test confirmed the statistical significance of the difference (p-value less than 0.01). Through continuous feedback-driven optimization, the system's diagnostic accuracy rate gradually increased from 85.4% at the initial deployment stage to 91.2% after three months, an increase of 5.8 percentage points. User satisfaction improved from 4.1 to 4.6, an increase of 0.5 points.

[0120] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of protection of the present invention. Furthermore, the elements therein can be updated as technology develops.

Claims

1. A domain knowledge fusion and semantic question answering system based on text and multimodal large models, characterized in that, It includes a multimodal data intelligent preprocessing and feature extraction module, a domain knowledge fusion and semantic enhancement module, an intelligent reasoning and answer generation module, and a quality monitoring and adaptive optimization module; The multimodal data intelligent preprocessing and feature extraction module is responsible for unified access, deep feature extraction and preliminary alignment of heterogeneous data sources; the domain knowledge fusion and semantic enhancement module is responsible for building a domain knowledge base and realizing deep fusion of knowledge and multimodal features; the intelligent reasoning and answer generation module is responsible for performing problem decomposition, semantic reasoning and answer generation; and the quality monitoring and adaptive optimization module is responsible for consistency detection, confidence assessment and system adaptive optimization. The multimodal data intelligent preprocessing and feature extraction module includes a heterogeneous data source access and identification unit, a submodal deep feature extraction unit, and a cross-modal preliminary alignment and normalization unit. The heterogeneous data source access and identification unit is used to automatically receive and identify the modality type, format specification, and quality level of the data. The submodal deep feature extraction unit is used to extract feature vector sequences specific to each modality. The cross-modal preliminary alignment and normalization unit is used to map the features of different modalities to a unified dimension representation space. The domain knowledge fusion and semantic enhancement module includes a domain knowledge base construction and indexing unit, a problem-aware knowledge precise retrieval unit, and a multimodal feature and knowledge deep fusion unit. The domain knowledge base construction and indexing unit is used to establish an efficient knowledge indexing mechanism that supports semantic retrieval and relational reasoning. The problem-aware knowledge precise retrieval unit is used to parse the problem intent and generate a knowledge demand graph. The multimodal feature and knowledge deep fusion unit is used to effectively inject domain knowledge into the multimodal feature representation. The intelligent reasoning and answer generation module includes a problem decomposition and reasoning path planning unit, a knowledge-enhanced semantic reasoning unit, and a multi-strategy answer generation and integration unit. The problem decomposition and reasoning path planning unit is used to output a structured reasoning plan to guide the subsequent reasoning process. The knowledge-enhanced semantic reasoning unit performs logical reasoning based on the fused semantic representation to generate intermediate conclusions with complete reasoning chains. The multi-strategy answer generation and integration unit generates multiple candidate answers from different reasoning paths and generates the final answer through an intelligent integration strategy. The quality monitoring and adaptive optimization module includes a multimodal consistency detection and conflict resolution unit, an inference result verification and confidence assessment unit, and a user feedback-driven system adaptive optimization unit. The multimodal consistency detection and conflict resolution unit automatically detects semantic conflicts between data of different modalities. The inference result verification and confidence assessment unit is used to verify the inference results and calculate the comprehensive confidence score. The user feedback-driven system adaptive optimization unit is used to dynamically update the knowledge retrieval strategy, inference path selection, and generation control parameters.

2. The domain knowledge fusion and semantic question answering system based on text and multimodal large model as described in claim 1, characterized in that, The cross-modal preliminary alignment and normalization unit includes a dimension mapping processor, a semantic alignment processor, and a feature normalization processor. The dimension mapping processor maps feature vectors of different modalities to a unified representation space of the same dimension. The semantic alignment processor is used to learn the semantic correspondence between different modalities to achieve coarse-grained semantic alignment. The feature normalization processor performs numerical range normalization and distribution standardization on the aligned feature vectors. The semantic alignment processor calculates the hierarchical weighted semantic distance D(f1, f2) between two different modal features according to the following formula: ； Where f1 and f2 represent feature vectors of two different modalities, and L is the total level of the feature vectors. This represents the i-th layer feature in the feature vector f1. w represents the i-th layer feature in feature vector f2. i Let be the alignment weight of the i-th layer. To smooth out the penalty coefficient.

3. The domain knowledge fusion and semantic question answering system based on text and multimodal large model as described in claim 2, characterized in that, The problem-aware knowledge retrieval unit includes a problem intent parsing processor, a knowledge demand planning processor, and a multi-hop knowledge retrieval processor. The problem intent parsing processor performs deep semantic analysis on the user-input question to generate a structured question representation to guide the knowledge retrieval process. The knowledge demand planning processor plans the scope of knowledge to be recalled and the depth of reasoning based on the problem intent analysis results. The multi-hop knowledge retrieval processor starts from the seed nodes in the knowledge demand graph, performs multi-hop relationship reasoning and path search, recalls the most relevant knowledge subgraph, and filters out irrelevant knowledge. For node k in the knowledge graph i The multi-hop knowledge retrieval processor calculates the retention probability of a node in the t-th inference stage according to the following formula: ； Where q represents the user question, H t This represents the entire reasoning history from the beginning to the current stage. It is the Sigmoid activation function. For learnable weight vectors, N(k) represents the five-dimensional feature vector of a node. i ) represents node k i The set of neighboring nodes, The influence coefficient of the neighbor For the previous stage node k j The retention probability.

4. The domain knowledge fusion and semantic question answering system based on text and multimodal large model as described in claim 3, characterized in that, The multimodal feature and knowledge deep fusion unit includes a semantic anchor discovery processor, a cross-attention fusion processor, and a unified representation generation processor. The semantic anchor discovery processor automatically identifies the semantic correspondence between multimodal features and recalled knowledge entities, and establishes a cross-modal semantic anchor mapping by calculating the similarity between feature vectors and knowledge embedding vectors. The cross-attention fusion processor realizes bidirectional enhancement of multimodal features and knowledge representation, achieving deep interaction and mutual enhancement of features and knowledge at the semantic level. The unified representation generation processor integrates the fused multimodal features and knowledge information into a unified semantic representation vector. The cross-attention fusion processor first constructs an interaction tensor T. Where M is the number of modalities, N is the feature sequence length, K is the number of knowledge nodes recalled, and the tensor elements are defined according to the following formula: ； in, Let k represent the nth eigenvector of mode m. k W is the embedding vector of knowledge node k. fus The fusion weight matrix is ​​learnable; The cross-attention fusion processor performs Tucker decomposition on the interaction tensor T according to the following formula: ； in, For the core tensor, , , It consists of three factor matrices, with R1, R2, and R3 being the compressed dimensions. Representing tensor-matrix multiplication along the i-th dimension, Tucker decomposition decomposes the high-dimensional interaction tensor into a low-rank core structure and three pattern matrices, capturing "modal patterns", "feature patterns" and "knowledge patterns" respectively. Based on the decomposition results, the unified representation generation processor calculates the knowledge-enhanced unified semantic representation according to the following formula: ； Where K = [k1, ..., k K ] represents the knowledge matrix.

5. The domain knowledge fusion and semantic question answering system based on text and multimodal large models as described in claim 4, characterized in that, The reasoning result verification and confidence assessment unit includes a logical consistency verification processor, an evidence sufficiency check processor, and a comprehensive confidence assessment processor. The logical consistency verification processor checks whether the reasoning result conforms to formal logic rules and domain knowledge constraints. The evidence sufficiency check processor assesses whether the evidence supporting the reasoning conclusion is sufficient. The comprehensive confidence assessment processor is used to calculate the comprehensive confidence of the reasoning result and quantify the sources of uncertainty. For the inference chain C = {s1, s2, ..., s...} A The comprehensive confidence assessment processor constructs a probability distribution of the result for each inference step according to the following formula: ； Where A is the length of the inference chain, and y is the candidate result. e is the summation variable that iterates through all candidate results y h is the embedding vector of the candidate results. i For the fusion representation of the i-th step of reasoning, For smoothing parameters, Let represent the probability of obtaining the result y in the i-th step, given the known reasoning history of the previous i-1 steps. Let be the similarity between the candidate result y and the mode m in step i. Let m be the weight exponent of mode m in step i; The logical consistency verification processor calculates the consistency H(C) of the inference process according to the following formula: ； in, Let D represent the Shannon entropy at step i. KL () represents the KL divergence between adjacent reasoning steps. This is the consistency weighting coefficient; The comprehensive confidence assessment processor models the probability distribution described above and calculates the comprehensive confidence level C according to the following formula. final (C): ； in, Let represent the optimal candidate result for step i.

Images