An interdisciplinary intelligent answering method and system

Abstract

The application discloses an interdisciplinary intelligent answering method and system, relates to the fields of artificial intelligence and intelligent answering technology, and is used for solving the technical problems that the existing intelligent answering system is difficult to balance the processing capability of complex interdisciplinary problems and the credibility and explainability of generated answers, and is prone to fact hallucination. The application receives a multimodal query input by a user, completes multimodal feature extraction and fusion to generate a unified query semantic vector, searches for related knowledge fragments and interdisciplinary reasoning paths in a pre-constructed multimodal interdisciplinary knowledge graph, assembles the enhanced prompt input large language model to generate a preliminary answer, performs fact triple extraction and matching verification on the preliminary answer, completes answer quality evaluation and iterative optimization, and finally outputs a final answer with structured reference annotations. The application constructs a closed-loop architecture of generation and verification collaboration, and realizes high-trust and explainable interdisciplinary intelligent answering.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent question answering technology, specifically to an interdisciplinary intelligent question answering method and system. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent question-answering systems have been widely used in education, scientific research, and corporate training. Currently, mainstream question-answering technologies are mainly divided into two categories: knowledge graph-based question-answering systems and generative question-answering systems based on large language models. Knowledge graph-based question-answering systems achieve accurate factual answers through structured queries, but their ability to understand natural language is limited, and they struggle to handle complex questions requiring interdisciplinary knowledge integration. While generative question-answering systems based on large language models excel in semantic understanding and content generation, the content they generate may contain factual errors, especially in interdisciplinary professional fields where this "illusion" problem is more pronounced. How to effectively combine the semantic understanding capabilities of large language models with the structured knowledge of knowledge graphs, while ensuring the verifiability of the generated answers, has become a pressing technical challenge for those skilled in the art.

[0003] In the prior art, invention patent CN117609589A discloses a retrieval method and system based on GPT large model generation and knowledge fusion. This method constructs a knowledge base and combines it with the GPT large model for retrieval, fusing the knowledge in the knowledge base with the answers generated by GPT to improve the accuracy and credibility of the retrieval results. This scheme constrains the generation results of the large model through the knowledge base, alleviating the illusion problem of pure generative models to some extent. However, the knowledge fusion in this scheme is mainly reflected in the retrieval stage, that is, using the knowledge base to assist in retrieval to obtain a more accurate knowledge set, without performing structured fact verification on the final answers generated by the GPT model itself. When the model deviates from the retrieved knowledge during the generation process, this scheme lacks an effective error correction mechanism. In addition, some prior art has proposed a multimodal knowledge graph enhanced generation framework. This framework integrates multidisciplinary knowledge by constructing a multimodal knowledge graph in the field of education and retrieving relevant knowledge fragments as context before generating answers. This scheme combines knowledge graphs and large models at the prompting engineering level, but its verification mechanism only stays at the level of shallow checks on semantic overlap and entity coverage, and cannot perform accurate structured comparison and quantitative scoring of every fact triple in the generated answer.

[0004] The common drawback of the aforementioned existing technologies is that knowledge graphs are only used as an auxiliary tool in the retrieval stage, failing to provide fine-grained structured verification and feedback optimization for the answers generated by large language models. When the generated answers contain factual content that contradicts the knowledge graph, existing solutions struggle to accurately identify and correct them, resulting in the credibility and interpretability of the answers still not being effectively guaranteed. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a cross-disciplinary intelligent question-answering method and system. By constructing a closed-loop architecture in which generation and verification are mutually coordinated, the knowledge graph is transformed from a one-way retrieval source into a two-way collaborative verification tool. Based on fine-grained extraction of fact triples from the generated answers, the scoring function of the knowledge graph embedding model is used to quantitatively verify each fact, and the generated results are iteratively optimized based on the verification feedback, thereby achieving highly reliable and interpretable cross-disciplinary intelligent question-answering.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On one hand, a cross-disciplinary intelligent question-answering method, which includes the following steps: Step 1: Receive a multimodal query input by the user, wherein the multimodal query includes at least one of text data, image data, or formula data; Step 2: Extract and fuse multimodal features from the multimodal query to generate a unified query semantic vector; Step 3: Based on the unified query semantic vector, search in the pre-constructed multimodal interdisciplinary knowledge graph to obtain knowledge fragments and interdisciplinary reasoning paths related to the multimodal query. The multimodal interdisciplinary knowledge graph covers at least two disciplines in mathematics, physics, chemistry, biology, and computer science, and the concepts of different disciplines are associated through interdisciplinary bridging relationships. Step four: Assemble the knowledge fragment, the interdisciplinary reasoning path, and the multimodal query into an enhanced hint, and input the enhanced hint into the large language model to generate a preliminary answer; Step 5: Extract fact triples from the preliminary answer to obtain at least one fact triple. Each fact triple includes a subject entity, a predicate relation, and an object entity. Step 6: Match and verify each fact triple with the multimodal interdisciplinary knowledge graph, and calculate the fact confidence of each fact triple based on the scoring function of the knowledge graph embedding model; Step 7: Evaluate the quality of the preliminary answer based on the factual confidence level. If the evaluation result is lower than the preset quality threshold, adjust the retrieval strategy based on the verification feedback, and re-retrieve and iteratively generate an optimized answer based on the adjusted retrieval strategy until the evaluation result of the optimized answer meets the preset quality threshold. Step 8: Output the final answer and generate structured reference annotations for the knowledge points in the final answer; By constructing a closed-loop architecture where generation and verification work in tandem, highly reliable and explainable interdisciplinary intelligent Q&A can be achieved.

[0007] Furthermore, step two specifically includes: The text data is encoded using a Transformer-based text encoder to generate text feature vectors; The image data is encoded using a vision-language pre-trained model to generate image feature vectors; The formula data is converted into a LaTeX representation using a formula recognition model, and the LaTeX representation is encoded to generate a formula feature vector; A multimodal fusion strategy based on an attention mechanism is adopted to calculate the attention weights corresponding to the text feature vector, the image feature vector, and the formula feature vector. The text feature vector, the image feature vector, and the formula feature vector are then weighted and fused according to the attention weights to generate the unified query semantic vector. The formula for calculating the attention weights is as follows: in, Represents a set of modes. Including text modality, img modality, and formula modality. Indicates the current calculated mode. Representing modes Attention weights Representing modes The attention parameter vector to be trained, express transpose, Representing modes eigenvectors, Represents an exponential function. This represents the summation operation. Traversal All modes; By calculating the attention weights of each modality and performing weighted fusion using a multimodal fusion strategy based on an attention mechanism, text, image, and formula features can be uniformly represented as query semantic vectors, enabling accurate understanding of users' complex query intentions.

[0008] Furthermore, the construction of the multimodal interdisciplinary knowledge graph in step three specifically includes: For each discipline in mathematics, physics, chemistry, biology, and computer science, define the conceptual system and relationship types of each discipline to form a discipline ontology definition; Entities, relationships, and attributes are extracted from multi-source data, including textbook data, academic paper data, and encyclopedia data. Entity alignment is performed on the same entities from different data sources. The entity alignment adopts a similarity calculation method based on entity embedding. Establishing interdisciplinary bridging relationships between concepts from different disciplines, wherein the interdisciplinary bridging relationships include at least one of those applied to relations, similar to relations, or based on relations in theory; By establishing interdisciplinary bridging relationships between concepts from different disciplines and aligning entities, knowledge from multiple disciplines can be integrated into a structured knowledge network, providing a knowledge foundation for reasoning on interdisciplinary problems.

[0009] Furthermore, in step three, acquiring knowledge fragments and interdisciplinary reasoning paths related to the multimodal query specifically includes: A multi-path recall strategy is adopted, simultaneously performing vector similarity retrieval, keyword matching retrieval, and knowledge graph structured path retrieval; The vector similarity retrieval is based on HNSW dense vector index to perform approximate nearest neighbor search and obtain knowledge fragments that are semantically similar to the unified query semantic vector. The keyword matching retrieval is based on BM25 scoring using a sparse inverted index to obtain knowledge fragments containing the query keywords; The knowledge graph structured path retrieval is based on graph databases to perform entity matching and relation path discovery, and to obtain cross-disciplinary reasoning paths that connect concepts from different disciplines; By simultaneously executing a multi-path recall strategy that combines vector similarity retrieval, keyword matching retrieval, and knowledge graph structured path retrieval, it is possible to comprehensively acquire knowledge fragments and interdisciplinary reasoning paths related to the query from different dimensions.

[0010] Furthermore, in step four, the knowledge fragment, the interdisciplinary reasoning path, and the multimodal query are assembled into an enhanced hint, specifically including: The knowledge fragments are reordered, and the reordering score is calculated using the following formula: in, Indicates the first A fragment of knowledge, Representing knowledge fragments Reordered scores Representing knowledge fragments Vector retrieval relevance score, Representing knowledge fragments With multimodal queries semantic relevance score, Representing knowledge fragments The importance weight in interdisciplinary reasoning paths , , Preset weighting coefficients and satisfying ; Based on the context length limitation of the large language model, a greedy selection algorithm is used to select the optimal subset from the reordered knowledge fragments to construct the context window; The enhanced prompt is generated by concatenating the context window with the formatted representation of the interdisciplinary reasoning path. By sorting knowledge fragments based on reordering scores and constructing context windows using a greedy selection algorithm, the most relevant knowledge information can be retained within a limited context length, providing high-quality input for answer generation.

[0011] Furthermore, in step six, the fact confidence score of each fact triple is calculated based on the scoring function of the knowledge graph embedding model, specifically including: For each fact triple The TransE model is used as the knowledge graph embedding model to calculate the score. The calculation formula is as follows: in, Indicates the subject entity, Indicates predicate relationship, Indicates the object entity, This indicates the rating of the fact triple. Indicates the subject entity Embedded vector, Indicating predicate relations Embedded vector, Indicates the object entity Embedded vector, Represents L2 norm operations; The rating is converted into fact confidence score using the sigmoid function, calculated as follows: in, Representing fact triples Confidence level of facts This is the scaling factor. For threshold parameters, It is a natural constant; By employing the TransE model to calculate the embedding vector distance of fact triples and converting it into fact confidence using the sigmoid function, precise structured verification of each fact in the generated answer can be achieved.

[0012] Furthermore, in step seven, the preliminary answer is assessed for quality based on the stated confidence level, specifically including: To calculate sentence-level confidence, for a sentence containing multiple fact triples, the sentence-level confidence is the weighted average of the fact confidence of all fact triples in the sentence, with the weights determined by the importance of each fact triple in the sentence; To calculate the answer-level confidence score, the preliminary answer-level confidence score is determined by the weighted average and minimum of the sentence-level confidence scores. The calculation formula is as follows: in, Indicates a preliminary answer Answer-level confidence level Indicates a preliminary answer The collection of sentences in the text Indicates the number of sentences. Sentence Sentence-level confidence, To balance the weighting coefficients of the average confidence level and the lowest confidence level, This indicates the minimum value operation; The answer-level confidence score is compared with a preset confidence threshold. If the answer-level confidence score is lower than the preset confidence threshold, the evaluation result is determined to be lower than the preset quality threshold. By using the weighted average and minimum values ​​of sentence-level confidence to determine answer-level confidence, the quality of generated answers can be comprehensively evaluated from both local and global perspectives.

[0013] Furthermore, in step seven, adjusting the retrieval strategy based on verification feedback specifically includes: Based on the fact triples whose fact confidence level is lower than the preset fact threshold in the verification feedback, the knowledge points to be optimized are identified. For the knowledge points that need optimization, the scope of the search subjects should be expanded, and the number of searches for related subjects should be increased; For the knowledge points that need optimization, the search depth of interdisciplinary reasoning paths is deepened and the number of reasoning steps is increased; The search operation is re-executed based on the adjusted search strategy to obtain supplementary knowledge fragments and supplementary reasoning paths; By identifying the knowledge points to be optimized based on low-confidence facts in the verification feedback and adjusting the subject scope and reasoning depth of the retrieval, missing evidence can be supplemented in a targeted manner, thereby achieving iterative optimization of the answer.

[0014] Furthermore, in step eight, structured reference annotations are generated for the knowledge points in the final answer, specifically including: Establish a tracing link from the knowledge unit in the final answer to the original knowledge fragment, and determine the source knowledge fragment, source location range and matching confidence of each knowledge unit; For each reference, a structured field is generated, which includes a unique reference identifier, source name, source type, reference credibility score, original resource access link, and reference content summary; In the final answer, a corresponding reference tag is inserted for each knowledge unit, and the reference tag is associated with the structured field; Generate and visualize the complete reasoning path from query to final answer. The reasoning path includes query nodes, intermediate concept nodes, answer nodes, and relational edges connecting each node. By establishing a traceability link from knowledge units to original fragments and generating structured fields for each reference, users can be provided with a complete traceability of the source of the answer and a visualized reasoning path.

[0015] On the other hand, an interdisciplinary intelligent Q&A system, applicable to an interdisciplinary intelligent Q&A method, comprises the following components: The multimodal query understanding module is used to receive multimodal queries input by users, wherein the multimodal queries include at least one of text data, image data or formula data, and to extract and fuse multimodal features of the multimodal queries to generate a unified query semantic vector; The interdisciplinary knowledge graph construction and management module is used to construct, store, and dynamically update multimodal interdisciplinary knowledge graphs. The multimodal interdisciplinary knowledge graphs cover at least two disciplines among mathematics, physics, chemistry, biology, and computer science, and the concepts of different disciplines are associated through interdisciplinary bridging relationships. The hybrid retrieval and reasoning module is used to retrieve knowledge fragments and interdisciplinary reasoning paths related to the multimodal interdisciplinary knowledge graph based on the unified query semantic vector. The answer generation module is used to assemble the knowledge fragment, the interdisciplinary reasoning path, and the multimodal query into an enhanced hint, input the enhanced hint into a large language model, and generate a preliminary answer; The fact verification module is used to extract fact triples from the preliminary answer, obtain at least one fact triple, and match and verify each fact triple with the multimodal interdisciplinary knowledge graph. The fact confidence of each fact triple is calculated based on the scoring function of the knowledge graph embedding model. The answer optimization control module is used to evaluate the quality of the preliminary answer based on the factual confidence level. When the evaluation result is lower than the preset quality threshold, the retrieval strategy is adjusted according to the verification feedback, and the hybrid retrieval and reasoning module and the answer generation module are controlled to perform iterative optimization until the evaluation result of the generated optimized answer meets the preset quality threshold. An interpretable output module is used to output the final answer and generate structured reference annotations for the knowledge points in the final answer; By setting up a fact verification module to perform structured verification of the generated answers and setting up an answer optimization control module to iteratively optimize based on verification feedback, the entire process from multimodal input to highly reliable answer output can be automated.

[0016] Compared with existing technologies, this interdisciplinary intelligent question-answering method and system has the following beneficial effects: I. This invention upgrades knowledge graphs from a one-way retrieval aid to a two-way collaborative verification and feedback optimization tool by constructing a closed-loop architecture that coordinates generation and verification. It extracts fine-grained fact triples from answers generated by large language models, and performs quantitative verification and confidence assessment on each fact based on a knowledge graph embedding model. This can accurately identify factual errors in the generated answers, dynamically adjust the retrieval strategy based on verification feedback, and iteratively optimize the generated results. This effectively solves the illusion problem of content generated by large language models in existing technologies, and significantly improves the credibility and accuracy of cross-disciplinary Q&A answers.

[0017] Second, this invention achieves unified semantic understanding of multimodal queries such as text, images, and formulas through multimodal feature extraction and adaptive fusion strategies based on attention mechanisms. At the same time, it establishes bridging relationships between concepts of different disciplines through a pre-constructed multimodal interdisciplinary knowledge graph, which can fully capture the user's complex query intent, explore the deep connections and reasoning paths between interdisciplinary knowledge, and generate structured reference annotations and visualized reasoning links for the final answer. This effectively adapts to the question-answering needs of complex interdisciplinary problems and improves the interpretability of the answer and the adaptability of the application in all scenarios.

[0018] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 This is a schematic diagram of the overall execution flow of the present invention; Figure 2 This is a schematic diagram of the modular architecture of the system of the present invention; Figure 3 This is a schematic diagram illustrating the structure and interdisciplinary bridging relationship of the multimodal interdisciplinary knowledge graph of the present invention. Detailed Implementation

[0021] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0022] Example like Figure 1 As shown, the interdisciplinary intelligent Q&A method disclosed in this embodiment is executed in its entirety according to the following steps to achieve fully automated processing from user multimodal query input to highly reliable and explainable answer output.

[0023] Step 1, Multimodal Query Reception: This step receives multimodal queries input by the user. Multimodal queries include at least one of the following: text data, image data, or formula data. Specifically, text data includes user-inputted natural language descriptions of the question, core keywords, supplementary explanations, etc., covering professional question expressions from various disciplines as well as everyday conversational question formats. Image data includes user-uploaded diagrams of subject principles, experimental setups, chemical molecular structures, screenshots of handwritten formulas, screenshots of computer code snippets, experimental data charts, and other visualizations related to the query content. Formula data includes user-inputted editable formula text, structured formula sequences, and other directly parsable formula-type data. This step supports both unimodal and multimodal mixed query input, adapting to user questioning habits in different scenarios such as education, scientific research, and corporate training.

[0024] Step 2, Multimodal Feature Extraction and Fusion: This step extracts and fuses multimodal features from the received multimodal queries to generate a unified query semantic vector.

[0025] Specifically, a text encoder based on the Transformer architecture is used to encode the text data and generate text feature vectors. The text encoder uses the RoBERTa-wwm-ext pre-trained model for the Chinese domain, which has been fine-tuned beforehand based on professional corpora from textbooks, core paper abstracts, and authoritative encyclopedia entries in five disciplines: mathematics, physics, chemistry, biology, and computer science. This ensures that the encoder can accurately recognize subject-specific terminology and complex semantic logic. The input text data undergoes preprocessing, including word segmentation, stop word removal, and terminology normalization. The preprocessed text is then input into the text encoder, ultimately outputting a 768-dimensional text feature vector.

[0026] A vision-language pre-trained model is used to encode image data and generate image feature vectors. The vision-language pre-trained model uses the CLIP pre-trained model, which is pre-tuned based on a subject-specific image-text pairing dataset. This dataset includes paired data of subject-specific principle diagrams, formula images, molecular structure images, and corresponding text descriptions, ensuring the model can accurately extract deep semantic features from subject-related images. For the input image data, preprocessing is performed, including size normalization to 224×224 pixels, pixel value normalization, and noise reduction and enhancement. The preprocessed image is then input into the fine-tuned CLIP model's visual encoder, ultimately outputting a 768-dimensional image feature vector.

[0027] A formula recognition model is used to convert formula data into LaTeX representation, and then the LaTeX representation is encoded to generate formula feature vectors. The formula recognition model uses a LaTeX-OCR model, which has been pre-tuned based on a mixed dataset of handwritten and printed formulas to adapt to formula recognition needs in different scenarios. The converted LaTeX formula sequence is then encoded using a Transformer-based formula encoder. This encoder is pre-trained based on a subject-specific formula corpus, accurately capturing the mathematical semantics and logical relationships of the formulas, and ultimately outputting a 768-dimensional formula feature vector.

[0028] A multimodal fusion strategy based on an attention mechanism is adopted. The attention weights corresponding to the text feature vector, image feature vector, and formula feature vector are calculated respectively. The three types of feature vectors are then weighted and fused according to the attention weights to generate a unified query semantic vector. The formula for calculating the attention weights is as follows: in, Represents a set of modes. Including text modality, img modality, and formula modality. Indicates the current calculated mode. Representing modes Attention weights Representing modes The attention parameter vector to be trained, express transpose, Representing modes eigenvectors, Represents an exponential function. This represents the summation operation. Traversal All modalities.

[0029] During model training, the attention parameter vectors to be trained are optimized end-to-end based on a cross-disciplinary question-answering dataset. The similarity between the query semantic vector and the corresponding standard answer semantic vector is used as the optimization objective to ensure that attention weights are adaptively allocated according to the modality proportion of different queries. When the user's input query is primarily a text question with only supplementary explanatory images, the model automatically assigns higher attention weights to the text modality; when the core query is a formula image, the model automatically assigns higher attention weights to the formula and image modalities, achieving adaptive fusion of different modal inputs. After calculating the attention weights, the feature vectors of the three modalities are weighted and summed according to their weights, ultimately generating a unified query semantic vector with a uniform dimension of 768.

[0030] Step 3, Knowledge Graph Retrieval and Reasoning Path Acquisition: This step, based on a unified query semantic vector, retrieves knowledge fragments and interdisciplinary reasoning paths related to the multimodal query from a pre-constructed multimodal interdisciplinary knowledge graph. The multimodal interdisciplinary knowledge graph covers five complete disciplines: mathematics, physics, chemistry, biology, and computer science, and concepts from different disciplines are linked through interdisciplinary bridging relationships.

[0031] like Figure 3 As shown, the multimodal interdisciplinary knowledge graph in this embodiment is pre-constructed according to the following process.

[0032] First, for each of the five disciplines, a conceptual system and relationship types are defined to form a discipline ontology definition. The discipline ontology definition references the national textbook writing standards and discipline classification standards for the corresponding discipline to ensure the professionalism and completeness of the ontology system. The core concepts of mathematics include functions, derivatives, integrals, differential equations, and linear algebra, with relationship types including defined as, properties as, derived from, and applied to. The core concepts of physics include mechanics, electromagnetism, thermodynamics, and quantum physics, with relationship types including principles as, formula correspondence, experimental verification, and applicable scenarios. The ontology definitions for the remaining disciplines are completed according to the corresponding discipline's knowledge system.

[0033] Entities, relationships, and attributes are extracted from multi-source data, including publicly available professional data such as national-level textbooks, core journal articles, authoritative subject encyclopedia entries, and subject standards and norms. For text data, an entity-relationship extraction model based on a pre-trained language model is used to extract entity, relationship, and attribute information. For image data, the corresponding text description, core concept annotations, and image feature vectors are extracted and stored as multimodal attributes in the knowledge graph. For formula data, the corresponding concepts, applicable scenarios, and derivation relationships are extracted and stored as formula entities in the knowledge graph. A confidence threshold is set during the extraction process, and only entity, relationship, and attribute data with a confidence level higher than the threshold are retained to ensure the basic data quality of the knowledge graph.

[0034] Entity alignment is performed on identical entities from different data sources using a similarity calculation method based on entity embedding. First, all extracted entities are converted into entity embedding vectors of a uniform dimension using a knowledge graph embedding model. The cosine similarity between different entity embedding vectors is calculated. When the cosine similarity between two entities is higher than a preset alignment threshold, the two entities are determined to be the same entity and are merged. During the merging process, all attributes and relationship information of the two entities are integrated to eliminate entity redundancy and ambiguity between different data sources.

[0035] This project establishes interdisciplinary bridging relationships between concepts from different disciplines. These relationships include at least one of the following: applied relationships, similar relationships, or relationships based on theoretical foundations. The bridging relationship construction process consists of two stages: automatic extraction and manual verification. First, potential interdisciplinary relationships are automatically mined through co-occurrence analysis of interdisciplinary corpora and a relationship extraction model. Then, professionals from the corresponding disciplines verify and supplement these relationships to ensure their accuracy and rationality. After completing this process, a complete multimodal interdisciplinary knowledge graph is finally constructed and stored in the Neo4j graph database, supporting efficient entity queries and relationship path retrieval.

[0036] This step of the retrieval process employs a multi-path recall strategy, simultaneously performing vector similarity retrieval, keyword matching retrieval, and knowledge graph structured path retrieval.

[0037] Vector similarity retrieval is based on HNSW dense vector indexing for approximate nearest neighbor search. All knowledge fragments and entity nodes in the knowledge graph are pre-converted into embedding vectors of the same dimension as the unified query semantic vector, and an HNSW index is constructed. During retrieval, the cosine similarity between the unified query semantic vector and the vectors in the index is calculated, and the top N knowledge fragments with the highest similarity are returned, where N is the preset vector retrieval recall number.

[0038] Keyword matching retrieval is based on BM25 scoring using a sparse inverted index. First, keywords are extracted from the user's multimodal queries, prioritizing the retention of subject-specific terminology and core concepts while removing meaningless stop words. Based on the extracted keywords, a search is performed in a pre-constructed inverted index of knowledge fragments, sorted by BM25 score, and the top M knowledge fragments by score are returned, where M is the preset keyword retrieval recall number.

[0039] The knowledge graph structured path retrieval is based on graph database for entity matching and relation path discovery. First, it matches the core entity nodes in the knowledge graph by using a unified query semantic vector. Starting from the matched core entity nodes, it performs a breadth-first search in the graph database with a search depth of a preset path search depth to discover interdisciplinary reasoning paths that connect concepts from different disciplines. At the same time, it records the entities, relationships and corresponding knowledge fragments contained in the path and returns interdisciplinary reasoning paths that meet the semantic association requirements.

[0040] Step 4, Enhanced Hint Assembly and Preliminary Answer Generation: This step assembles knowledge fragments, interdisciplinary reasoning paths, and multimodal queries into enhanced hints, inputs the enhanced hints into the large language model, and generates a preliminary answer.

[0041] First, the knowledge fragments obtained from multi-path recall are reordered. The reordering score is calculated using the following formula: in, Indicates the first A fragment of knowledge, Representing knowledge fragments Reordered scores Representing knowledge fragments Vector retrieval relevance score, Representing knowledge fragments With multimodal queries semantic relevance score, Representing knowledge fragments The importance weight in interdisciplinary reasoning paths , , Preset weighting coefficients and satisfying .

[0042] In this embodiment, Set to 0.3, Set to 0.4, Setting the weight to 0.3 balances the influence of three dimensions: vector retrieval relevance, query semantic relevance, and inference path importance. The vector retrieval relevance score is the cosine similarity score obtained during the vector similarity retrieval phase; the semantic relevance score is the semantic matching score between the knowledge fragment and the multimodal query, calculated using a pre-trained semantic matching model; and the path importance weight is the centrality score of the interdisciplinary inference path to which the knowledge fragment resides, calculated using the betweenness centrality algorithm in graph computing. Knowledge fragments located on core inference paths receive higher path importance weights.

[0043] Due to the context length limitation of the large language model, a greedy selection algorithm is used to select the optimal subset from the reordered knowledge fragments to construct the context window. In this embodiment, the ChatGLM3-6B open-source large language model is selected, and it has been fine-tuned in advance for subject-specific question-answering scenarios. The context window length is set to 4096 tokens. The greedy selection algorithm selects knowledge fragments in descending order of reordering score until the total number of tokens of the selected knowledge fragments does not exceed the reserved length of the context window, ensuring that the most relevant and critical knowledge information for reasoning is retained within the limited context length.

[0044] The context window is concatenated with a formatted representation of the interdisciplinary reasoning path to generate an enhanced hint. The structure of the enhanced hint is fixed at four parts: the user's original multimodal query content, a formatted description of the interdisciplinary reasoning path, a set of reordered and filtered knowledge fragments, and answer generation constraints. These constraints include requirements such as the answer being generated based on the provided knowledge fragments, reflecting interdisciplinary reasoning logic, and ensuring the professionalism and accuracy of the content. The assembled enhanced hint is then input into a fine-tuned large language model. The large language model uses the information in the enhanced hint to perform reasoning and generation, outputting a preliminary answer that meets the requirements.

[0045] Step 5, Preliminary Answer Fact Triple Extraction: This step extracts fact triples from the preliminary answer to obtain at least one fact triple. Each fact triple includes a subject entity, a predicate relation, and an object entity.

[0046] Specifically, a fact triplet extraction model based on a pre-trained language model is employed. This model is pre-tuned for specialized texts within a specific subject area, enabling accurate identification of entities and relationships within the text. During extraction, the initial answer is first segmented into multiple independent sentences. Entity recognition and relationship extraction are then performed on each sentence, outputting the corresponding fact triplets. All extracted fact triplets are then deduplicated and cleaned, removing duplicate and meaningless triplets, ultimately yielding the complete set of fact triplets corresponding to the initial answer.

[0047] Step 6, Fact Triple Matching Validation and Confidence Calculation: This step matches and validates each fact triple with a multimodal interdisciplinary knowledge graph, and calculates the fact confidence of each fact triple based on the scoring function of the knowledge graph embedding model.

[0048] In this embodiment, the TransE model is used as the knowledge graph embedding model. The TransE model is pre-trained based on the constructed multimodal interdisciplinary knowledge graph. After training, all entities and relations in the knowledge graph are mapped to embedding vectors of a unified dimension. The dimension of the embedding vector is set to 1024 dimensions, which can fully capture the semantic features of entities and relations.

[0049] For each fact triple The TransE model is used as the knowledge graph embedding model to calculate the score. The calculation formula is as follows: in, Indicates the subject entity, Indicates predicate relationship, Indicates the object entity, This indicates the rating of the fact triple. Indicates the subject entity Embedded vector, Indicating predicate relations Embedded vector, Indicates the object entity Embedded vector, Represents L2 norm operations; The rating is converted into fact confidence score using the sigmoid function, calculated as follows: in, Representing fact triples Confidence level of facts This is the scaling factor. For threshold parameters, It is a natural constant. In this embodiment, the scaling factor is... Set to 5, threshold parameter Setting this parameter to 0 will smoothly map the rating to a confidence level between 0 and 1. The closer the confidence level is to 1, the higher the credibility of the fact; the closer the confidence level is to 0, the lower the credibility of the fact and the higher the probability of factual error.

[0050] Step 7, Answer Quality Assessment and Iterative Optimization: This step assesses the quality of the initial answer based on factual confidence. If the assessment result is lower than the preset quality threshold, the retrieval strategy is adjusted based on the verification feedback. The retrieval is then repeated based on the adjusted retrieval strategy, and the optimized answer is generated iteratively until the assessment result of the optimized answer meets the preset quality threshold.

[0051] First, answer quality is assessed by calculating sentence-level confidence. For sentences containing multiple fact triples, the sentence-level confidence is the weighted average of the fact confidence of all fact triples in the sentence. The weights are determined by the importance of each fact triple in the sentence, with core fact triples assigned higher weights and supplementary fact triples assigned lower weights. The weight allocation is automatically calculated by a pre-trained importance scoring model.

[0052] Next, calculate the answer-level confidence score. The initial answer-level confidence score is determined by the weighted average and minimum of the sentence-level confidence scores, calculated using the following formula: in, Indicates a preliminary answer Answer-level confidence level Indicates a preliminary answer The collection of sentences in the text Indicates the number of sentences. Sentence Sentence-level confidence, To balance the weighting coefficients of the average confidence level and the lowest confidence level, This indicates the minimum value operation.

[0053] In this embodiment, the weighting coefficient Setting the weight to 0.7 allows for a balance between the overall average confidence of the answer and the lowest confidence of individual sentences, preventing situations where the overall average confidence is high while a single core sentence contains a serious factual error. The preset quality threshold is set to 0.85. When the calculated answer-level confidence is greater than or equal to 0.85, the evaluation result is considered to meet the preset quality threshold; when the answer-level confidence is lower than 0.85, the evaluation result is considered to be below the preset quality threshold, requiring iterative optimization.

[0054] During the iterative optimization process, the knowledge points to be optimized are first identified based on fact triples with a confidence level lower than a preset fact threshold in the verification feedback. The preset fact threshold is set to 0.7. The entities and concepts corresponding to fact triples with a confidence level lower than 0.7 are the knowledge points to be optimized. For the knowledge points to be optimized, the subject range of the search is expanded, and the number of searches in related subjects is increased. Related subjects that were not originally included in the search scope are included in the search scope in the iterative search, and the recall limit of the corresponding subject for the knowledge point is increased. For the knowledge points to be optimized, the search depth of cross-disciplinary reasoning paths is deepened. The original path search depth is 2 hops, which is increased to 3 to 4 hops in the iterative search, increasing the number of reasoning steps and mining deeper-level cross-disciplinary related knowledge. Based on the adjusted search strategy, the search operation is re-executed to obtain supplementary knowledge fragments and supplementary reasoning paths. The supplementary knowledge fragments and reasoning paths are integrated into the enhanced prompts, re-input into the large language model to generate optimized answers, and then the fact verification and quality assessment process is repeated on the optimized answers until the answer-level confidence level of the generated optimized answers meets the preset quality threshold. In this embodiment, the maximum number of iterations is set to 3. When the quality threshold is not met after 3 iterations, the iteration is terminated and a prompt is output to inform the user that the current question cannot generate an answer that meets the credibility requirements, thus avoiding the waste of resources caused by infinite iteration.

[0055] Step 8, Final Answer Output and Reference Annotation Generation: This step outputs the final answer and generates structured reference annotations for the knowledge points in the final answer.

[0056] Specifically, a traceability link is first established from the knowledge units in the final answer to the original knowledge fragments. The final answer is then divided into multiple independent knowledge units, each corresponding to a core knowledge point. For each knowledge unit, the corresponding source knowledge fragment is matched, and the source location range and matching confidence of the knowledge unit are determined. Only the traceability link of knowledge units with a matching confidence higher than a preset citation threshold is retained.

[0057] For each valid citation, a structured field is generated, including a unique citation identifier, source name, source type, citation confidence score, original resource access link, and citation content summary. The source name is the name of the original data source corresponding to the knowledge fragment; the source type includes textbooks, papers, encyclopedias, industry standards, etc.; the citation confidence score corresponds to the factual confidence level of the corresponding knowledge unit; the original resource access link is the publicly accessible address of the data source; and the citation content summary is the core content of the source knowledge fragment.

[0058] The final answer inserts a corresponding citation marker for each knowledge unit, which is associated with a corresponding structured field. Users can view the citation details through the citation marker. Simultaneously, a complete reasoning path from query to final answer is generated and visualized. The reasoning path includes query nodes, intermediate concept nodes, answer nodes, and relational edges connecting each node, fully presenting the interdisciplinary reasoning logic in the answer generation process and improving the interpretability of the answer.

[0059] like Figure 2 As shown, this embodiment also discloses an interdisciplinary intelligent Q&A system. The system is used to execute the above-mentioned interdisciplinary intelligent Q&A method. The system consists of a multimodal query understanding module, an interdisciplinary knowledge graph construction and management module, a hybrid retrieval and reasoning module, an answer generation module, a fact verification module, an answer optimization control module, and an interpretability output module. The modules interact with each other through a preset communication interface to achieve automated collaborative processing throughout the entire process.

[0060] The multimodal query understanding module receives multimodal queries input by the user and performs multimodal feature extraction and fusion to generate a unified query semantic vector. This module integrates a text encoder, a vision-language pre-trained model, a formula recognition model, and a multimodal fusion unit to perform feature extraction from text, image, and formula data, as well as weighted fusion of multimodal features, outputting a unified query semantic vector.

[0061] The interdisciplinary knowledge graph construction and management module is used to build, store, and dynamically update multimodal interdisciplinary knowledge graphs. This module includes an ontology definition unit, entity relation extraction unit, entity alignment unit, interdisciplinary bridging unit, and graph database storage unit, completing the full lifecycle management of the knowledge graph. It also supports dynamic updates of the knowledge graph; when new subject data is published, it can quickly extract data, align entities, supplement relations, and update the knowledge graph content.

[0062] The hybrid retrieval and reasoning module is used to retrieve knowledge fragments and interdisciplinary reasoning paths related to multimodal queries within a multimodal, interdisciplinary knowledge graph based on a unified query semantic vector. This module integrates a vector retrieval unit, a keyword retrieval unit, and a graph path retrieval unit. These three units execute multi-path retrieval in parallel to comprehensively acquire relevant knowledge fragments and reasoning paths.

[0063] The answer generation module assembles knowledge fragments, interdisciplinary reasoning paths, and multimodal queries into enhanced hints, which are then input into a large language model to generate a preliminary answer. This module includes a knowledge fragment reordering unit, a context window construction unit, a hint assembly unit, and a large language model reasoning unit, completing the selection of knowledge fragments, the assembly of enhanced hints, and the generation of the answer.

[0064] The fact verification module extracts fact triples from the initial answer, obtaining at least one fact triple. Each fact triple is then matched and verified against a multimodal, interdisciplinary knowledge graph. The fact confidence score for each fact triple is calculated based on the scoring function of the knowledge graph embedding model. This module incorporates a triple extraction unit, a knowledge graph embedding unit, and a confidence score calculation unit, enabling fine-grained, structured fact verification of the generated answer.

[0065] The answer optimization control module assesses the quality of the initial answer based on factual confidence. If the assessment result falls below a preset quality threshold, the retrieval strategy is adjusted based on verification feedback. This involves iterative optimization of the hybrid retrieval and reasoning module and the answer generation module until the assessment result of the generated optimized answer meets the preset quality threshold. This module integrates a quality assessment unit, a retrieval strategy adjustment unit, and an iterative control unit, achieving closed-loop collaboration between generation and verification.

[0066] The interpretability output module is used to output the final answer and generate structured reference annotations for the knowledge points in the final answer. This module includes a reference tracing unit, a reference structure generation unit, and a reasoning path visualization unit to complete the output of the final answer and enhance its interpretability.

[0067] In some alternative implementations, the system can be deployed on a cloud server to provide users with interdisciplinary intelligent Q&A services through a web interface or mobile application, or it can be deployed on a local terminal device to provide Q&A functionality in offline mode, adapting to different application scenarios.

[0068] In some optional implementations, multimodal interdisciplinary knowledge graphs can be expanded to cover more subject areas according to the needs of application scenarios. The knowledge graph can be expanded by simply adjusting the subject ontology definition and data source accordingly, thus adapting to the interdisciplinary Q&A needs of more fields.

[0069] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A cross-disciplinary intelligent question-answering method, characterized in that, The method includes the following steps: Step 1: Receive a multimodal query input by the user, wherein the multimodal query includes at least one of text data, image data, or formula data; Step 2: Extract and fuse multimodal features from the multimodal query to generate a unified query semantic vector; Step 3: Based on the unified query semantic vector, search in the pre-constructed multimodal interdisciplinary knowledge graph to obtain knowledge fragments and interdisciplinary reasoning paths related to the multimodal query. The multimodal interdisciplinary knowledge graph covers at least two disciplines in mathematics, physics, chemistry, biology, and computer science, and the concepts of different disciplines are associated through interdisciplinary bridging relationships. Step four: Assemble the knowledge fragment, the interdisciplinary reasoning path, and the multimodal query into an enhanced hint, and input the enhanced hint into the large language model to generate a preliminary answer; Step 5: Extract fact triples from the preliminary answer to obtain at least one fact triple. Each fact triple includes a subject entity, a predicate relation, and an object entity. Step 6: Match and verify each fact triple with the multimodal interdisciplinary knowledge graph, and calculate the fact confidence of each fact triple based on the scoring function of the knowledge graph embedding model; Step 7: Evaluate the quality of the preliminary answer based on the factual confidence level. If the evaluation result is lower than the preset quality threshold, adjust the retrieval strategy based on the verification feedback, and re-retrieve and iteratively generate an optimized answer based on the adjusted retrieval strategy until the evaluation result of the optimized answer meets the preset quality threshold. Step 8: Output the final answer and generate structured reference annotations for the knowledge points in the final answer.

2. The interdisciplinary intelligent question-answering method according to claim 1, characterized in that, Step two specifically includes: The text data is encoded using a Transformer-based text encoder to generate text feature vectors; The image data is encoded using a vision-language pre-trained model to generate image feature vectors; The formula data is converted into a LaTeX representation using a formula recognition model, and the LaTeX representation is encoded to generate a formula feature vector; A multimodal fusion strategy based on an attention mechanism is adopted to calculate the attention weights corresponding to the text feature vector, the image feature vector, and the formula feature vector. The text feature vector, the image feature vector, and the formula feature vector are then weighted and fused according to the attention weights to generate the unified query semantic vector. The formula for calculating the attention weights is as follows: in, Represents a set of modes. Including text modality, img modality, and formula modality. Indicates the current calculated mode. Representing modes Attention weights Representing modes The attention parameter vector to be trained, express transpose, Representing modes eigenvectors, Represents an exponential function. This represents the summation operation. Traversal All modalities.

3. The interdisciplinary intelligent question-answering method according to claim 1, characterized in that, The construction of the multimodal interdisciplinary knowledge graph in step three specifically includes: For each discipline in mathematics, physics, chemistry, biology, and computer science, define the conceptual system and relationship types of each discipline to form a discipline ontology definition; Entities, relationships, and attributes are extracted from multi-source data, including textbook data, academic paper data, and encyclopedia data. Entity alignment is performed on the same entities from different data sources. The entity alignment adopts a similarity calculation method based on entity embedding. Establishing interdisciplinary bridging relationships between concepts from different disciplines, wherein the interdisciplinary bridging relationships include at least one of those applied to relations, similar to relations, or based on relations in theory.

4. The interdisciplinary intelligent question-answering method according to claim 1, characterized in that, Step three involves acquiring knowledge fragments and interdisciplinary reasoning paths related to the multimodal query, specifically including: A multi-path recall strategy is adopted, simultaneously performing vector similarity retrieval, keyword matching retrieval, and knowledge graph structured path retrieval; The vector similarity retrieval is based on HNSW dense vector index to perform approximate nearest neighbor search and obtain knowledge fragments that are semantically similar to the unified query semantic vector. The keyword matching retrieval is based on BM25 scoring using a sparse inverted index to obtain knowledge fragments containing the query keywords; The knowledge graph structured path retrieval is based on graph databases to perform entity matching and relation path discovery, and to obtain interdisciplinary reasoning paths that connect concepts from different disciplines.

5. The interdisciplinary intelligent question-answering method according to claim 1, characterized in that, In step four, the knowledge fragment, the interdisciplinary reasoning path, and the multimodal query are assembled into an enhanced hint, specifically including: The knowledge fragments are reordered, and the reordering score is calculated using the following formula: in, Indicates the first A fragment of knowledge, Representing knowledge fragments Reordered scores Representing knowledge fragments Vector retrieval relevance score, Representing knowledge fragments With multimodal queries semantic relevance score, Representing knowledge fragments The importance weight in interdisciplinary reasoning paths , , Preset weighting coefficients and satisfying ; Based on the context length limitation of the large language model, a greedy selection algorithm is used to select the optimal subset from the reordered knowledge fragments to construct the context window; The enhanced hint is generated by concatenating the context window with the formatted representation of the interdisciplinary reasoning path.

6. The interdisciplinary intelligent question-answering method according to claim 1, characterized in that, In step six, the fact confidence score of each fact triple is calculated based on the scoring function of the knowledge graph embedding model, specifically including: For each fact triple The TransE model is used as the knowledge graph embedding model to calculate the score. The calculation formula is as follows: in, Indicates the subject entity, Indicates predicate relationship, Indicates the object entity, This indicates the rating of the fact triple. Indicates the subject entity Embedded vector, Indicating predicate relations Embedded vector, Indicates the object entity Embedded vector, Represents L2 norm operations; The rating is converted into fact confidence score using the sigmoid function, calculated as follows: in, Representing fact triples Confidence level of facts This is the scaling factor. For threshold parameters, It is a natural constant.

7. The interdisciplinary intelligent question-answering method according to claim 1, characterized in that, In step seven, the preliminary answer is assessed for quality based on the stated confidence level, specifically including: To calculate sentence-level confidence, for a sentence containing multiple fact triples, the sentence-level confidence is the weighted average of the fact confidence of all fact triples in the sentence, with the weights determined by the importance of each fact triple in the sentence; To calculate the answer-level confidence score, the preliminary answer-level confidence score is determined by the weighted average and minimum of the sentence-level confidence scores. The calculation formula is as follows: in, Indicates a preliminary answer Answer-level confidence level Indicates a preliminary answer The collection of sentences in the text Indicates the number of sentences. Sentence Sentence-level confidence, To balance the weighting coefficients of the average confidence level and the lowest confidence level, This indicates the minimum value operation; The answer-level confidence level is compared with a preset confidence threshold. If the answer-level confidence level is lower than the preset confidence threshold, the evaluation result is determined to be lower than the preset quality threshold.

8. The interdisciplinary intelligent question-answering method according to claim 1, characterized in that, In step seven, adjusting the retrieval strategy based on verification feedback specifically includes: Based on the fact triples whose fact confidence level is lower than the preset fact threshold in the verification feedback, the knowledge points to be optimized are identified. For the knowledge points that need optimization, the scope of the search subjects should be expanded, and the number of searches for related subjects should be increased; For the knowledge points that need optimization, the search depth of interdisciplinary reasoning paths is deepened and the number of reasoning steps is increased; The search operation is re-executed based on the adjusted search strategy to obtain supplementary knowledge fragments and supplementary reasoning paths.

9. The interdisciplinary intelligent question-answering method according to claim 1, characterized in that, In step eight, structured reference annotations are generated for the knowledge points in the final answer, specifically including: Establish a tracing link from the knowledge unit in the final answer to the original knowledge fragment, and determine the source knowledge fragment, source location range and matching confidence of each knowledge unit; For each reference, a structured field is generated, which includes a unique reference identifier, source name, source type, reference credibility score, original resource access link, and reference content summary; In the final answer, a corresponding reference tag is inserted for each knowledge unit, and the reference tag is associated with the structured field; Generate and visualize the complete reasoning path from query to final answer. The reasoning path includes query nodes, intermediate concept nodes, answer nodes, and relational edges connecting each node.

10. A cross-disciplinary intelligent question-answering system, applicable to the cross-disciplinary intelligent question-answering method according to any one of claims 1 to 9, characterized in that, The system consists of: The multimodal query understanding module is used to receive multimodal queries input by users, wherein the multimodal queries include at least one of text data, image data or formula data, and to extract and fuse multimodal features of the multimodal queries to generate a unified query semantic vector; The interdisciplinary knowledge graph construction and management module is used to construct, store, and dynamically update multimodal interdisciplinary knowledge graphs. The multimodal interdisciplinary knowledge graphs cover at least two disciplines among mathematics, physics, chemistry, biology, and computer science, and the concepts of different disciplines are associated through interdisciplinary bridging relationships. The hybrid retrieval and reasoning module is used to retrieve knowledge fragments and interdisciplinary reasoning paths related to the multimodal interdisciplinary knowledge graph based on the unified query semantic vector. The answer generation module is used to assemble the knowledge fragment, the interdisciplinary reasoning path, and the multimodal query into an enhanced hint, input the enhanced hint into a large language model, and generate a preliminary answer; The fact verification module is used to extract fact triples from the preliminary answer, obtain at least one fact triple, and match and verify each fact triple with the multimodal interdisciplinary knowledge graph. The fact confidence of each fact triple is calculated based on the scoring function of the knowledge graph embedding model. The answer optimization control module is used to evaluate the quality of the preliminary answer based on the factual confidence level. When the evaluation result is lower than the preset quality threshold, the retrieval strategy is adjusted according to the verification feedback, and the hybrid retrieval and reasoning module and the answer generation module are controlled to perform iterative optimization until the evaluation result of the generated optimized answer meets the preset quality threshold. An interpretability output module is used to output the final answer and generate structured reference annotations for the knowledge points in the final answer.

Images