Cross-domain heterogeneous data query system and method based on large model and knowledge graph

Abstract

The invention discloses a cross-domain heterogeneous data query system and method based on a large model and a knowledge graph, belongs to the technical field of information retrieval, and aims to solve the technical problem of complex relation reasoning in cross-domain heterogeneous data query. Comprising a data input and preprocessing module used for collecting multi-modal data to obtain feature vectors; the knowledge graph construction and management module is used for constructing a knowledge graph and providing query service based on the knowledge graph; the bidirectional enhancement module is used for writing the reasoning result of the large language model into a knowledge graph and carrying out version management; the domain adaptation layer is used for carrying out model training on the large language model based on a lightweight adapter and a domain adaptation mechanism; the real-time query and reasoning module verifies and supplements the candidate answers based on a knowledge graph to generate an initial answer, and explains a reasoning path based on a causal reasoning network to generate a final answer; and the interpretability and transparency module is used for displaying knowledge in the knowledge graph through a visual interface and providing auditing service based on the operation day.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval technology, and more specifically to a cross-domain heterogeneous data query system and method based on a large model and a knowledge graph. Background Art

[0002] With the development of the internet, the amount of data has grown exponentially, encompassing a wide range of formats, including text, images, and video. Traditional single-modality data analysis methods struggle to cope with this diverse data environment. In recent years, large language models and knowledge graphs, two important technologies, have made significant progress in natural language processing and structured knowledge representation, respectively. However, effectively combining the strengths of these two approaches to achieve efficient querying and reasoning on heterogeneous, cross-domain data remains an urgent challenge.

[0003] Large language models such as Deepseek and Tongyi Qianwen, through large-scale pre-training, possess powerful language understanding and generation capabilities. These models have demonstrated excellent performance in fields such as question-answering systems, sentiment analysis, and machine translation. However, large language models are often limited in complex multi-entity relationship reasoning tasks due to their lack of structured knowledge support, resulting in reduced performance in certain specific domains. Knowledge graphs are a structured form of knowledge representation that effectively captures relationships between entities. While knowledge graphs provide rich contextual information, their static nature makes them less adaptable to dynamically changing data and scenarios. Furthermore, traditional knowledge graphs have limited reasoning capabilities and struggle to handle complex semantic relationships. Existing knowledge graph construction is complex, requiring the coordinated implementation of multiple tasks such as named entity recognition and relationship extraction, which is costly and prone to semantic loss. Cross-domain data integration is difficult, and heterogeneous data cannot be shared due to formatting and semantic differences, leading to data silos. Knowledge flows unidirectionally: existing methods only utilize knowledge graphs to enhance large language models or construct knowledge graphs from large language models, failing to form a bidirectional optimization loop. The query efficiency is low, and the retrieval method based on vector matching is prone to missing key information in complex relational reasoning.

[0004] How to solve the complex relational reasoning problem in cross-domain heterogeneous data queries is a technical problem that needs to be solved. Summary of the Invention

[0005] The technical task of the present invention is to address the above shortcomings and provide a cross-domain heterogeneous data query system and method based on a large model and knowledge graph to solve the complex relationship reasoning problem in cross-domain heterogeneous data query.

[0006] In the first aspect, the present invention provides a cross-domain heterogeneous data query system based on a large model and a knowledge graph, comprising a data input and preprocessing module, a knowledge graph construction and management module, a bidirectional enhancement module, a domain adaptation layer, a real-time query and reasoning module, and an explainability and transparency module;

[0007] The data input and preprocessing module is used to collect multimodal data, perform data input and preprocessing, feature extraction and feature alignment operations on the data, and obtain feature vectors;

[0008] The knowledge graph construction and management module is used to extract entities, relationships and attributes based on the feature vectors output by the data input and preprocessing module, and to construct a knowledge graph using entities, relationships and attributes as knowledge, to dynamically update the knowledge graph based on the inference results output by the large language model, and to provide query services based on the knowledge graph;

[0009] The bidirectional enhancement module is used to read knowledge from the knowledge graph and perform model training on the large language model based on the knowledge, to write the reasoning results of the large language model into the knowledge graph, and to perform version management on the updates of the knowledge graph and the large language model;

[0010] The domain adaptation layer is used to perform model training on the large language model based on the lightweight adapter and the domain adaptation mechanism;

[0011] The real-time query and reasoning module is used to receive user input, analyze the user input intent through the trained large language model, output candidate answers, verify and supplement the candidate answers based on the knowledge graph to generate an initial answer, and generate a final answer based on the causal reasoning network to interpret the reasoning path;

[0012] The explainability and transparency module is used to display the knowledge in the knowledge graph through a visual interface, and to record user input, the reasoning process of the large language model, and the update process of the knowledge graph, generate operation logs, and provide audit services based on the operation date.

[0013] Preferably, the data input and preprocessing module is used to perform the following operations:

[0014] Collect multimodal data through input interfaces, including text, images, audio, and video;

[0015] Perform data preprocessing on multimodal data, extract the content of text data, convert the format of image, audio and video data, and extract images and audio from video data;

[0016] For pre-processed text data, we use a pre-trained language model to perform word embedding and combine it with a domain dictionary for syntactic analysis to identify entities in the text and the relationships between them. We then output the word embedding vectors, dependency syntactic structure, and entity and relationship lists for the text data.

[0017] For pre-processed image data, feature extraction is performed through a convolutional neural network to obtain visual features. Facial expressions and gestures are then recognized using a posture estimation model to obtain key point coordinates and action sequences. The visual feature vector, key point coordinates, and action sequence of the image data are then output.

[0018] For pre-processed audio data, we extract voiceprint features through spectral analysis and combine it with the emotional speech recognition model to analyze the acoustic features of the audio, including intonation and pauses, and output the emotional label and acoustic feature vector.

[0019] By maximizing the inter-modal similarity and minimizing the intra-modal difference, the word embedding vector, visual feature vector and voiceprint feature vector are mapped to a unified space. The maximum mean difference is used to constrain the inter-modal distribution difference for feature alignment to obtain the final feature vector.

[0020] Preferably, the knowledge graph construction and management module is used to perform the following operations:

[0021] Based on the feature vectors output by the data input and preprocessing module, entities are extracted through the domain ontology library and entity recognition model. Based on the feature vectors output by the data input and preprocessing module, relationships are extracted through the relationship classification model. The legitimacy of the relationships is verified by the rule engine, and attributes are filled based on the inference results output by the large language model.

[0022] Build a knowledge graph using entities, relationships, and attributes as knowledge;

[0023] Update the knowledge graph based on the inference results of the large language model. If the inference results output by the large language model conflict with the existing knowledge in the knowledge graph, perform knowledge screening through manual review or confidence-weighted voting mechanism.

[0024] It provides query services through knowledge graphs, calls the SPARQL query engine to support complex queries, and provides index optimization based on inverted indexes and graph databases.

[0025] Preferably, the bidirectional enhancement module is configured to perform the following operations:

[0026] When reasoning through a large language model, the knowledge graph is called to obtain background information;

[0027] Update the inference results of the large language model to the knowledge graph;

[0028] Inject the structured knowledge of the knowledge graph into the large language model as the input of the large language model;

[0029] Updates to knowledge graphs and large language models are managed through real-time update mechanisms and version control mechanisms.

[0030] Preferably, the domain adaptation layer is configured to perform the following operations:

[0031] Add an adapter module to the last layer of the model and fine-tune the parameters using labeled data from the target domain;

[0032] Add a domain classifier, calculate the domain adaptation loss, and update the parameters of the adapter module through backpropagation;

[0033] We use unlabeled data in the source domain to pre-train the adapter through self-supervised learning, and use labeled data in the target domain to fine-tune the adapter through semi-supervised learning, and then optimize the adapter by combining knowledge distillation techniques from the source domain.

[0034] Preferably, the real-time query and reasoning module is used to perform the following operations:

[0035] Parse user multimodal input and identify user intent;

[0036] Based on user intent, candidate answers are generated through a large language model;

[0037] Call the knowledge graph to obtain background information, supplement the candidate answers based on user input and background information, and generate preliminary answers;

[0038] Verify and supplement the preliminary answers based on the knowledge graph, verify the legitimacy of the answers through knowledge graph queries, and supplement missing information;

[0039] Based on the probability of the large language model output and the confidence of the knowledge graph, the confidence of the output of the large language model and the knowledge in the knowledge graph is evaluated to generate the final answer.

[0040] Construct a causal graph to represent the causal relationship between entities, and explain the reasoning path through causal reasoning algorithms.

[0041] Preferably, the explainability and transparency module is configured to perform the following operations:

[0042] Visualization tools are used to display reasoning paths and attention heat maps, graphically displaying the knowledge graph entities, relationships, and weights used by the large language model. Heat maps are used to show the degree to which the large language model focuses on different modal features when processing input.

[0043] Record the input of each query, the large language model inference process, and the knowledge graph update record, and provide an API for third-party audit systems to call.

[0044] In a second aspect, the present invention provides a cross-domain heterogeneous data query method based on a large model and a knowledge graph, which is used to implement cross-domain heterogeneous data query through a cross-domain heterogeneous data query system based on a large model and a knowledge graph as described in any one of the first aspects, and the method includes the following steps:

[0045] Data input and preprocessing: Collect multimodal data, perform data input and preprocessing, feature extraction and feature alignment operations on the data to obtain feature vectors;

[0046] Knowledge graph construction and management: Extract entities, relationships, and attributes based on data input and feature vectors output by the preprocessing module, and construct a knowledge graph using entities, relationships, and attributes as knowledge. Dynamically update the knowledge graph based on the inference results output by the large language model, and provide query services based on the knowledge graph.

[0047] Bidirectional enhancement: Read knowledge from the knowledge graph, train the large language model based on the knowledge, write the inference results of the large language model into the knowledge graph, and manage the versioning of updates to the knowledge graph and the large language model.

[0048] Domain Adaptation: Training large language models based on lightweight adapters and domain adaptation mechanisms;

[0049] Real-time query and reasoning: Receive user input, analyze the user's input intent using a trained large language model, output candidate answers, verify and supplement the candidate answers based on the knowledge graph to generate an initial answer, and generate the final answer by interpreting the reasoning path based on the causal reasoning network.

[0050] Interaction management: Display the knowledge in the knowledge graph through a visual interface, record user input, the reasoning process of the large language model, and the update process of the knowledge graph, generate operation logs, and provide audit services based on the operation date.

[0051] The cross-domain heterogeneous data query system and method based on a large model and knowledge graph of the present invention has the following advantages:

[0052] 1. Significantly Improved Query Accuracy: Through a bidirectional enhancement mechanism, the large language model can dynamically acquire background information from the knowledge graph, improving the accuracy of complex relational reasoning tasks. Simultaneously, the knowledge graph is updated using the reasoning output of the large language model, further enhancing the real-time and comprehensive nature of the knowledge graph. Experimental results show that the system significantly improves query accuracy compared to traditional methods on multiple benchmark datasets.

[0053] 2. Efficient cross-domain data fusion: The system introduces cross-modal contrastive learning and modality-adaptive regularization to effectively address the semantic gap problem in multimodal data fusion. Experimental verification shows that the system outperforms other advanced methods in cross-domain data fusion tasks, especially when processing unstructured data, demonstrating greater robustness.

[0054] 3. Flexible domain adaptability: A lightweight adapter structure and domain-adaptive loss function are designed to enable rapid migration to new domains with a small amount of labeled data. This significantly reduces the reliance on large amounts of labeled data in fields such as healthcare and finance.

[0055] 4. Enhanced explainability and transparency: A detailed visualization interface and causal reasoning network are provided to enhance the explainability and transparency of the system. In high-risk applications, users can clearly understand the basis for the system's decision-making, thereby improving user trust.

[0056] 5. Real-time performance and high efficiency: By optimizing the system architecture, we ensure high efficiency and low latency in large-scale data environments. Experimental data shows that when processing millions of query requests, the average system response time is far lower than the industry standard.

[0057] 6. Multi-task support: Supports multiple query tasks, including but not limited to relational reasoning, entity linking, attribute prediction, etc., providing users with a full range of services. Whether it is a simple text query or a complex cross-modal data fusion task, the system can complete it efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] The present invention will be further described below with reference to the accompanying drawings.

[0060] Figure 1 This is a structural block diagram of Example 1, a cross-domain heterogeneous data query system based on a large model and a knowledge graph. DETAILED DESCRIPTION

[0061] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0062] Embodiments of the present invention provide a cross-domain heterogeneous data query system and method based on a large model and a knowledge graph, which are used to solve complex relationship reasoning problems in cross-domain heterogeneous data queries.

[0063] Example 1:

[0064] The present invention provides a cross-domain heterogeneous data query system based on a large model and a knowledge graph, including a data input and preprocessing module, a knowledge graph construction and management module, a bidirectional enhancement module, a domain adaptation layer, a real-time query and reasoning module, and an explainability and transparency module.

[0065] The data input and preprocessing module is used to collect multimodal data, perform data input and preprocessing, feature extraction and feature alignment operations on the data to obtain feature vectors.

[0066] As a specific implementation of the data input and preprocessing module, this module is used to perform the following operations:

[0067] (1) Collect multimodal data through the input interface, including text, images, audio and video;

[0068] (2) Preprocess multimodal data, extract the content of text data, convert the format of image, audio, and video data, and extract images and audio from video data;

[0069] (3) For pre-processed text data, word embedding is performed using a pre-trained language model, and syntactic analysis is performed in combination with a domain dictionary to identify entities in the text and the relationships between entities. The word embedding vector, dependency syntactic structure, and entity and relationship lists of the text data are output;

[0070] (4) For the pre-processed data of the image type, feature extraction is performed through a convolutional neural network to obtain visual features, and facial expression and gesture recognition is performed in combination with a posture estimation model to obtain key point coordinates and action sequences, and the visual feature vector, key point coordinates and action sequence of the image type data are output;

[0071] (5) For the pre-processed audio data, the voiceprint features are extracted through spectral analysis, and the acoustic features including intonation and pauses in the audio are analyzed in combination with the emotional speech recognition model, and the emotional label and acoustic feature vector are output;

[0072] (6) The word embedding vector, visual feature vector, and voiceprint feature vector are mapped to a unified space by maximizing the inter-modal similarity and minimizing the intra-modal difference. The maximum mean difference is used to constrain the inter-modal distribution difference for feature alignment to obtain the final feature vector.

[0073] The knowledge graph construction and management module is used to extract entities, relationships and attributes based on the feature vectors output by the data input and preprocessing module, and to construct a knowledge graph using entities, relationships and attributes as knowledge. It is used to dynamically update the knowledge graph based on the inference results output by the large language model, and to provide query services based on the knowledge graph.

[0074] As a specific implementation of the knowledge graph construction and management module, this module is used to perform the following operations:

[0075] (1) Based on the feature vectors output by the data input and preprocessing module, entities are extracted through the domain ontology library and entity recognition model. Based on the feature vectors output by the data input and preprocessing module, relationships are extracted through the relationship classification model. The legitimacy of the relationships is verified by the rule engine, and attributes are filled based on the inference results output by the large language model.

[0076] (2) Constructing knowledge graphs by taking entities, relationships, and attributes as knowledge;

[0077] (3) Update the knowledge graph based on the reasoning results of the large language model and update the knowledge graph. If the reasoning results output by the large language model conflict with the existing knowledge in the knowledge graph, perform knowledge screening through manual review or confidence-weighted voting mechanism;

[0078] (4) Provide query services through knowledge graphs, call the SPARQL query engine to support complex queries, and provide index optimization based on inverted indexes and graph databases.

[0079] The bidirectional enhancement module is used to read knowledge from the knowledge graph and train the large language model based on the knowledge, write the inference results of the large language model into the knowledge graph, and perform version management on the updates of the knowledge graph and the large language model.

[0080] As a specific implementation of the bidirectional enhancement module, this module is used to perform the following operations:

[0081] (1) When reasoning through a large language model, the knowledge graph is called to obtain background information;

[0082] (2) Update the reasoning results of the large language model to the knowledge graph;

[0083] (3) Injecting the structured knowledge of the knowledge graph into the large language model as the input of the large language model;

[0084] (4) Version management of knowledge graph and large language model updates through real-time update mechanism and version control mechanism.

[0085] The domain adaptation layer is used to train large language models based on lightweight adapters and domain adaptation mechanisms.

[0086] As a specific implementation of the domain adaptation layer, it is used to perform the following operations:

[0087] (1) Add an adapter module to the last layer of the model and use the labeled data of the target domain to fine-tune the parameters;

[0088] (2) Add a domain classifier, calculate the domain adaptation loss, and update the parameters of the adapter module through backpropagation;

[0089] (3) Use unlabeled data in the source domain to pre-train the adapter for self-supervised learning, use labeled data in the target domain to fine-tune the adapter for semi-supervised learning, and optimize the adapter by combining knowledge distillation technology in the source domain.

[0090] The real-time query and reasoning module is used to receive user input, analyze the user input intent through the trained large language model, output candidate answers, verify and supplement the candidate answers based on the knowledge graph to generate initial answers, and generate the final answer based on the causal reasoning network to explain the reasoning path.

[0091] As a specific implementation of the real-time query and reasoning module, this module is used to perform the following operations:

[0092] (1) Analyze user multimodal input and identify user intent;

[0093] (2) Generate candidate answers based on user intent using a large language model;

[0094] (3) Call the knowledge graph to obtain background information, supplement the candidate answers based on user input and background information, and generate a preliminary answer;

[0095] (4) Verify and supplement the preliminary answers based on the knowledge graph, verify the legitimacy of the answers through knowledge graph queries, and supplement missing information;

[0096] (5) Based on the probability of the large language model output and the confidence of the knowledge graph, the confidence of the large language model output and the knowledge in the knowledge graph are evaluated to generate the final answer;

[0097] (6) Construct a causal graph to represent the causal relationship between entities and explain the reasoning path through causal reasoning algorithms.

[0098] The explainability and transparency module is used to display the knowledge in the knowledge graph through a visual interface, record user input, the reasoning process of the large language model, and the update process of the knowledge graph, generate operation logs, and provide audit services based on the operation date.

[0099] As a specific implementation of the Explainability and Transparency module, this module is used to perform the following operations:

[0100] (1) Visualization tools are used to display the reasoning path and attention heat map, and the knowledge graph entities, relationships and weights called by the large language model are displayed in a graphical form. The heat map shows the degree of attention paid by the large language model to different modal features when processing input;

[0101] (2) Record the input of each query, the large language model reasoning process, and the knowledge graph update record, and provide an API for third-party audit systems to call.

[0102] The system of this embodiment uses a bidirectional enhancement mechanism to fully utilize the advantages of both to solve the complex relationship reasoning problem in cross-domain heterogeneous data queries. Specifically, it includes:

[0103] (1) Bidirectional enhancement mechanism: A bidirectional enhancement framework is designed to enable the large language model to dynamically obtain background information from the knowledge graph, and at the same time use the reasoning output of the large language model to update and enrich the knowledge graph, thus forming a closed-loop knowledge flow system;

[0104] (2) Cross-domain data fusion: Develop an efficient cross-domain data fusion algorithm that can build bridges between different domains and data types to achieve effective integration of multimodal data;

[0105] (3) Domain Adaptation Layer: The domain adaptation layer is introduced to quickly migrate to new domains through a small amount of labeled data, reducing the dependence on a large amount of labeled data and improving the generalization ability and applicability of the model;

[0106] (4) Explainability and transparency: Design a series of explainability tools and methods to ensure the reliability and transparency of the system in high-risk areas and enhance user trust;

[0107] (5) Real-time performance and efficiency: Optimize the system architecture to ensure high efficiency and low latency in large-scale data environments to meet the needs of real-time queries;

[0108] (6) Multi-task support: Supports multiple query tasks, including but not limited to relational reasoning, entity linking, attribute prediction, etc., providing users with a full range of services.

[0109] Example 2:

[0110] The present invention provides a cross-domain heterogeneous data query method based on a large model and a knowledge graph, which realizes cross-domain heterogeneous data query through the system disclosed in Example 1. The method includes six operations: data input and preprocessing, knowledge graph construction and management, two-way enhancement, domain adaptation, real-time query and reasoning, and interactive management.

[0111] Data input and preprocessing: Collect multimodal data, perform data input and preprocessing, feature extraction, and feature alignment operations on the data to obtain feature vectors.

[0112] As a specific implementation of data input and preprocessing, perform the following operations:

[0113] (1) Collect multimodal data through the input interface, including text, images, audio and video;

[0114] (2) Preprocess multimodal data, extract the content of text data, convert the format of image, audio, and video data, and extract images and audio from video data;

[0115] (3) For pre-processed text data, word embedding is performed using a pre-trained language model, and syntactic analysis is performed in combination with a domain dictionary to identify entities in the text and the relationships between entities. The word embedding vector, dependency syntactic structure, and entity and relationship lists of the text data are output;

[0116] (4) For the pre-processed data of the image type, feature extraction is performed through a convolutional neural network to obtain visual features, and facial expression and gesture recognition is performed in combination with a posture estimation model to obtain key point coordinates and action sequences, and the visual feature vector, key point coordinates and action sequence of the image type data are output;

[0117] (5) For the pre-processed audio data, the voiceprint features are extracted through spectral analysis, and the acoustic features including intonation and pauses in the audio are analyzed in combination with the emotional speech recognition model, and the emotional label and acoustic feature vector are output;

[0118] (6) The word embedding vector, visual feature vector, and voiceprint feature vector are mapped to a unified space by maximizing the inter-modal similarity and minimizing the intra-modal difference. The maximum mean difference is used to constrain the inter-modal distribution difference for feature alignment to obtain the final feature vector.

[0119] Knowledge graph construction and management: Extract entities, relationships, and attributes based on the feature vectors output by the data input and preprocessing modules, and construct a knowledge graph using entities, relationships, and attributes as knowledge. Dynamically update the knowledge graph based on the inference results output by the large language model, and provide query services based on the knowledge graph.

[0120] As a specific implementation of the knowledge graph construction and management module, perform the following operations:

[0121] (1) Based on the feature vectors output by the data input and preprocessing module, entities are extracted through the domain ontology library and entity recognition model. Based on the feature vectors output by the data input and preprocessing module, relationships are extracted through the relationship classification model. The legitimacy of the relationships is verified by the rule engine, and attributes are filled based on the inference results output by the large language model.

[0122] (2) Constructing knowledge graphs by taking entities, relationships, and attributes as knowledge;

[0123] (3) Update the knowledge graph based on the reasoning results of the large language model and update the knowledge graph. If the reasoning results output by the large language model conflict with the existing knowledge in the knowledge graph, perform knowledge screening through manual review or confidence-weighted voting mechanism;

[0124] (4) Provide query services through knowledge graphs, call the SPARQL query engine to support complex queries, and provide index optimization based on inverted indexes and graph databases.

[0125] Bidirectional enhancement: Read knowledge from the knowledge graph, train the large language model based on the knowledge, write the inference results of the large language model into the knowledge graph, and perform version management on the updates of the knowledge graph and the large language model.

[0126] As a specific implementation of bidirectional enhancement, perform the following operations:

[0127] (1) When reasoning through a large language model, the knowledge graph is called to obtain background information;

[0128] (2) Update the reasoning results of the large language model to the knowledge graph;

[0129] (3) Injecting the structured knowledge of the knowledge graph into the large language model as the input of the large language model;

[0130] (4) Version management of knowledge graph and large language model updates through real-time update mechanism and version control mechanism.

[0131] Domain Adaptation: Model training for large language models based on lightweight adapters and domain adaptation mechanisms.

[0132] As a specific implementation of domain adaptation, perform the following operations:

[0133] (1) Add an adapter module to the last layer of the model and use the labeled data of the target domain to fine-tune the parameters;

[0134] (2) Add a domain classifier, calculate the domain adaptation loss, and update the parameters of the adapter module through backpropagation;

[0135] (3) Use unlabeled data in the source domain to pre-train the adapter for self-supervised learning, use labeled data in the target domain to fine-tune the adapter for semi-supervised learning, and optimize the adapter by combining knowledge distillation technology in the source domain.

[0136] Real-time query and reasoning: Receive user input, analyze the user input intent through the trained large language model, output candidate answers, verify and supplement the candidate answers based on the knowledge graph to generate an initial answer, and generate the final answer based on the causal reasoning network to explain the reasoning path.

[0137] As a specific implementation of real-time query and reasoning, perform the following operations:

[0138] (1) Analyze user multimodal input and identify user intent;

[0139] (2) Generate candidate answers based on user intent using a large language model;

[0140] (3) Call the knowledge graph to obtain background information, supplement the candidate answers based on user input and background information, and generate a preliminary answer;

[0141] (4) Verify and supplement the preliminary answers based on the knowledge graph, verify the legitimacy of the answers through knowledge graph queries, and supplement missing information;

[0142] (5) Based on the probability of the large language model output and the confidence of the knowledge graph, the confidence of the large language model output and the knowledge in the knowledge graph are evaluated to generate the final answer;

[0143] (6) Construct a causal graph to represent the causal relationship between entities and explain the reasoning path through causal reasoning algorithms.

[0144] Interaction management: Display the knowledge in the knowledge graph through a visual interface, record user input, the reasoning process of the large language model, and the update process of the knowledge graph, generate operation logs, and provide audit services based on the operation date.

[0145] As a concrete implementation of explainability and transparency, the following operations are performed:

[0146] (1) Visualization tools are used to display the reasoning path and attention heat map, and the knowledge graph entities, relationships and weights called by the large language model are displayed in a graphical form. The heat map shows the degree of attention paid by the large language model to different modal features when processing input;

[0147] (2) Record the input of each query, the large language model reasoning process, and the knowledge graph update record, and provide an API for third-party audit systems to call.

[0148] The method of this embodiment can perform cross-domain heterogeneous data query based on the system disclosed in Example 1.

[0149] The above is a detailed introduction to the cross-domain heterogeneous data query system and method based on a large model and knowledge graph provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A cross-domain heterogeneous data query system based on a large model and knowledge graph, characterized by: It includes data input and preprocessing module, knowledge graph construction and management module, bidirectional enhancement module, domain adaptation layer, real-time query and reasoning module, and explainability and transparency module; The data input and preprocessing module is used to collect multimodal data, perform data input and preprocessing, feature extraction and feature alignment operations on the data, and obtain feature vectors; The knowledge graph construction and management module is used to extract entities, relationships and attributes based on the feature vectors output by the data input and preprocessing module, and to construct a knowledge graph using entities, relationships and attributes as knowledge, to dynamically update the knowledge graph based on the inference results output by the large language model, and to provide query services based on the knowledge graph; The bidirectional enhancement module is used to read knowledge from the knowledge graph and perform model training on the large language model based on the knowledge, to write the reasoning results of the large language model into the knowledge graph, and to perform version management on the updates of the knowledge graph and the large language model; The domain adaptation layer is used to perform model training on the large language model based on the lightweight adapter and the domain adaptation mechanism; The real-time query and reasoning module is used to receive user input, analyze the user input intent through the trained large language model, output candidate answers, verify and supplement the candidate answers based on the knowledge graph to generate an initial answer, and generate a final answer based on the causal reasoning network to interpret the reasoning path; The explainability and transparency module is used to display the knowledge in the knowledge graph through a visual interface, and to record user input, the reasoning process of the large language model, and the update process of the knowledge graph, generate operation logs, and provide audit services based on the operation date.

2. The cross-domain heterogeneous data query system based on a large model and knowledge graph according to claim 1 is characterized in that: The data input and preprocessing module is used to perform the following operations: Collect multimodal data through input interfaces, including text, images, audio, and video; Perform data preprocessing on multimodal data, extract the content of text data, convert the format of image, audio and video data, and extract images and audio from video data; For pre-processed text data, we use a pre-trained language model to perform word embedding and combine it with a domain dictionary for syntactic analysis to identify entities in the text and the relationships between them. We then output the word embedding vectors, dependency syntactic structure, and entity and relationship lists for the text data. For pre-processed image data, feature extraction is performed through a convolutional neural network to obtain visual features. Facial expressions and gestures are then recognized using a posture estimation model to obtain key point coordinates and action sequences. The visual feature vector, key point coordinates, and action sequence of the image data are then output. For pre-processed audio data, we extract voiceprint features through spectral analysis and combine it with the emotional speech recognition model to analyze the acoustic features of the audio, including intonation and pauses, and output the emotional label and acoustic feature vector. By maximizing the inter-modal similarity and minimizing the intra-modal difference, the word embedding vector, visual feature vector and voiceprint feature vector are mapped to a unified space. The maximum mean difference is used to constrain the inter-modal distribution difference for feature alignment to obtain the final feature vector.

3. The cross-domain heterogeneous data query system based on a large model and knowledge graph according to claim 1 is characterized in that: The knowledge graph construction and management module is used to perform the following operations: Based on the feature vectors output by the data input and preprocessing module, entities are extracted through the domain ontology library and entity recognition model. Based on the feature vectors output by the data input and preprocessing module, relationships are extracted through the relationship classification model. The legitimacy of the relationships is verified by the rule engine, and attributes are filled based on the inference results output by the large language model. Build a knowledge graph using entities, relationships, and attributes as knowledge; Update the knowledge graph based on the inference results of the large language model. If the inference results output by the large language model conflict with the existing knowledge in the knowledge graph, perform knowledge screening through manual review or confidence-weighted voting mechanism. It provides query services through knowledge graphs, calls the SPARQL query engine to support complex queries, and provides index optimization based on inverted indexes and graph databases.

4. The cross-domain heterogeneous data query system based on a large model and knowledge graph according to claim 1 is characterized in that: The bidirectional enhancement module is used to perform the following operations: When reasoning through a large language model, the knowledge graph is called to obtain background information; Update the inference results of the large language model to the knowledge graph; Inject the structured knowledge of the knowledge graph into the large language model as the input of the large language model; Updates to knowledge graphs and large language models are managed through real-time update mechanisms and version control mechanisms.

5. The cross-domain heterogeneous data query system based on a large model and knowledge graph according to claim 1 is characterized in that: The domain adaptation layer is used to perform the following operations: Add an adapter module to the last layer of the model and fine-tune the parameters using labeled data from the target domain; Add a domain classifier, calculate the domain adaptation loss, and update the parameters of the adapter module through backpropagation; We use unlabeled data in the source domain to pre-train the adapter through self-supervised learning, and use labeled data in the target domain to fine-tune the adapter through semi-supervised learning, and then optimize the adapter by combining knowledge distillation techniques from the source domain.

6. The cross-domain heterogeneous data query system based on a large model and knowledge graph according to claim 1 is characterized in that: The real-time query and reasoning module is used to perform the following operations: Parse user multimodal input and identify user intent; Based on user intent, candidate answers are generated through a large language model; Call the knowledge graph to obtain background information, supplement the candidate answers based on user input and background information, and generate preliminary answers; Verify and supplement the preliminary answers based on the knowledge graph, verify the legitimacy of the answers through knowledge graph queries, and supplement missing information; Based on the probability of the large language model output and the confidence of the knowledge graph, the confidence of the large language model output and the knowledge in the knowledge graph are evaluated to generate the final answer; Construct a causal graph to represent the causal relationship between entities, and explain the reasoning path through causal reasoning algorithms.

7. The cross-domain heterogeneous data query system based on a large model and knowledge graph according to claim 1 is characterized in that: The Explainability and Transparency module is used to perform the following operations: Visualization tools are used to display reasoning paths and attention heat maps, graphically displaying the knowledge graph entities, relationships, and weights used by the large language model. Heat maps are used to show the degree to which the large language model focuses on different modal features when processing input. Record the input of each query, the large language model inference process, and the knowledge graph update record, and provide an API for third-party audit systems to call.

8. A cross-domain heterogeneous data query method based on a large model and knowledge graph, characterized by: A method for implementing cross-domain heterogeneous data query by a cross-domain heterogeneous data query system based on a large model and a knowledge graph as described in any one of claims 1 to 7, the method comprising the following steps: Data input and preprocessing: Collect multimodal data, perform data input and preprocessing, feature extraction and feature alignment operations on the data to obtain feature vectors; Knowledge graph construction and management: Extract entities, relationships, and attributes based on data input and feature vectors output by the preprocessing module, and construct a knowledge graph using entities, relationships, and attributes as knowledge. Dynamically update the knowledge graph based on the inference results output by the large language model, and provide query services based on the knowledge graph. Bidirectional enhancement: Read knowledge from the knowledge graph, train the large language model based on the knowledge, write the inference results of the large language model into the knowledge graph, and manage the versioning of the knowledge graph and the large language model. Domain Adaptation: Training large language models based on lightweight adapters and domain adaptation mechanisms; Real-time query and reasoning: Receive user input, analyze the user's input intent using a trained large language model, output candidate answers, verify and supplement the candidate answers based on the knowledge graph to generate an initial answer, and generate the final answer by interpreting the reasoning path based on the causal reasoning network. Interaction management: Display the knowledge in the knowledge graph through a visual interface, record user input, the reasoning process of the large language model, and the update process of the knowledge graph, generate operation logs, and provide audit services based on the operation date.

Images