A large language model knowledge question answering method and system integrating multimodal knowledge graph

Abstract

The present invention proposes a large language model knowledge question and answer method and system integrating multimodal knowledge graphs, which belongs to the technical field of artificial intelligence equipment, and includes: acquiring multimodal data, constructing a multimodal knowledge graph, constructing a multimodal large model, constructing a dynamic update mechanism, acquiring user questions, finding relevant text fragments from a professional knowledge base, obtaining answers to professional questions, obtaining triple information, finding knowledge nodes directly related to the questions, and acquiring multimodal resources matching the knowledge nodes; performing correlation analysis and logical combing on the multimodal resources and relevant text fragments to obtain integrated information, processing to obtain the final answer, and transmitting it to the user; not only can accurate and comprehensive answers be provided, but also a more intuitive and easy-to-understand knowledge acquisition experience is provided to users by integrating multimedia elements, which greatly improves user satisfaction and the practicality of the question and answer system.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a large language model knowledge question answering method and system integrating multimodal knowledge graphs. Background Art

[0002] In the field of natural language processing, large language models have attracted much attention for their powerful text understanding and generation capabilities. However, the information reliability issues caused by their "black box" characteristics have become a bottleneck restricting their application, especially in scenarios involving precise facts and professional knowledge. To solve this problem, the industry has proposed a method that combines the retrieval-augmented generation (RAG) strategy with large language models, introducing external knowledge bases to improve accuracy. However, the strict requirements of industry-level applications for information accuracy make models that rely solely on RAG strategies appear stretched. At this time, knowledge graphs, as a structured knowledge representation method, provide strong support for intelligent question and answering with their clear entity, attribute, and relationship triples, as well as high interpretability and flexibility.

[0003] With the enrichment of multimodal data and the advancement of technology, the combination of multimodal data processing and knowledge graph technology has brought new breakthroughs to question-answering systems. Multimodal data can describe the real world more comprehensively, and its application in knowledge graphs not only enriches the content, but also improves accuracy and completeness.

[0004] Developing a large-scale language model question-answering system that integrates multimodal knowledge graphs is crucial to improving question-answering performance. The system integrates multimodal data such as text, images, and voice to provide large-scale language models with accurate and rich domain knowledge, enhance their understanding and generation capabilities, and thus achieve continuous discovery and integration of knowledge. This innovative technology has broad application prospects in high-precision demand fields such as finance, medical care, manufacturing, and supply chain management. It can effectively improve information processing accuracy, optimize decision-making processes, enhance business operation efficiency, and promote corporate innovation to meet information needs in complex scenarios. Summary of the invention

[0005] The present invention aims at solving the problems in the prior art, and the technical solution adopted in this application is as follows:

[0006] In a first aspect, the present invention provides a large language model knowledge question answering method integrating a multimodal knowledge graph, comprising:

[0007] Acquire multimodal data, convert it into vector representation, and store it in the professional knowledge base; extract structured information from multimodal data to build a multimodal knowledge graph; build a multimodal large model, which is obtained by fine-tuning a general large-scale pre-trained language model; build a dynamic update mechanism to update the professional knowledge base with the new data obtained;

[0008] Obtain user questions, analyze user questions to obtain user intent, optimize the original questions based on user intent to obtain user optimization questions; convert user optimization questions into vectors through a pre-trained word vector model;

[0009] Receive the vectorized user optimization problem and find relevant text fragments from the professional knowledge base;

[0010] Use user optimization questions and related text snippets as input to the multimodal large model to obtain professional answers to questions;

[0011] Send the answers to professional questions to the information extraction module to obtain triple information;

[0012] Taking the triple information as the core of the query vector, we conduct a matching search in the multimodal knowledge graph based on the query vector to find the knowledge nodes directly related to the question and obtain the multimodal resources matching the knowledge nodes.

[0013] Conduct correlation analysis and logical sorting of multimodal resources and related text fragments to obtain integrated information, and input the integrated information into the multimodal macro model;

[0014] The multimodal large model processes the integrated information to obtain the final answer and transmits it to the user.

[0015] Furthermore, the multimodal data includes text, images, audio and video; data cleaning technology is applied to refine the data, and an embedding layer is designed to convert various modal data into a unified vector representation; for text data, text parsing and text segmentation are performed, and the text is converted into a vector through a pre-trained word vector model, and a vector index is constructed through an encoder, and the processed document information, vector data and index are stored in a database; for image data, video data and audio data, they are converted into vectors through an embedding model and stored in a database.

[0016] Furthermore, the process of constructing a multimodal knowledge graph includes using entity recognition and relationship extraction techniques to extract structured information from multimodal data and construct a knowledge graph. At the same time, multimodal data processing technology is used to integrate non-text features such as images and audio to enrich the content of the knowledge graph.

[0017] Furthermore, the information extraction module includes a text information extraction submodule, an image information extraction submodule, a video information extraction submodule, an audio information extraction submodule and a cross-modal alignment module.

[0018] Furthermore, the text information extraction submodule is used to identify specific types of information in the text; the image information extraction submodule is used to identify objects and scenes in the image, convert them into labels that can be indexed and queried, and analyze the relationship between objects in the image.

[0019] Furthermore, the video information extraction submodule is used to extract image information from each frame of the video to capture key information in dynamic changes; the audio information extraction submodule is used to convert audio into text; and the cross-modal alignment module is used to align and integrate the information extracted from different modalities to form a unified knowledge representation.

[0020] Furthermore, the multimodal resources are associated with related text fragments and logically sorted out, and processed using a collaborative enhancement framework. The collaborative enhancement framework integrates RAG, LLM and multimodal knowledge graph technologies. Under the collaborative enhancement framework, the user's original question is first split into multiple sub-questions, and each sub-question conducts precise queries on a specific field in the professional knowledge base; then, using the generation capability of LLM, the retrieved knowledge and the partial answers generated by LLM are organically integrated to form a complete answer; the multimodal knowledge graph is used to verify and calibrate the answer to ensure that each knowledge triple in the answer has a reliable source and basis.

[0021] In a second aspect, the present invention provides a large language model knowledge question answering system integrating a multimodal knowledge graph, comprising:

[0022] The data processing module is configured to: obtain multimodal data, convert the multimodal data into vector representation, and store it in the professional knowledge base; extract structured information from the multimodal data to build a multimodal knowledge graph; build a multimodal large model, which is obtained by fine-tuning a general large-scale pre-trained language model; build a dynamic update mechanism to update the professional knowledge base using the new data obtained;

[0023] The user question processing module is configured to: obtain user questions, analyze user questions to obtain user intentions, optimize the original questions based on user intentions to obtain user optimization questions; and convert the user optimization questions into vectors through a pre-trained word vector model;

[0024] The relevant text acquisition module is configured to: receive the vectorized user optimization problem and find relevant text fragments from the professional knowledge base;

[0025] The professional question answer acquisition module is configured to: use the user's optimized question and related text fragments as inputs of the multimodal large model to obtain the professional question answer;

[0026] The triple information acquisition module is configured to: send the answers to the professional questions to the information extraction module to obtain triple information;

[0027] The multimodal resource acquisition module is configured to: use the triple information as the core of the query vector, perform matching search in the multimodal knowledge graph according to the query vector, find the knowledge node directly related to the question, and obtain the multimodal resources matching the knowledge node;

[0028] The information integration module is configured to: perform correlation analysis and logical sorting of multimodal resources and related text fragments to obtain integrated information, and input the integrated information into the multimodal macro model;

[0029] The answer output module is configured as follows: the multimodal large model processes the integrated information to obtain the final answer and transmits it to the user.

[0030] In a third aspect, the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model knowledge question and answer method that integrates multimodal knowledge graphs as described in the first aspect.

[0031] In a fourth aspect, the present invention provides an electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the large language model knowledge question and answer method integrating a multimodal knowledge graph as described in the first aspect.

[0032] Compared with the prior art, the advantages and positive effects of the present invention are:

[0033] 1. The present invention provides a large language model knowledge question answering method and system that integrates multimodal knowledge graphs. By combining the generalization ability of the large language model and the rich information of the multimodal knowledge graph, the performance of the model in knowledge question answering tasks is significantly enhanced. It not only broadens the knowledge scope of the large language model, but also improves its ability to understand and answer complex questions. It has important practical significance and application value for promoting the development of the field of artificial intelligence.

[0034] 2. This invention successfully integrates large language models, retrieval enhancement generation technology and multimodal knowledge graphs by introducing an innovative collaborative enhancement framework (CogRLMG). It provides an effective solution to the problems of insufficient reliability, interpretability and cross-modal understanding of large language models in knowledge question and answer. It is particularly suitable for complex application scenarios that require extremely high information accuracy and need to integrate multimodal knowledge. It not only provides accurate and comprehensive answers, but also integrates multimedia elements to provide users with a more intuitive and easy-to-understand knowledge acquisition experience, greatly improving user satisfaction and the practicality of the question and answer system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 It is a flowchart of the large language model knowledge question answering system integrating multimodal knowledge graph of the present invention. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0039] Embodiment 1, as Figure 1 As shown, this application provides a large language model knowledge question answering method that integrates multimodal knowledge graphs. By integrating large language models, retrieval enhancement generation, multimodal data processing technology and knowledge graphs, it aims to solve the problem of large language models lacking reliability, interpretability and cross-modal understanding ability in knowledge question answering in the prior art. The steps include: S1, preparation stage; S11, professional knowledge base construction; S12, multimodal knowledge graph construction; S13, model fine-tuning and dynamic update;

[0040] S2, user question input and analysis;

[0041] S3, professional knowledge base information retrieval;

[0042] S4, multimodal large model professional question and answer generation;

[0043] S5, cross-modal triple information extraction;

[0044] S6, matching multimodal knowledge graph;

[0045] S7, output matching node data;

[0046] S8. Output the final result.

[0047] In step S11, professional knowledge in a specific field is collected and organized, including multimodal data such as text, images, audio, and video. The specific technical field includes medical image analysis. Data cleaning technology is applied to refine the data, and an embedding layer is designed to convert various modal data into a unified vector representation.

[0048] For text data, we perform text parsing and text segmentation. We convert text into vectors through a pre-trained word vector model, build vector indexes through an encoder, and store the processed document information, vector data, and indexes in a database. For image data, video data, and audio data, we also convert them into vectors through an embedding model and store them in a database. For image data, we use a pre-trained ResNet50 network to feed the pre-processed images into the network, and select its top-level output as the vector representation of the visual features. For video data, we extract key frames and use the same processing method as for image data to convert each frame in the video into a vector representation. For audio data, we use the VGGish model to convert audio signals into recognizable vector features.

[0049] In step S12, based on the above professional knowledge base, entity recognition and relationship extraction techniques are used to extract structured information from multimodal data to construct a knowledge graph. At the same time, non-text features such as images and audio are integrated through multimodal data processing technology to enrich the content of the knowledge graph.

[0050] Specifically, the knowledge extraction task first performs entity extraction to complete the extraction of key semantics in different modal data, determine keyword pairs, complete ontology construction, and then perform relationship extraction based on business logic to complete the relationship construction between single modalities.

[0051] Specifically, the present invention designs a multimodal knowledge graph entity extraction module, the core of which includes an embedding module and an attention module. The embedding module converts different modal data into embedding vectors according to different input data types.

[0052] Text knowledge representation: Focusing on the classic (h, r, t) triple form, each entity and relationship is mapped to a high-dimensional vector space through its unique ID to achieve implicit expression of information. In specific implementation, the ID of the entity or relationship is used as input and converted into a dense vector using the embedding layer. This conversion mechanism is essentially a matrix multiplication operation of the one-hot encoding of each ID and the embedding matrix, thereby obtaining a vector representation with rich semantics. For image and audio data, through pre-trained image recognition and speech recognition models, it is converted into text descriptions for entity recognition and relationship extraction;

[0053] The attention module adopts multi-head self-attention and inter-modal attention strategies. The former captures complex feature interactions within a single modality, and the latter coordinates the information proportions between different modalities and dynamically adjusts the weights to highlight key information, jointly promoting the accuracy and efficiency of knowledge graph construction and achieving deep fusion and intelligent extraction of multi-source heterogeneous data.

[0054] Specifically, a knowledge graph automatic construction and update mechanism is designed, and a continuously running data monitoring system is deployed, which is connected to external data sources to monitor and capture new incoming data. Once the new data is captured, it is immediately cleaned and formatted by the preprocessing module. This includes data denoising, normalization, and standardization, as well as decoding and preprocessing of multimedia data to ensure that the new data meets the input requirements of the knowledge graph. New entities and entity types are automatically identified for multimodal data in the new data. The contextual relationship between entities is analyzed through a graph neural network (GNN), and the relationship between new entities and existing knowledge graph entities is automatically inferred and established. This includes relationship prediction based on features such as co-occurrence frequency and context similarity. For the identified new entities and relationships, new nodes and edges are inserted immediately to form an incremental graph. New data nodes in modalities such as text, image, and audio are associated with their corresponding multimodal information to achieve cross-modal knowledge fusion. An automatic evaluation mechanism is introduced to regularly check the accuracy and completeness of newly added knowledge. A user feedback system is established to allow users to evaluate or correct the content of the knowledge graph. These feedback data are used for iterative training and optimization of the model, forming a closed-loop self-evolution process.

[0055] In step S13, a multimodal large model is constructed, and a general large-scale pre-trained language model is selected for fine-tuning to adapt it to specific fields and multimodal knowledge graphs. A dynamic update mechanism is established to ensure that the professional knowledge base is continuously updated with new data.

[0056] Among them, a general large-scale pre-trained language model is selected for fine-tuning, specifically, the Qwen-VL model is selected as the base model. Use the fine-tuning script provided in the official project to perform LoRA fine-tuning tests, load the Qwen-VL model, and organize the new data set, dividing it into a training set and a validation set. Next, configure the fine-tuning parameters, including determining the learning rate (for example, from 1e-5 to 1e-3), batch size (based on the available GPU memory size, such as 8 or 16), number of iterations, and evaluation frequency. Then, use the official fine-tuning script to load the model and apply LoRA, define the optimizer and loss function. In each training epoch, iterate the batches in the dataset, perform forward propagation, calculate losses, backpropagation, and weight updates. Evaluate performance on the validation set regularly, and adjust hyperparameters such as learning rate and batch size based on feedback to optimize the model. Finally, evaluate the final performance of the selected model on an independent test set.

[0057] The dynamic update mechanism specifically includes:

[0058] A. Data collection and preprocessing,Set up an automated system to regularly collect new data from various sources (such as social media, professional literature, news reports, etc.).Preprocess the collected data, including deduplication, noise removal, data standardization, and multimodal data annotation.

[0059] B. Knowledge graph integration: Integrate entities, relationships and other information in new data into the existing multimodal knowledge graph.

[0060] C. Model fine-tuning: Use newly collected data to incrementally train the model, incorporate the knowledge of the new data into the model, and use LoRA technology to update only some of the model's weights.

[0061] D. Performance evaluation and verification: Evaluate the performance of the model on new data in real time or regularly, deploy old and new versions of the model in the actual environment, and compare their performance to decide whether to adopt the new update.

[0062] E. Update strategy, regular updates: Set a fixed period (such as weekly or monthly) to update the model to ensure that the model always reflects the latest data and knowledge.

[0063] Threshold-triggered update: When the model performance drops to a certain threshold, the update process is triggered.

[0064] In step S2, the user question input is obtained, the user input question is analyzed, and the keywords, syntactic structure and potential intent of the question are identified. Based on the in-depth understanding of the user's intent, the original question is optimized or rewritten to improve the accuracy and efficiency of the retrieval. This includes correcting grammatical errors, eliminating ambiguity, standardizing expressions, and adding necessary contextual information. The question is converted into a vector through a pre-trained word vector model; the user asks a question in natural language, which can be a query about a specific knowledge point or a complex question that requires cross-modal understanding.

[0065] In step S3, after receiving the vectorized user question, the text segment most relevant to the question raised by the user is found from the knowledge base.

[0066] The specific steps of finding the relevant text fragments include: S31, using cosine similarity to measure the similarity between the user query vector and each document vector in the knowledge base;

[0067] S32, vector database search: using the search function of the vector database to find the text segment most similar to the user query;

[0068] S33, sorting the results according to the similarity scores;

[0069] S34. Return results: Return the text segments with the highest scores as the most relevant answer candidates.

[0070] In step S4, the questions raised by the user in S2 and the relevant text paragraphs retrieved in S3 are used as input and sent to the exclusive multimodal large model that has been fine-tuned in the professional field in stage S13. This large model not only understands text, but also integrates the understanding capabilities of modalities such as images and audio, and can comprehensively analyze the relevance of text information and other modal data. Based on the contextual understanding capabilities and domain knowledge of deep learning, the model generates professional questions and answers that are not only accurate but also rich in domain characteristics.

[0071] In step S5, the professional answer generated in S4 is fed into the information extraction module. This module not only analyzes the text, but also uses advanced computer vision and speech recognition technology to process image, video, audio and other modal data, and automatically extracts subject-relation-object triple information from them.

[0072] Specifically, the information extraction module includes a text information extraction submodule, an image information extraction submodule, a video information extraction submodule, an audio information extraction submodule; a cross-modal alignment module;

[0073] The text information extraction submodule identifies specific types of information in the text, such as names of people, places, and organizations. On the basis of identifying entities, it further analyzes the semantic relationships between them, such as "someone was born in a certain place" and "Company A acquired Company B". These relationships constitute the "relationship" part of the triple. The image information extraction submodule identifies objects and scenes in the image, converts them into labels that can be indexed and queried, and analyzes the relationship between objects in the image, such as "person-holding-cup". The video information extraction submodule extracts image information from each frame of the video to capture key information in dynamic changes. It identifies the action and event sequence in the video, such as "person A walks towards vehicle B and drives away", forming a relationship extraction on the time series. The audio information extraction submodule converts audio into text and then calls the text information extraction submodule. The cross-modal alignment module aligns and integrates the information extracted from different modalities, such as matching text descriptions with objects in the image to form a unified knowledge representation.

[0074] In step S6, a refined multimodal search strategy is adopted. First, based on the answers to the professional questions extracted from the original question and step S5, we accurately extracted a series of key triples of information, which constitute the core of the query vector. Subsequently, we applied these query vectors to the multimodal knowledge graph and performed matching searches through precise algorithms, aiming to find knowledge nodes directly related to the question. Once the match is successful, the system not only automatically collects text descriptions closely related to the question, but also further mines multimedia resources such as images and short video clips that match these text descriptions. These multimedia resources not only enrich the expression of the answer, but also improve the intuitiveness and comprehension of the information. In order to ensure the accuracy and relevance of the information obtained by the user, the system also preliminarily screens and sorts these multimedia resources to ensure that the user can first access the most valuable and relevant information.

[0075] In step S7, the multimodal resources collected in step S6 are further deeply integrated with the detailed information retrieved from the professional knowledge base. This integration is not limited to the splicing of text, but also includes the correlation analysis and logical sorting of multimedia resources and text information. The integrated information is re-entered into the exclusive multimodal large model that has been fine-tuned in the professional field in stage S13.

[0076] In terms of specific implementation, a CogRLMG collaborative augmentation framework (Collaborative Augmentation between RAG, LLM and MMKG) is proposed, which integrates RAG, LLM and multi-modal knowledge graph (Multi-modal Knowledge Graph, MMKG) technology. Under the CogRLMG framework, the user's original question is first split into multiple sub-questions, each of which accurately queries a specific field in the professional knowledge base. Then, using the generation capability of LLM, the retrieved knowledge is organically integrated with the partial answers generated by LLM to form a complete answer. In order to further ensure the accuracy and completeness of the answer, the multi-modal knowledge graph is used to verify and calibrate the answer to ensure that each knowledge triple in the answer has a reliable source and basis.

[0077] Specifically, the CogRLMG framework can explicitly identify and decompose knowledge triples that do not exist in the knowledge graph. When the knowledge graph cannot directly find enough information to support the completeness of the answer, the framework initiates an active learning mechanism to guide the LLM to extract information from its intrinsic knowledge to complete the missing knowledge triples. When the knowledge graph cannot support the question-answering process, the model decomposes the knowledge and calls itself to complete the knowledge before providing the final answer.

[0078] In step S8, based on the information processed in S7, the large language model outputs the final answer. This answer not only includes text descriptions, but also may incorporate multimedia elements such as relevant pictures and video links, providing users with a more intuitive, easy-to-understand and informative answer.

[0079] This embodiment significantly enhances the performance of the model in knowledge question-answering tasks by combining the generalization ability of the large language model and the rich information of the multimodal knowledge graph. It not only broadens the knowledge scope of the large language model, but also improves its ability to understand and answer complex questions. It has important practical significance and application value for promoting the development of the field of artificial intelligence.

[0080] Embodiment 2: This embodiment provides a large language model knowledge question answering system integrating multimodal knowledge graphs, including:

[0081] The data processing module is configured to: obtain multimodal data, convert the multimodal data into vector representation, and store it in the professional knowledge base; extract structured information from the multimodal data to build a multimodal knowledge graph; build a multimodal large model, which is obtained by fine-tuning a general large-scale pre-trained language model; build a dynamic update mechanism to update the professional knowledge base using the new data obtained;

[0082] The user question processing module is configured to: obtain user questions, analyze user questions to obtain user intentions, optimize the original questions based on user intentions to obtain user optimization questions; and convert the user optimization questions into vectors through a pre-trained word vector model;

[0083] The relevant text acquisition module is configured to: receive the vectorized user optimization problem and find relevant text fragments from the professional knowledge base;

[0084] The professional question answer acquisition module is configured to: use the user's optimized question and related text fragments as inputs of the multimodal large model to obtain the professional question answer;

[0085] The triple information acquisition module is configured to: send the answers to the professional questions to the information extraction module to obtain triple information;

[0086] The multimodal resource acquisition module is configured to: use the triple information as the core of the query vector, perform matching search in the multimodal knowledge graph according to the query vector, find the knowledge node directly related to the question, and obtain the multimodal resources matching the knowledge node;

[0087] The information integration module is configured to: perform correlation analysis and logical sorting of multimodal resources and related text fragments to obtain integrated information, and input the integrated information into the multimodal macro model;

[0088] The answer output module is configured as follows: the multimodal large model processes the integrated information to obtain the final answer and transmits it to the user.

[0089] Specifically, the working method of the large language model knowledge question answering system takes the field of medical image analysis technology as an example:

[0090] S1. Preparation stage:

[0091] S11, professional knowledge base construction, collects and organizes professional knowledge from sources such as medical literature, electronic health records, and medical imaging datasets, and converts it into a structured form;

[0092] Text data: collect text data such as medical literature and medical records, perform data cleaning and preprocessing, and then use the pre-trained word vector model to convert the text into vector representation;

[0093] Image data: collect medical images such as X-rays, CT scans, and MRIs, and use a pre-trained ResNet50 network to extract image features;

[0094] Video data: Collect surgical videos or other related videos, extract key frames, and process them as image data;

[0095] Audio data: Collect audio materials such as doctors’ verbal explanations or meeting recordings, transcribe them into text using speech recognition technology, and then convert them into vector representations.

[0096] S12. Multimodal knowledge graph construction: Build a multimodal knowledge graph that contains entities (diseases, organs, treatments, etc.), attributes, and the relationships between them;

[0097] Entity recognition: Use named entity recognition technology to extract entities such as disease names, organ names, treatment methods, etc. from the text;

[0098] Relationship extraction: Use natural language processing technology to extract relationships between entities from text, such as "disease-symptom", "organ-disease", etc.

[0099] Multimodal fusion: Map data vectors of different modalities such as text and images into a unified vector space to establish associations between entity and modal data.

[0100] S13, model fine-tuning and dynamic update:

[0101] Model fine-tuning: Fine-tune the pre-trained model using domain-specific datasets;

[0102] Dynamic update: As new data is added, the professional knowledge base and knowledge graph are continuously updated.

[0103] S2. User question input and analysis:

[0104] Question parsing: When a user asks a question about a symptom or disease, the system performs semantic parsing to understand the user's intention.

[0105] As an implementation, suppose a doctor uploads a lung X-ray and asks if it shows signs of pneumonia.

[0106] Doctor uploads X-ray: The doctor uploads a lung X-ray through the interface provided by the system.

[0107] Ask a question: The doctor asks a side question: "Are there any signs of pneumonia on this X-ray?"

[0108] Question analysis: The system analyzes the doctor's question and understands that what the doctor wants to know is whether there are symptoms of pneumonia in this X-ray.

[0109] S3, Professional knowledge base information retrieval,Information retrieval: Based on the user's questions, relevant text, images and other information are retrieved from the professional knowledge base.

[0110] Specifically, text information retrieval: retrieve text materials related to pneumonia from professional knowledge bases, such as medical literature, case reports, etc.

[0111] Image data retrieval: Retrieve X-ray samples marked as pneumonia from the image database for comparison and analysis.

[0112] Retrieval strategy: Use keyword matching, semantic similarity calculation and other methods to filter out the most relevant information.

[0113] S4. Multimodal large model professional question and answer generation, answer generation: Use multimodal large models to generate professional answers, including text explanations, recommended image analysis, etc.

[0114] Specifically, image analysis: Use pre-trained deep learning models to analyze uploaded X-rays, extract features and identify potential abnormal areas.

[0115] Text information integration: Combine the retrieved text materials to systematically integrate relevant medical knowledge, such as typical symptoms and common locations of pneumonia.

[0116] Generate answers: Combine image analysis results and text information to generate a detailed diagnostic recommendation, including the judgment and basis for the signs of pneumonia in the X-ray.

[0117] S5. Cross-modal triple information extraction. Triple extraction: extract entities and their relations from question-answer interactions to form cross-modal triples (entity 1, relation, entity 2).

[0118] Specifically, entity recognition: using named entity recognition technology to identify entities from questions and answers, such as "pneumonia", "X-ray", etc.;

[0119] Relation extraction: Use relation extraction techniques to identify relationships between entities, such as "pneumonia-appears-on-X-rays";

[0120] Triple construction: The identified entities and relations are constructed into triple form.

[0121] S6, matching multimodal knowledge graph, graph matching: matching the extracted triple information with the existing information in the knowledge graph;

[0122] Specifically, graph query: query nodes related to pneumonia in the multimodal knowledge graph, such as symptoms, diagnosis methods, treatment plans, etc.;

[0123] Node matching: Match the triples generated in step S5 with the entities and relations in the knowledge graph to find relevant information;

[0124] Association analysis: Analyze the associations between these nodes to better understand the relevant knowledge of pneumonia.

[0125] S7. Output matching node data. Node data output: Output knowledge graph node data related to the question, including related entities and relationships.

[0126] Specifically, output entities: output entities related to pneumonia, such as "pneumonia", "lung infection", "fever", etc.;

[0127] Output relations: output the association relations between entities, such as "pneumonia-cause-fever";

[0128] Output attributes: Output attributes related to the entity, such as "pneumonia-location-lungs".

[0129] S8. Output the final result. Final result: Integrate the information from all steps to provide the user with a detailed answer, which may include text explanations, image analysis results, recommended further examinations, etc.

[0130] Specifically, diagnostic suggestions: diagnostic suggestions are given based on the results of image analysis and information from the professional knowledge base;

[0131] Image analysis results: Provides a detailed interpretation of the uploaded X-rays, including the location and extent of suspected pneumonia;

[0132] Possible treatment options: Based on the information matched in the knowledge graph, list several possible treatment options and briefly explain the advantages and disadvantages of each option;

[0133] Other recommendations: Provide further examination or consultation with a specialist as appropriate.

[0134] Sample output:

[0135] Answer received by the doctor: The analysis results show that there are indeed signs of suspected pneumonia in the uploaded X-ray. Spotty shadows were observed in the right lower lobe, which is a common imaging manifestation of pneumonia. According to the information in the professional knowledge base, these signs are more consistent with bacterial pneumonia. It is recommended to conduct a blood test to confirm the type of pathogen and consider the use of antibiotics. In addition, considering the patient's age and medical history, it may be necessary to closely monitor the progression of the disease and perform a follow-up CT scan to evaluate the effectiveness of treatment. Please consult a respiratory specialist for more detailed treatment guidance.

[0136] Embodiment 3, this embodiment provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model knowledge question and answer method integrating multimodal knowledge graphs described in Embodiment 1.

[0137] Embodiment 4. This embodiment provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the large language model knowledge question and answer method integrating multimodal knowledge graphs described in Embodiment 1.

[0138] The electronic device may include: a processor, a memory and a communication unit. These components communicate via one or more buses. Those skilled in the art will appreciate that the structure of the electronic device does not limit the embodiments of the present invention, and it may be a bus structure or a star structure, or a combination of certain components, or different component arrangements.

[0139] The communication unit is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0140] The processor is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0141] The memory is used to store the execution instructions of the processor. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0142] When the execution instructions in the memory are executed by the processor, the electronic device is enabled to execute part or all of the steps of Embodiment 1.

[0143] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A large language model knowledge question answering method integrating multimodal knowledge graph, characterized in that: include: Acquire multimodal data, convert the multimodal data into vector representation, and store it in the professional knowledge base; Extract structured information from multimodal data to build a multimodal knowledge graph; Build a large multimodal model. The large multimodal model is obtained by fine-tuning a general large-scale pre-trained language model. Textual knowledge representation: Focusing on the classic (h, r, t) triple form, each entity and relationship is mapped to a high-dimensional vector space through its unique ID to achieve implicit expression of information. In specific implementation, the ID of the entity or relationship is used as input and converted into a dense vector using the embedding layer. This conversion mechanism is essentially a matrix multiplication operation of the one-hot encoding of each ID and the embedding matrix, thereby obtaining a vector representation with rich semantics. For image and audio data, the pre-trained image recognition and speech recognition models are used to convert them into text descriptions for entity recognition and relationship extraction. Build a dynamic update mechanism to update the professional knowledge base with new data; Obtain user questions, analyze user questions to obtain user intent, optimize the original questions based on user intent to obtain user optimization questions; convert user optimization questions into vectors through a pre-trained word vector model; Receive the vectorized user optimization problem and find relevant text fragments from the professional knowledge base; Use user optimization questions and related text snippets as input to the multimodal large model to obtain professional answers to questions; Send the answers to professional questions to the information extraction module to obtain triple information; Taking the triple information as the core of the query vector, we conduct a matching search in the multimodal knowledge graph based on the query vector to find the knowledge nodes directly related to the question and obtain the multimodal resources matching the knowledge nodes. Conduct correlation analysis and logical sorting of multimodal resources and related text fragments to obtain integrated information, and input the integrated information into the multimodal macro model; The collaborative enhancement framework is used for processing. The collaborative enhancement framework integrates RAG, LLM and multimodal knowledge graph technology. Under the collaborative enhancement framework, the user's original question is first split into multiple sub-questions, each of which accurately queries a specific field in the professional knowledge base; then, using the generation capability of LLM, the retrieved knowledge is organically integrated with the partial answer generated by LLM to form a complete answer; the multimodal knowledge graph is used to verify and calibrate the answer to ensure that each knowledge triple in the answer has a reliable source and basis; The multimodal large model processes the integrated information to obtain the final answer and transmits it to the user.

2. The large language model knowledge question answering method integrating multimodal knowledge graph as claimed in claim 1, characterized in that: The multimodal data includes text, images, audio and video; data cleaning technology is applied to refine the data, and an embedding layer is designed to convert various modal data into a unified vector representation; for text data, text parsing and text segmentation are performed, and the text is converted into a vector through a pre-trained word vector model, and a vector index is constructed through an encoder, and the processed document information, vector data and index are stored in a database; for image data, video data and audio data, they are converted into vectors through an embedding model and stored in a database.

3. The large language model knowledge question answering method integrating multimodal knowledge graph as claimed in claim 1, characterized in that: The process of building a multimodal knowledge graph includes using entity recognition and relationship extraction techniques to extract structured information from multimodal data and construct a knowledge graph. At the same time, multimodal data processing technology is used to fuse image and audio non-text features to enrich the content of the knowledge graph.

4. The large language model knowledge question answering method integrating multimodal knowledge graph as claimed in claim 1, characterized in that: The information extraction module includes a text information extraction submodule, an image information extraction submodule, a video information extraction submodule, an audio information extraction submodule and a cross-modal alignment module.

5. The large language model knowledge question answering method integrating multimodal knowledge graph as claimed in claim 4, characterized in that: The text information extraction submodule is used to identify specific types of information in the text; the image information extraction submodule is used to identify objects and scenes in the image, convert them into labels that can be indexed and queried, and analyze the relationship between objects in the image.

6. The large language model knowledge question answering method integrating multimodal knowledge graph as claimed in claim 4, characterized in that: The video information extraction submodule is used to extract image information from each frame of the video and capture key information in dynamic changes; the audio information extraction submodule is used to convert audio into text; The cross-modal alignment module is used to align and integrate the information extracted from different modalities to form a unified knowledge representation.

7. A large language model knowledge question answering system integrating multimodal knowledge graph, characterized in that: include: The data processing module is configured to: acquire multimodal data, convert the multimodal data into vector representation, and store it in a professional knowledge base; Extract structured information from multimodal data to build a multimodal knowledge graph; build a multimodal large model, which is obtained by fine-tuning a general large-scale pre-trained language model; build a dynamic update mechanism to update the professional knowledge base with new data; text knowledge representation: focus on the classic (h, r, t) triple form, each entity and relationship is mapped to a high-dimensional vector space through its unique ID to achieve implicit expression of information; in specific implementation, the ID of the entity or relationship is used as input and converted into a dense vector using the embedding layer; this conversion mechanism is essentially a matrix multiplication operation of the one-hot encoding of each ID and the embedding matrix, thereby obtaining a vector representation with rich semantics; for image and audio data, the pre-trained image recognition and speech recognition models are used to convert them into text descriptions for entity recognition and relationship extraction; Build a dynamic update mechanism to update the professional knowledge base with new data; The user question processing module is configured to: obtain user questions, analyze user questions to obtain user intentions, optimize the original questions based on user intentions to obtain user optimization questions; and convert the user optimization questions into vectors through a pre-trained word vector model; The relevant text acquisition module is configured to: receive the vectorized user optimization problem and find relevant text fragments from the professional knowledge base; The professional question answer acquisition module is configured to: use the user's optimized question and related text fragments as inputs of the multimodal large model to obtain the professional question answer; The triple information acquisition module is configured to: send the answers to the professional questions to the information extraction module to obtain triple information; The multimodal resource acquisition module is configured to: use the triple information as the core of the query vector, perform matching search in the multimodal knowledge graph according to the query vector, find the knowledge node directly related to the question, and obtain the multimodal resources matching the knowledge node; The information integration module is configured to: perform correlation analysis and logical sorting of multimodal resources and related text fragments to obtain integrated information, and input the integrated information into the multimodal macro model; The collaborative enhancement framework is used for processing. The collaborative enhancement framework integrates RAG, LLM and multimodal knowledge graph technology. Under the collaborative enhancement framework, the user's original question is first split into multiple sub-questions, each of which accurately queries a specific field in the professional knowledge base; then, using the generation capability of LLM, the retrieved knowledge is organically integrated with the partial answer generated by LLM to form a complete answer; the multimodal knowledge graph is used to verify and calibrate the answer to ensure that each knowledge triple in the answer has a reliable source and basis; The answer output module is configured as follows: the multimodal large model processes the integrated information to obtain the final answer and transmits it to the user.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model knowledge question and answer method integrating multimodal knowledge graphs as described in any one of claims 1-6.

9. An electronic device, characterized in that: It includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the large language model knowledge question and answer method integrating multimodal knowledge graphs as described in any one of claims 1-6.

Images