Medical visual question and answer method and device based on knowledge graph and large language model

Abstract

The invention relates to a medical visual question and answer method and device based on a knowledge graph and a large language model. The method comprises the following steps: firstly, training a medical visual question and answer classification model by using a medical image, a knowledge graph and a question text; then, inputting the medical image and the question text into the medical visual question and answer classification model to obtain a predicted answer; inputting the knowledge graph and the question text into a knowledge graph retrieval enhancement generation module, and retrieving the knowledge graph to obtain a structured cue word; and integrating the predicted answer, the structured cue word and a basic prompt containing role positioning and question and answer requirements by adopting LLM to obtain a final medical answer. According to the method, the response content is richer and more professional, the diagnosis logic is better, and a medical visual question-answering system better meets the real clinical requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual question answering, in particular to a medical visual question answering method and device based on a knowledge graph and a large language model. BACKGROUND

[0002] Visual Question Answering (VQA) is a multi-disciplinary problem that combines Computer Vision (CV) and Natural Language Processing (NLP), aiming to understand images and related questions and provide accurate answers. It is one of the important visual language understanding tasks. Currently, VQA is widely used in assistive technology for the visually impaired, visual chat robots, and medical diagnosis. Medical Visual Question Answering (MedVQA) is the application of visual question answering technology in the medical field. It provides accurate answers to related questions through medical images and is commonly used to solve specific problems such as clinical decision-making and telemedicine. The medical images in it are usually grayscale images obtained by CT, MRI, X-ray, and other imaging methods, and the questions include medical details, imaging methods, and organ systems involved.

[0003] Existing research shows that radiologists have a heavy workload of imaging examinations. They need to judge a CT or MRI image in an average of 3 to 4 seconds. Despite this, radiologists must also answer 27 phone calls from doctors and patients every day and help them solve practical medical problems, which leads to the dual dilemma of a huge workload and decreased work efficiency for radiologists. However, thanks to the extensive research of deep learning in the medical field, some research in the NLP field proposes to use language modalities to build question answering systems. In the CV field, many existing research proposes to use deep learning technology to learn medical information from medical images and apply it to segmentation and reconstruction tasks. In the cross-modal field, MedVQA is a new and promising research direction. The first Medical VQA Challenge was held in 2018, and since then, more and more researchers, doctors, and research organizations have joined the MedVQA field. They continuously propose new datasets and methods, expand field tasks, and enhance field technology applications in real-world scenarios, making MedVQA a vibrant and highly valuable research field. A mature MedVQA system can assist radiologists, pathologists, or medical assistants, helping to reduce the workload of medical staff, reduce misdiagnosis rates and the spread of false information, and thus improve the safety and overall quality of healthcare services.

[0004] The progress in the field of MedVQA is closely related to the development of technologies in the fields of CV and NLP, and due to the particularity of the medical field and the complexity of medical visual question answering, there are significant differences between MedVQA and general question answering tasks. At present, MedVQA still faces the following three challenges: first, the existing model has limited feature interaction learning ability when processing clinical problems and medical images, and fails to fully understand the information in the clinical problems and medical images, which limits the application potential of the model in actual medical scenarios. Second, medical question answering usually requires the answerer to have deep professional knowledge, while the existing model mainly focuses on extracting effective information from medical images, and less considers the guidance of external medical field knowledge, which restricts the performance of the model in dealing with professional problems. Finally, the existing model usually takes simple question answering after classification prediction as the main, and there are obvious defects in the diagnosis strategy, which is difficult to understand the deeper user demand in the medical problem.

[0005] Taking the images in the VQA-RAD data set for medical visual question answering, the existing MedVQA model shows that the replies to the two types of problems of open and closed questions are thin in content, lack of diagnostic logic, and have a large gap from the real doctor-patient dialogue scene. SUMMARY

[0006] Therefore, it is necessary to provide a medical visual question answering method and device based on a knowledge graph and a large language model in view of the above technical problems.

[0007] A medical visual question answering method based on a knowledge graph and a large language model, the method comprising: According to the medical image, the question text and the knowledge graph, a trained medical visual question answering classification model is used for processing to obtain a predicted answer; the medical visual question answering classification model is used for extracting features from the medical image, the knowledge graph and the question text by using a multi-modal encoder, performing multi-stage fusion of the extracted features by using an attention compression and a gate mechanism improved MCAN fusion method, and generating a predicted answer according to the fused features.

[0008] The knowledge graph and the question text are input into a knowledge graph retrieval enhancement generation module, and through retrieval of the knowledge graph, a structured prompt word is obtained; the knowledge graph retrieval enhancement generation module is used for dynamically retrieving the knowledge graph according to the question text to generate a structured prompt word containing node attributes and knowledge graph relationship edges related to the question.

[0009] The predicted answer, the structured prompt word and the basic prompt containing the role positioning and the question answering requirement are integrated by using an LLM to obtain a final medical answer.

[0010] A medical visual question answering device based on a knowledge graph and a large language model, the device comprising: The medical visual question answering classification unit is used to process medical images, question texts and knowledge graphs using a trained medical visual question answering classification model to obtain predicted answers. The medical visual question answering classification model is used to extract features from medical images, knowledge graphs and question texts using a multimodal encoder, and to fuse the extracted features using an improved MCAN (Multi-Channel Detection and Interpretation) mechanism using attention compression and gating mechanisms. The extracted features are then fused in multiple stages to generate predicted answers based on the fused features.

[0011] The knowledge graph retrieval enhancement unit is used to input the knowledge graph and question text into the knowledge graph retrieval enhancement generation module, and obtain structured prompt words by searching the knowledge graph; the knowledge graph retrieval enhancement generation module is used to dynamically search the knowledge graph based on the question text, and generate structured prompt words containing node attributes related to the question and knowledge graph relationship edges.

[0012] The question-answering unit is used to integrate the predicted answers, structured prompt words, and basic prompts including role positioning and question-answering requirements using LLM to obtain the final medical answer.

[0013] The aforementioned medical visual question-answering method and device based on a knowledge graph and large language model includes: inputting medical images and question text into a medical visual question-answering classification model to obtain a predicted answer; inputting the knowledge graph and question text into a knowledge graph retrieval enhancement generation module to obtain structured prompts by searching the knowledge graph; and integrating the predicted answer, structured prompts, and basic prompts including role positioning and question-answering requirements using the LLM to obtain the final medical answer. This method results in a medical visual question-answering system with richer, more professional, more diagnostically sound, and more aligned with real-world clinical needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 1 is a flowchart of a medical visual question answering method based on a knowledge graph and a large language model in one embodiment; Figure 2 This is a diagram showing the structure of a medical visual question answering framework based on a knowledge graph and a large language model in one embodiment; Figure 3 is a visualization of three entity graphs in another embodiment; Figure 4 A comparison between KGLMQA and a general large model in three response dimensions in another embodiment; Figure 5 The response of ChatGPT and DeepSeek-V3 large models to the question in another embodiment; Figure 6This is KGLMQA's response to a question in another embodiment. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0016] This application proposes a medical visual question answering method based on knowledge graph and large language model, abbreviated as KGLMQA.

[0017] In one embodiment, Figure 1 As shown, a medical visual question answering method based on knowledge graph and large language model is provided, which includes the following steps: Step 100: Based on the medical image, question text and knowledge graph, a trained medical visual question answering classification model is used to process them to obtain a predicted answer; the medical visual question answering classification model is used to extract features from the medical image, knowledge graph and question text using a multimodal encoder, and the extracted features are fused in multiple stages using a fusion method after improving MCAN using an attention compression and gating mechanism, and a predicted answer is generated based on the fused features.

[0018] Specifically, the medical visual question-answering classification model includes: a multimodal encoder, a feature fusion module, and an answer generation module. After obtaining the features of the question text, medical image, and knowledge graph, the model uses the feature fusion module to perform multi-stage feature fusion to obtain a fused embedding. Finally, the fused embedding is passed through the answer generation module to obtain the classification prediction result and generate the predicted answer.

[0019] This application improves the feature fusion mechanism and multimodal fusion layer to achieve a medical visual question answering classification model with better performance, which serves as the input receiver of the framework and outputs the medical visual question answering classification results.

[0020] In the feature fusion module, the gating mechanism was used to improve the feature fusion mechanism and the multimodal fusion layer, and a medical visual question answering classification model was trained. It was verified on three public datasets that better classification performance was achieved and it was more suitable for downstream tasks.

[0021] Step 102: Input the knowledge graph and question text into the knowledge graph retrieval enhancement generation module, and obtain structured prompt words by searching the knowledge graph; the knowledge graph retrieval enhancement generation module is used to dynamically search the knowledge graph based on the question text, and generate structured prompt words containing node attributes related to the question and knowledge graph relationship edges.

[0022] Specifically, the Knowledge Graph Retrieval Augmented Generation (KGRAG) module retrieves and generates node and relationship information from the knowledge graph. The KGRAG module is used to effectively incorporate external medical knowledge from the knowledge graph into the medical visual question answering process.

[0023] The Knowledge Graph Retrieval Enhancement Generation module (KGRAG) generates a subgraph centered on each node in the knowledge graph, converts the node information in the subgraph into textual representations, and generates semantic embeddings using the BERT model. For each user question, the module calculates the cosine similarity between the question embedding and the subgraph embedding, selects the N subgraphs with the highest similarity, and uses a dynamic pruning strategy to dynamically assign weights to the nodes and edges in the subgraph based on the semantic similarity of the query, retaining highly relevant nodes and reducing the weights of irrelevant nodes and edges. The module ultimately outputs structured prompts containing node attributes relevant to the question and knowledge graph relationship edges.

[0024] Knowledge graphs play a crucial role in the field of natural language processing (NLP). They enhance semantic understanding, reasoning capabilities, and task accuracy through structured knowledge, and are widely used in question-answering, retrieval, generation, reasoning, and domain-specific tasks. They overcome the knowledge limitations of traditional language models, performing particularly well in scenarios requiring precise facts or expertise. Early approaches to extracting and utilizing knowledge from knowledge graphs (KGs) involved query-based knowledge retrieval and rule-based semantic reasoning. However, Yasunaga et al. retrieved relevant nodes to create a joint graph containing relevant nodes and question-answer context, providing richer knowledge content. Furthermore, to obtain more complex relationship information between nodes, Kim et al. shifted node retrieval to triple retrieval. Li et al. enhanced the retrieval process by combining text and topological information, allowing the model to better capture structural relationships within graph-structured data. KGRAG technology improves answer quality by associating medical questions with expertise, providing external medical knowledge support to the answer generator.

[0025] Step 104: The predicted answer, structured prompt words, and basic prompts including role positioning and question-answering requirements are integrated using LLM to obtain the final medical answer.

[0026] Specifically, three parts of prompts are integrated: the predicted answers of the medical visual question-answering classification model, the structured prompt words of the knowledge graph retrieval enhancement generation module, and the basic prompts containing role definitions and question-answering requirements. The integrated prompts are input into the LLM to generate the final medical answer.

[0027] LLM, with its superior natural language understanding capabilities, has opened up unprecedented opportunities for innovation and development in medical conversations and demonstrated tremendous potential in numerous medical applications. LLM's applications in the medical field encompass clinical decision support, doctor-patient dialogue, and the dissemination of medical knowledge. These technologies are expected to optimize medical conversation processes, enhance the accuracy of medical conversations, and ultimately achieve more efficient, personalized, and comprehensive medical services.

[0028] In certain clinical scenarios, medical procedures may rely on analyzing patient symptoms or using clinical recommendations. LLMs can comprehensively analyze patient symptoms and related medical knowledge and data to accurately identify potential diseases or health issues, ultimately assisting in diagnosis and providing medical advice. This demonstrates the enormous potential of LLMs in areas such as assisting diagnosis, clinical decision support, and patient health education. For example, researchers have begun using LLMs to diagnose Alzheimer's disease and dementia. Meanwhile, a growing body of research advocates leveraging LLMs' natural language understanding capabilities to meet the needs of medical conversations. Although LLMs may produce hallucinations due to a lack of sufficient expertise in specific medical fields, their outstanding performance in natural language processing tasks makes them a powerful tool for MedVQA answer generation. By controlling LLM output through external knowledge augmentation and hint engineering to mitigate hallucinations, LLMs continue to have great potential as answer generators for medical visual question answering.

[0029] The aforementioned medical visual question-answering method based on a knowledge graph and a large language model includes: inputting medical images and question text into a medical visual question-answering classification model to obtain a predicted answer; inputting the knowledge graph and question text into a knowledge graph retrieval enhancement generation module to retrieve structured prompts through knowledge graph retrieval; and integrating the predicted answer, structured prompts, and basic prompts including role positioning and question-answering requirements using the LLM to obtain the final medical answer. This method results in a medical visual question-answering system with richer, more professional, more diagnostically sound, and more aligned with real-world clinical needs.

[0030] In one embodiment, a medical visual question answering classification model includes: a multimodal encoder, a feature fusion module and an answer generation module; step 100 includes: using the multimodal encoder to perform feature encoding on the medical image, question text and knowledge graph to obtain image encoding features, question encoding features and knowledge graph encoding features; inputting the question encoding features, image encoding features and knowledge graph encoding features into the feature fusion module for multi-stage feature fusion to obtain fused embedded features; inputting the fused embedded features into the answer generation module to obtain a predicted answer.

[0031] In one embodiment, the multimodal encoder includes an image encoder, a language encoder, and a knowledge graph encoder; the multimodal encoder is used to perform feature encoding on the medical image, question text, and knowledge graph to obtain image encoding features, question encoding features, and knowledge graph encoding features, including: inputting the question text into the language encoder, processing the question text through BERT to obtain an initial sub-embedding, and then using a dual-gated recurrent unit to reduce the dimension of the initial sub-embedding to obtain question encoding features; inputting the preprocessed medical image into the image encoder to obtain image encoding features; using the knowledge graph encoder to reconstruct the knowledge graph into an entity graph containing disease nodes, relationship nodes, and attribute nodes, connecting the disease nodes, relationship nodes, and attribute nodes according to clustering attributes, generating three types of edges: disease-attribute, relationship-attribute, and attribute-attribute, to obtain knowledge graph encoding features.

[0032] Specifically, the multimodal encoder includes an image encoder, a language encoder, and a knowledge graph encoder.

[0033] (1) Language encoder and image encoder. Extracting features from question text and medical images and using their features to train the model is a general method for training MedVQA models, and it is also the first step in giving the MedVQA model visual understanding capabilities. The goal of the multimodal encoder is to extract features from question text, medical images, and knowledge graphs respectively. In the field of general visual question answering, the training of visual attention models often relies on large-scale labeled data. However, due to the professionalism and complexity of medical data labeling, the MedVQA field lacks large-scale labeled datasets, which makes it difficult to develop methods for training MedVQA using large-scale data. Therefore, for the question part, BERT is first used to obtain an initial word embedding of 768 dimensions, and then a bidirectional gated recurrent unit (GRU) is used to reduce the word embedding dimension to h dimensions, expressed as , l Indicates the maximum length of the question, h For medical images, we first preprocess the image, resize it, convert the tensor, and normalize it. We then use the ResNet-50 model pre-trained on ImageNet to encode the image, map it to a 1024-dimensional feature vector through a fully connected layer, and then convert the 1024-dimensional feature vector to a unified hidden dimension through a fully connected layer. h , expressed as .

[0034] (2) Knowledge Graph Encoder. At this stage, this application follows the Tri-entityGraph Construction method proposed by Huang et al. First, the knowledge graph is reconstructed into an entity graph containing three entity nodes: disease, relationship, and attribute. , connect the three nodes according to the clustering attributes, and generate three types of edges: disease-attribute, relationship-attribute, and attribute-attribute. D is the set of disease nodes , R is the set of relationship nodes , A is a collection of attribute nodes , E is formulated as the edges of the adjacency matrix: (1) in, O is a zero matrix, is the disease-attribute adjacency matrix, is the transposed matrix of the relationship-attribute adjacency matrix, attribute-attribute adjacency matrix, The visual representation of Figure 3 shown.

[0035] For disease nodes, the feature outputs extracted by ResNet-50 are averaged to obtain the central embedding of each disease, which is expressed as , the relationship and attribute nodes are initially embedded using the pre-trained BERT encoding, expressed as and , and use a fully connected layer to transform the initial embedding into a uniform hidden dimension 、 、 Subsequently, a graph convolutional network (GCN) is applied to learn node embeddings for 、 、 Three adjacency matrices are used to set three feature aggregation directions for clustering.

[0036] Taking the relationship node as an example, the embedding of the relationship node is obtained by aggregating the attributes related to the relationship and used to learn the relationship-specific clustering features. The output of the layer and the The input of the layer is represented as ,when When , the disease node is the same as the attribute node. The update of the relation node in layer GCN can be expressed as: (2) in, Indicates the Relationship nodes in the layer A collection of attribute neighbors. The Agg() function aggregates attribute information related to the relationship. Relationship nodes Together with its related attributes, a “relationship cluster” is formed. In the initial stage of the aggregation mechanism, the model calculates the attention matrix to model the correlation between the relationship node and its key attributes. The calculation is as follows: (3) By calculating the attention scores of each relation node and its attribute neighbors, the model focuses on the most relevant attributes, where is a trainable weight matrix used to learn the relation-attribute attention mapping, symbolized by ” indicates element-wise multiplication. is the relation-attribute attention matrix, where is the number of relationships, is the number of attributes. Based on the relationship-attribute attention matrix , first project the attribute features into the relation feature space, and then perform weighted aggregation on the attributes according to the attention matrix. The process can be expressed as: (4) (5) in, is the transformation matrix used to map attribute features to relationship feature space, is the attribute feature after projection. Formulas (2) to (5) are the formal descriptions of the relationship node aggregation process. The aggregation process of disease nodes and attribute nodes is similar. Finally, by adding a residual connection to strengthen the original information and alleviate the gradient disappearance problem, it can be expressed as: (6) (7) (8) in, 、 、 is the initial embedding, 、 、 is the output of the last layer of GCN, 、 、 is the final embedding of the node.

[0037] In one embodiment, the feature fusion module includes an image-question fusion module, an image-disease fusion module, a question-relationship fusion module, an image-question-disease fusion module, and a question-image-relationship fusion module; the knowledge graph encoding features include disease features and relationship features; the fused embedding features include image-question-disease features and question-image-relationship features; the question encoding features, image encoding features, and knowledge graph encoding features are input into the feature fusion module for multi-stage feature fusion to obtain fused embedding features, including: inputting the question encoding features and image encoding features into the image-question fusion module to obtain question-image features; inputting the image encoding features and knowledge disease features into the image-disease fusion module to obtain image-disease features; inputting the question encoding features and relationship features into the question-relationship fusion module to obtain question-relationship features; inputting the question-image features and image-disease features into the image-question-disease fusion module to obtain image-question-disease features; inputting the question-relationship features and image-disease features into the question-image-relationship module to obtain question-image-relationship features.

[0038] Specifically, the image-question fusion module, image-disease fusion module, question-relationship fusion module, image-question-disease fusion module, and question-image-relationship fusion module all adopt the same feature fusion method (the method after improving MCAN through attention compression and gating mechanism, see formula (14) F ), the fusion module receives two different features (such as image features and question features) and fuses them to obtain a fusion feature.

[0039] In one embodiment, the image-question fusion module is used to align the two modal information with each other through bidirectional cross attention using an improved fusion method based on MCAN, and perform weighted averaging on the aligned image coding features and question coding features to obtain image average features and question average features, and use a gating mechanism to process the image average features and question average features to obtain gating weights, and then concatenate and fuse the image average features and question average features according to the gating weights to obtain question-image features.

[0040] Specifically, in the feature fusion module, the MCAN fusion method is improved, and the contribution of different modalities is dynamically adjusted in multi-stage fusion using attention compression and gating mechanisms, which enhances the adaptability of the model to complex feature combinations. The improved fusion method is expressed as . Based on image features and problem characteristics Taking the fusion of , as an example, we first use MCAN to align the two modal information with each other through bidirectional cross attention. and , then from and Extracting a weighted average one-way representation from a sequence and , using the gating mechanism to obtain the gating weight , and and The concatenation and fusion are performed into one representation, as shown in formulas (9) to (14).

[0041] (9) (10) (11) (12) (13) (14) in, 、 represents the sequence length, is the weight obtained through the attention mechanism, is the weight matrix in gated fusion, is the Hadamard product.

[0042] At the same time, a multi-level fusion process is introduced to enhance the fusion depth, extract more complex cross-modal features, and improve the model's understanding of the relationship between "entities in the question" and "graph knowledge". Specifically, in the first stage of feature fusion, the association between the image and the question is captured by fusing the image-question feature to generate the image-question feature. In the second stage, the image-disease features and question-relationship features are fused to generate 、 In the third stage, the image-question-disease features and question-image-relationship features are integrated to enhance the semantic richness and context relevance of the features. 、 The complete fusion process can be formulated as: (15) (16) (17) (18) (19) The above This is the improved fusion method of this application. 、 、 、 、 It is the fusion feature generated in the three fusion stages.

[0043] In one embodiment, the fused embedding features include image-question-disease features and question-image-relationship features; the fused embedding features are input into the answer generation module to obtain a predicted answer, including: splicing and normalizing the image-question-disease features and the question-image-relationship features to obtain multiple semantic features; the multiple semantic features are used The layer is processed to obtain the question label prediction result; the question label prediction result is normalized and then binarized according to the preset binarization threshold to obtain the binarized prediction result; the binarized prediction result is mapped to the index of the knowledge graph using the argmat function to obtain the binarized index result; the binarized index result is formatted into a readable string description using the Format function to obtain the predicted answer.

[0044] Specifically, in the answer generation module of the medical visual question answering classification model, first 、 The two fusion features are spliced ​​together. The spliced ​​features contain multiple semantic associations of questions, images, diseases, and relationships. After normalization, they are passed into layer( Layer Figure 2 The classifier in the classifier is used to map the concatenated and normalized features to a 419-dimensional category space (419 nodes of the knowledge graph) to obtain the output of the linear layer. .

[0045] (20) The loss function used for target optimization is: (twenty one) in, is the true label.

[0046] Afterwards, Perform normalization and use it in the validation set By maximizing the weighted F1 score, the optimal binarization threshold is determined ,application Generate binary predictions for the test problem and use Function implements the index mapping of knowledge graph to obtain ,use Format the information of the knowledge graph node into a readable string description and finally output the predicted answer. The specific process can be formulated as follows: (twenty two) (twenty three) (twenty four) (25) (26) in, is the linear layer output on the validation set, is the true label on the validation set, represents normalization processing, represents binarization, Indicates indexing after binarization.

[0047] in, Figure 2 The specific process of Binary Index Mapping in the code includes: normalizing the question label prediction results and then binarizing them according to the preset binarization threshold to obtain the binarized prediction results; using the argmat function to implement the index mapping of the knowledge graph to obtain the binarized index results; using the Format function to format the information of the knowledge graph node into a readable string description to obtain the predicted answer.

[0048] In one embodiment, step 102 includes: performing sub-graph segmentation on the knowledge graph to obtain multiple sub-graphs, embedding all sub-graphs into text using the BERT model and saving them as cache files; encoding the question text into a query vector through BERT, and calculating the cosine similarity with each sub-graph to obtain a similarity score between the question and the sub-graph; selecting the top-N sub-graph center nodes with the highest similarity scores to obtain a set S; for the sub-graph corresponding to each center node in the set S, retaining the nodes and edges whose edge weights with the query vector exceed a preset threshold to obtain a sub-graph retrieval refinement result; the edge weight is the minimum value of the similarity between the query vector and two nodes on the same side; generating structured prompt words based on the sub-graph retrieval refinement result; the result prompt words include: node content description and relationship description.

[0049] Specifically, in the KGLMQA framework, the KGRAG module improves the accuracy, professionalism, and contextual relevance of answers by retrieving knowledge graphs and generating enhanced prompts. Its core role is to combine the preliminary prediction results of the Med-VQA classification model with rich medical knowledge, make up for the limitations of the model, enhance KGLMQA's ability to handle complex medical problems, and provide more reliable support for clinical decision-making. In addition, the KGRAG module can dynamically retrieve information based on the latest medical knowledge graph, avoiding the limitations of traditional static embedding models. At the same time, by analyzing the relationships between nodes in the knowledge graph, it ensures that the information provided is highly relevant to the user's questions. Specifically, the implementation process of the KGRAG module is mainly divided into four stages: subgraph segmentation and textual cache embedding, query embedding and similarity calculation, subgraph retrieval and refinement, and prompt information generation.

[0050] Subgraph segmentation and textual cache embedding. In this stage, a knowledge graph is defined as a directed graph. ,in is a collection of nodes, Is the edge set, including the head node , tail node and relationship types For any node in the complete graph , taking it as the central node , the construction radius is Subgraph of , which is expressed in formula: (27) (28) (29) (30) in yes node Neighbor nodes that can be reached in steps, is a node With node In the figure The shortest path distance in It's text, is the textual embedding obtained using the same BERT model as the MedVQA classification model. The subgraph is saved as a cache file after being embedded in text.

[0051] Query embedding and similarity calculation. The question text is input into KGRAG and is defined as , After BERT encoding, it is converted into a query vector and the cosine similarity with each subgraph is calculated.

[0052] (31) (32) Subgraph retrieval and refinement. With each node After the similarity scores are calculated, the top-N subgraph center nodes with the highest similarity scores are selected to obtain the set .

[0053] (33) based on The central node in gets the N most relevant subgraphs, and for each subgraph , retain the query similarity exceeding the threshold The nodes and edges of The minimum similarity value with the two end nodes: (34) (35) like , then the edge is pruned.

[0054] Prompt information generation. After subgraph retrieval and refinement, the prompt words generated by KGRAG contain two parts: node content description and relationship description , which is expressed in formula: (36) (37) in, is a node Type, is a node Name, is a node content, is a node With node The edge between them contains the relationship between the two nodes.

[0055] In one embodiment, step 104 includes: integrating user query, role positioning, basic prompts of question and answer requirements, visual information, visual information confidence, and result prompt words into final prompt words; inputting the final prompt words into the LLM model, completing information processing through visual information confidence and basic prompts of question and answer requirements, adjusting the question and answer style through role positioning, and obtaining the final medical answer.

[0056] In one embodiment, the LLM model is the DeepSeek-v3 large language model.

[0057] Specifically, the defect of traditional medical visual question answering models is that they lack semantic understanding of question texts, and the answers they produce are simple and lack consideration of the user's real needs. KARAG technology retrieves relevant nodes and relationships from the knowledge graph, generates context-rich medical domain knowledge, and supplements the medical visual question answering model with richer and more professional relevant medical knowledge. It also provides the explainability of knowledge associations by describing the relationships between nodes. However, although KARAG has improved the quality of responses of traditional medical visual question answering models, there are still problems such as lack of readability of structured response content, inability to handle ambiguous information well, and mismatch between question answering style and scenario. Therefore, the user query is combined with the KARAG technology to form a complete set of medical visual question answering models. , role positioning , task requirements , visual information and confidence , knowledge graph information Integrate into the final prompt word , Used to pass into the DeepSeek-v3 large language model to generate responses. The generated formula is as follows: (38) (39) (40) in, and Can be adjusted according to the task type, It is a complete description of the visual information and preserves the confidence of the visual information .after, Pass in the DeepSeek-v3 large language model and pass the confidence and Complete information processing through Adjust the question-answering style. Ultimately, the large model responds to users by leveraging its own information fusion and reasoning capabilities, as well as its excellent natural language expression capabilities.

[0058] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0059] In a validation example, the proposed medical visual question answering classification model was evaluated on the P-VQA, VQA-RAD, and SLAKE datasets and compared with multiple baselines. Subsequently, the performance of KGRAG in terms of knowledge retrieval accuracy and pruning effectiveness was evaluated. Finally, the quality of the final responses generated by the KGLMQA framework was evaluated.

[0060] (1) Dataset VQA-RAD and SLAKE are datasets commonly used for evaluating the performance of MedVQA models. VQA-RAD contains 315 images and 3,515 question-answer pairs, while SLAKE contains 642 images and 14,028 question-answer pairs. SLAKE provides a structured medical knowledge base in the form of a knowledge graph, which is represented by triples. To provide more constructive medical advice, Huang et al. proposed P-VQA, a larger dataset covering imaging diagnosis, clinical diagnosis, and treatment recommendations. It contains 2,169 images and 24,800 question-answer pairs, and includes a medical knowledge graph with 419 entities. Despite the increasing comprehensiveness and richness of MedVQA datasets, existing models still face challenges in leveraging the dataset information for real-world applications.

[0061] For the P-VQA, VQA-RAD, and SLAKE datasets, this example divides the datasets into training, validation, and test sets based on their respective data sizes. The sizes and division ratios of each dataset are shown in Table 1.

[0062] Table 1 Dataset size and partition ratio

[0063] Among them, VQA-RAD does not include KG, so this embodiment only uses question-image encoding fusion. However, P-VQA and SLAKE datasets provide KG, and this embodiment uses Figure 2 The complete feature encoding and fusion process.

[0064] (2) Baseline The performance of the KGLMQA Chinese Medical Visual Question Answering classification model is compared with some existing SOTA methods. The baseline model is described as follows: 1) VQA feature fusion methods: MCB, MFB, MFH, SAN, BAN and MCAN.

[0065] 2) Medical VQA methods: MEVF, CPRD, MMixup, VQAMix, PUBMEDCLIP, MKBN.

[0066] (3) Experimental setup Experiments on P-VQA dataset. This embodiment classifies the experimental setting on P-VQA as a task of classification, and uses Accuracy, Recall, Precision and F1 score as evaluation indicators. First, the performance of KGLMQA and 6 feature fusion baseline methods is compared on the P-VQA dataset without loading the KG, and they use the same question and image encoder. Subsequently, the performance of the state-of-the-art MKBN model and KGLMQA is compared on the complete P-VQA dataset.

[0067] Experiments on VQA-RAD dataset. In the VQA-RAD dataset, the questions are divided into “Closed-ended” questions and “Open-ended” questions according to the answer type. Since the size of the VQA-RAD dataset is the smallest, this embodiment performs performance tests on both “Closed-ended” and “Open-ended” questions on the division of training set: validation set: test set equal to 6:2:2, using Accuracy as the evaluation indicator. In addition, the VQA-RAD dataset does not provide a related medical knowledge graph, so this embodiment cancels the use of the knowledge graph encoder.

[0068] Experiments on SLAKE dataset. The questions in the SLAKE dataset are divided into “Closed-ended” questions and “Open-ended” questions according to the answer type. In addition, SLAKE provides a medical knowledge graph, which contains two types of triples: disease-relation-attribute and organ-relation-attribute, and the disease entities and organ entities in it are uniformly treated as disease entities. In the dataset division, the original training set, validation set and test set are divided according to the ratio of 7:1.5:1.5, and Accuracy is used as the evaluation indicator.

[0069] KGRAG module and KGLMQA framework generated final response quality evaluation experiment. The KGRAG module aims to retrieve and generate knowledge graph information related to the query. The KGLMQA framework integrates a large language model (LLM) to generate more readable, more professional, and diagnostic logic medical visual question answering responses. In specific implementation, KGLMQA utilizes a medical visual question answering classification model trained on the P-VQA dataset to generate visual diagnosis prompts; at the same time, combined with the knowledge graph prompts generated by the KGRAG module, it jointly drives the DeepSeek-V3 large model to generate higher quality responses. Finally, the generated responses are manually evaluated by human experts, and compared with the results generated by the original DeepSeek-V3 and ChatGPT.

[0070] (4) Quantitative results The experimental results on the P-VQA, VQA-RAD, and SLAKE datasets are shown in Tables 2, 3, and 4, respectively. The best performance is marked in bold, and the suboptimal performance is marked with an underscore. Table 2 summarizes the overall performance of each VQA fusion method on the P-VQA dataset. MCAN performs best without using KG, which indicates its advantage in multi-modal fusion. Therefore, this embodiment improves the fusion method based on MCAN, introduces feature refinement and gating mechanism, and uses a multi-stage fusion method, ultimately achieving the best performance of KGLMQA in classification accuracy on the P-VQA dataset.

[0071] Table 2 Performance on P-VQA dataset

[0072] Wherein: NO / Yes indicates whether to use KG (knowledge graph).

[0073] According to the results in Table 2, the following information can be obtained: 1) Compared with the MedVQA model that only uses images and questions as input, both MKBN based on original MCAN fusion knowledge graph and KGLMQA improved on this basis have improved performance on the P-VQA dataset. This shows that using medical knowledge graph to assist in training MedVQA model helps to improve its classification performance.

[0074] (2) Compared with MKBN using original MCAN as feature fusion method, KGLMQA further improves performance by introducing feature refinement and gating mechanism, and achieves the optimal results of accuracy and precision on the P-VQA dataset. This shows that weight allocation in feature fusion process and multi-stage feature fusion help the model to better learn multi-modal features.

[0075] Table 3 Performance on the VQA-RAD dataset.

[0076] Table 4 Performance on the SLAKE dataset

[0077] According to the results in Table 3 and Table 4, we can get the following information: 1) KGLMQA achieved the best performance in both Open-ended and Overall metrics on the VQA-RAD and SLAKE datasets, indicating that the medical visual question answering classification model in the KGLMQA framework has good generalization ability.

[0078] 2) Compared to the MMixup model, KGLMQA's accuracy on closed-ended tasks is slightly lower. This may be because MMixup focuses more on simplifying the model structure and effectively training on small-scale data, while KGLMQA's more complex model structure fails to fully utilize its advantages on VQA-RAD and SLAKE, two tasks with smaller data volumes.

[0079] 3) KGLMQA performs well on open-ended questions, further proving that medical knowledge graphs can help models better answer questions with more flexible forms.

[0080] Since the responses generated by LLM when facing medical dialogue tasks are natural language texts, in order to better evaluate the response quality of the large model in the medical field, Yang et al. designed a set of medical question-answering ranking and annotation standards for evaluating LLM responses, which are divided into 3 capability dimensions and 9 specific capabilities. Chervenak et al. evaluated the large model responses from the aspects of response length, number of factual statements, and emotional tendency. Wei et al. evaluated the large model responses from the aspects of accuracy, completeness, and humanistic care. Dou et al. evaluated the large model responses from the aspects of intent accuracy, medical terminology accuracy, and human expert evaluation. Response accuracy (RA), Diagnostic Logicality (DL), and Response Integrity (RI) were selected as evaluation indicators. The importance of RA, DI, and RI is from the front to the back. RA aims to evaluate the accuracy of the response, DL aims to evaluate whether the response contains diagnostic logic, and RI aims to evaluate the knowledge integrity of the response rather than the simple length of the response. In the KGLMQA framework, a medical visual classification model trained with P-VQA was used. The DeepSeek-V3 model was selected for LLM. 50 groups of image-question pairs were randomly selected for comparative experiments on the three indicators of RA, DI, and RI. The results are as follows: Figure 4 shown.

[0081] ChatGPT and the original DeepSeek-V3 model performed similarly on RA, DI, and RI, but overall DeepSeek-V3 outperformed ChatGPT in response accuracy, response breadth and depth, and diagnostic logic. This may be due to DeepSeek-V3's richer training data in the medical field and its stronger semantic understanding and logical reasoning capabilities. However, KGLMQA, supplemented with visual diagnostic information and medical knowledge graph information, has an absolute advantage in response quality compared to ChatGPT and the original DeepSeek-V3, demonstrating the feasibility of this application's approach.

[0082] (5) Ablation studies To investigate the contribution of improvements made to the feature fusion process to the KGLMQA Chinese Medical Visual Question Answering classification model, this example conducted an ablation experiment. In this experiment, this example removed the image-question-disease fusion module and the question-image-relationship fusion module to examine the impact of images, questions, diseases, and relationships on classification performance, respectively. Furthermore, this example examined the effectiveness of the gating mechanism in dynamically adjusting modal contributions during multi-stage fusion. The results in Table 5 demonstrate that the multi-level fusion process introduced by this method enhances fusion depth, extracts more complex cross-modal features, and improves the model's understanding of the relationships between images, questions, and knowledge graphs.

[0083] Table 5 Ablation experiments of classification models in KGLMQA

[0084] Note: w / o in the table means none.

[0085] In addition, through a specific case, the impact of the medical visual question answering model and the KGRAG module on the DeepSeek-V3 response results was intuitively analyzed, further revealing the action path of visual diagnostic information and medical knowledge graph in KGLMQA and its intervention on LLM generation.

[0086] (6) Case analysis The case analysis shows that a representative case was selected from the model question answering results to facilitate an interpretable comparison of different solutions. The corresponding results are as follows: Figure 5 、 Figure 6 The following is a comparison of the quality evaluation of the answers given by the three models KGLMQA, DeepSeek-V3, and ChatGPT based on the three dimensions of RA, DL, and RI.

[0087] The RA indicator assesses the accuracy of the medical facts provided and the adequacy of the diagnostic basis. KGLMQA, combined with diagnostic imaging information, accurately identified lumbar disc herniation as the primary cause, ruling out other less likely causes and highlighting the warning signs of cauda equina syndrome. DeepSeek-V3 lists multiple possibilities (such as pudendal neuralgia, diabetes, and multiple sclerosis), but without integrating imaging findings, it cannot accurately determine the cause of the user's illness, potentially misleading and distracting the user. While comprehensive, it lacks specificity. ChatGPT only broadly mentions "nerve compression" or "pelvic issues" without specific medical evidence and lacks diagnostic logic.

[0088] DL evaluates whether the answer is logically clear and the causal chain is complete. KGLMQA stratifies the cause from anatomy (sacral innervation) to pathology (disk herniation and compression) to differential diagnosis (cauda equina syndrome) to treatment, with rigorous logic. DeepSeek-V3 lists possible causes in points but fails to establish priorities or correlations (e.g., it doesn't explain why lumbar spine problems are more likely than diabetes). ChatGPT simply lists possible causes without analyzing pathological mechanisms or priorities, resulting in a loose logic.

[0089] RI assesses whether key issues (etiology, treatment, emergency warnings, and next steps) are covered. KGLMQA covers etiology, differential diagnosis, conservative / surgical treatment, medication recommendations, prognosis, emergency warnings, and follow-up recommendations, demonstrating its comprehensiveness. DeepSeek-V3 mentions multiple etiologies and examination recommendations but does not specify treatment pathways (such as medication or surgical indications). ChatGPT simply recommends "seeking medical treatment" and lacks specific diagnostic and treatment guidance and warnings.

[0090] In summary, KGLMQA, after combining visual diagnostic information with medical KG information, achieved the best response quality for RA, DL, and RI, making it suitable for clinical decision-making. DeepSeek-V3 provides a broad differential diagnosis, but further screening is required based on specific cases. ChatGPT's responses are overly general and have limited clinical value. This demonstrates that visual diagnostic information and medical KG information improve the quality of LLM's responses.

[0091] This paper proposes a novel medical visual question answering framework, KGLMQA, consisting of a medical visual question answering classification model, a KGRAG module, and an LLM. It aims to achieve more realistic and professional medical visual question answering. To this end, based on feature refinement and gating mechanisms, an improved feature fusion method was developed, and a multi-stage fusion strategy was designed. Ultimately, a medical visual question answering classification model was trained. This model achieved state-of-the-art performance in terms of accuracy and precision on the P-VQA dataset, and in terms of open-ended and overall metrics on the VQA-RAD and SLAKE datasets. Furthermore, experimental verification shows that the visual diagnostic information provided by the medical visual question answering classification model and the medical knowledge graph retrieved by the KGRAG module significantly improve the response quality of DeepSeek-V3, further demonstrating the effectiveness of the KGLMQA framework. Ultimately, a medical visual question answering system with richer, more professional, more diagnostically sound responses is achieved, and more in line with real-world clinical needs.

[0092] In one embodiment, a medical visual question-answering device based on a knowledge graph and a large language model is provided, comprising: a medical visual question-answering classification unit, a knowledge graph retrieval enhancement unit, and a question-answering unit, wherein: The medical visual question answering classification unit is used to process medical images, question text, and knowledge graphs using a trained medical visual question answering classification model to obtain predicted answers. The medical visual question answering classification model uses a multimodal encoder to extract features from medical images, knowledge graphs, and question texts. The extracted features are then fused using an improved MCAN fusion method using attention compression and gating mechanisms. The extracted features are then fused in multiple stages to generate predicted answers based on the fused features. The knowledge graph retrieval enhancement unit is used to input the knowledge graph and question text into the knowledge graph retrieval enhancement generation module, and obtain structured prompt words by searching the knowledge graph; the knowledge graph retrieval enhancement generation module is used to dynamically search the knowledge graph based on the question text and generate structured prompt words containing node attributes related to the question and knowledge graph relationship edges; The question-answering unit is used to integrate the predicted answers, structured prompt words, and basic prompts including role positioning and question-answering requirements using LLM to obtain the final medical answer.

[0093] In one embodiment, a medical visual question answering classification model includes: a multimodal encoder, a feature fusion module and an answer generation module; the medical visual question answering classification unit is further used to use a multimodal encoder to perform feature encoding on medical images, question text and knowledge graphs to obtain image encoding features, question encoding features and knowledge graph encoding features; the question encoding features, image encoding features and knowledge graph encoding features are input into the feature fusion module for multi-stage feature fusion to obtain fused embedded features; the fused embedded features are input into the answer generation module to obtain a predicted answer.

[0094] In one embodiment, the multimodal encoder includes an image encoder, a language encoder, and a knowledge graph encoder; the medical visual question answering classification unit is also used to input the question text into the language encoder, process the question text through BERT to obtain an initial embedding, and then use a dual-gated recurrent unit to reduce the dimension of the initial embedding to obtain question encoding features; the preprocessed medical image is input into the image encoder to obtain image encoding features; the knowledge graph encoder is used to reconstruct the knowledge graph into an entity graph containing disease nodes, relationship nodes, and attribute nodes, and the disease nodes, relationship nodes, and attribute nodes are connected according to clustering attributes to generate three types of edges: disease-attribute, relationship-attribute, and attribute-attribute, to obtain knowledge graph encoding features.

[0095] In one embodiment, the feature fusion module includes an image-question fusion module, an image-disease fusion module, a question-relationship fusion module, an image-question-disease fusion module, and a question-image-relationship fusion module; the knowledge graph encoding features include disease features and relationship features; the fused embedding features include image-question-disease features and question-image-relationship features; the medical visual question answering classification unit is further used to input the question encoding features and the image encoding features into the image-question fusion module to obtain question-image features; input the image encoding features and the knowledge disease features into the image-disease fusion module to obtain image-disease features; input the question encoding features and the relationship features into the question-relationship fusion module to obtain question-relationship features; input the question-image features and the image-disease features into the image-question-disease fusion module to obtain image-question-disease features; input the question-relationship features and the image-disease features into the question-image-relationship module to obtain question-image-relationship features.

[0096] In one embodiment, the image-question fusion module of the medical visual question answering classification unit is used to align the two modal information with each other through bidirectional cross attention using an improved fusion method based on MCAN, and perform weighted averaging on the aligned image encoding features and question encoding features to obtain image average features and question average features, and use a gating mechanism to process the image average features and question average features to obtain gating weights, and then concatenate and fuse the image average features and question average features according to the gating weights to obtain question-image features.

[0097] In one embodiment, the fused embedding features include image-question-disease features and question-image-relationship features; the medical visual question answering classification unit is further used to concatenate and normalize the image-question-disease features and question-image-relationship features to obtain multiple semantic features; the multiple semantic features are used to The layer is processed to obtain the question label prediction result; the question label prediction result is normalized and then binarized according to the preset binarization threshold to obtain the binarized prediction result; the binarized prediction result is mapped to the index of the knowledge graph using the argmat function to obtain the binarized index result; the binarized index result is formatted into a readable string description using the Format function to obtain the predicted answer.

[0098] In one embodiment, the knowledge graph retrieval enhancement unit is further used to perform sub-graph segmentation on the knowledge graph to obtain multiple sub-graphs, and all sub-graphs are text-embedded using the BERT model and saved as cache files; the question text is encoded into a query vector through BERT, and the cosine similarity with each sub-graph is calculated to obtain a similarity score between the question and the sub-graph; the Top-N sub-graph center nodes with the highest similarity scores are selected to obtain a set S; for the sub-graph corresponding to each center node in the set S, the nodes and edges whose edge weights with the query vector exceed a preset threshold are retained to obtain a sub-graph retrieval refinement result; the edge weight is the minimum value of the similarity between the query vector and two nodes on the same side; structured prompt words are generated according to the sub-graph retrieval refinement result; the result prompt words include: node content description and relationship description.

[0099] In one embodiment, the question-and-answer unit is further used to integrate user queries, role positioning, basic prompts of question-and-answer requirements, visual information, visual information confidence, and result prompt words into final prompt words; the final prompt words are input into the LLM model, information processing is completed through visual information confidence and basic prompts of question-and-answer requirements, and the question-and-answer style is adjusted through role positioning to obtain the final medical answer.

[0100] In one embodiment, the LLM model in the question-answering unit is the DeepSeek-v3 large language model.

[0101] Regarding the specific limitations of the medical visual question-answering device based on the knowledge graph and the large language model, please refer to the limitations of the medical visual question-answering method based on the knowledge graph and the large language model above, which will not be repeated here. The various modules in the above-mentioned medical visual question-answering device based on the knowledge graph and the large language model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0102] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the scope of the present application, and such modifications and improvements are all within the scope of protection of the present application.

Claims

1. A medical visual question answering method based on knowledge graph and large language model, characterized by: The method comprises: Based on the medical image, question text, and knowledge graph, a trained medical visual question answering classification model is used to process the medical image, question text, and knowledge graph to obtain a predicted answer; the medical visual question answering classification model is used to extract features from the medical image, knowledge graph, and question text using a multimodal encoder, and an improved fusion method after MCAN using attention compression and gating mechanisms. The extracted features are fused in multiple stages to generate a predicted answer based on the fused features; The question text is input into the knowledge graph retrieval enhancement generation module, and structured prompt words are obtained by searching the knowledge graph; the knowledge graph retrieval enhancement generation module is used to dynamically search the knowledge graph according to the question text and generate structured prompt words containing node attributes related to the question and knowledge graph relationship edges; The predicted answer, the structured prompt words, and the basic prompts including role positioning and question-answering requirements are integrated using LLM to obtain a final medical answer.

2. The medical visual question answering method based on knowledge graph and large language model according to claim 1 is characterized in that: The medical visual question answering classification model includes: a multimodal encoder, a feature fusion module and an answer generation module; Based on the medical image, question text, and knowledge graph, the trained medical visual question answering classification model is used to process and obtain the predicted answer, including: A multimodal encoder is used to encode features of medical images, question texts, and knowledge graphs to obtain image encoding features, question encoding features, and knowledge graph encoding features; Inputting the question encoding feature, the image encoding feature, and the knowledge graph encoding feature into the feature fusion module for multi-stage feature fusion to obtain a fused embedded feature; The fused embedding features are input into the answer generation module to obtain the predicted answer.

3. The medical visual question answering method based on knowledge graph and large language model according to claim 2 is characterized in that: The multimodal encoder includes an image encoder, a language encoder, and a knowledge graph encoder; A multimodal encoder is used to encode features of medical images, question texts, and knowledge graphs to obtain image encoding features, question encoding features, and knowledge graph encoding features, including: Input the question text into the language encoder, process the question text through BERT to obtain an initial embedding, and then use a dual-gated recurrent unit to reduce the dimension of the initial embedding to obtain the question encoding feature; Inputting the preprocessed medical image into the image encoder to obtain image coding features; The knowledge graph encoder is used to reconstruct the knowledge graph into an entity graph containing disease nodes, relationship nodes and attribute nodes. The disease nodes, relationship nodes and attribute nodes are connected according to clustering attributes to generate three types of edges: disease-attribute, relationship-attribute and attribute-attribute, and obtain the knowledge graph encoding features.

4. The medical visual question answering method based on knowledge graph and large language model according to claim 2, characterized in that: The feature fusion module includes the image-question fusion module, the image-disease fusion module, the question-relationship fusion module, the image-question-disease fusion module, and the question-image-relationship fusion module; The knowledge graph encoding features include disease features, attribute features and relationship features; The fused embedding features include image-question-disease features and question-image-relationship features: Inputting the question encoding feature, the image encoding feature, and the knowledge graph encoding feature into the feature fusion module for multi-stage feature fusion to obtain fused embedded features, including: Inputting the question coding feature and the image coding feature into the image-question fusion module to obtain question-image features; Inputting the image coding feature and the knowledge disease feature into an image-disease fusion module to obtain an image-disease feature; Inputting the question coding feature and the relationship feature into the question-relationship fusion module to obtain the question-relationship feature; Inputting the problem-image feature and the image-disease feature into the image-problem-disease fusion module to obtain the image-problem-disease feature; The question-relationship feature and the image-disease feature are input into the question-image-relationship module to obtain the question-image-relationship feature.

5. The medical visual question answering method based on knowledge graph and large language model according to claim 4 is characterized in that: The image-question fusion module is used to align the two modal information with each other through bidirectional cross attention using an improved fusion method based on MCAN, and perform weighted averaging on the aligned image coding features and question coding features to obtain image average features and question average features. A gating mechanism is used to process the image average features and question average features to obtain gating weights, and the image average features and question average features are spliced ​​and fused according to the gating weights to obtain question-image features.

6. The medical visual question answering method based on knowledge graph and large language model according to claim 2, characterized in that: The fused embedding features include image-question-disease features and question-image-relationship features; The fused embedding features are input into the answer generation module to obtain the predicted answer, including: The image-question-disease features and the question-image-relationship features are concatenated and normalized to obtain multiple semantic features; The multiple semantic features are adopted The layer is processed to obtain the question label prediction result; After normalizing the question label prediction result, binarization is performed according to a preset binarization threshold to obtain a binarized prediction result; The binarized prediction result is used to implement the index mapping of the knowledge graph using the argmat function to obtain the index result after binarization; The Format function is used to format the information of the knowledge graph node into a readable string description for the index result after binarization to obtain the predicted answer.

7. The medical visual question answering method based on knowledge graph and large language model according to claim 1 is characterized in that: The knowledge graph and the question text are input into the knowledge graph retrieval enhancement generation module to obtain structured prompt words, including: The knowledge graph is segmented into subgraphs to obtain multiple subgraphs, all subgraphs are embedded into text using the BERT model and then saved as cache files; The question text is encoded into a query vector using BERT, and the cosine similarity with each subgraph is calculated to obtain the similarity score between the question and the subgraph; Select the top-N subgraph center nodes with the highest similarity scores to obtain the set S; For each subgraph corresponding to a central node in the set S, retain the nodes and edges whose edge weight with the query vector exceeds a preset threshold, and obtain the subgraph retrieval refinement result; the edge weight is the minimum value of the similarity between the query vector and the two nodes on the same side; Structured prompt words are generated according to the subgraph search and refinement results; the result prompt words include: node content description and relationship description.

8. The medical visual question answering method based on knowledge graph and large language model according to claim 1, characterized in that: The predicted answer, the structured prompt words, and the basic prompts including role positioning and question-answering requirements are integrated using LLM to obtain the final medical answer, including: Integrate user query, role positioning, basic prompts of question and answer requirements, visual information, visual information confidence and result prompt words into the final prompt words; The final prompt word is input into the LLM model, and information processing is completed through basic prompts of visual information confidence and question-answering requirements. The question-answering style is adjusted through role positioning to obtain the final medical answer.

9. The medical visual question answering method based on knowledge graph and large language model according to claim 1, characterized in that: The LLM model is the DeepSeek-v3 large language model.

10. A medical visual question-answering device based on knowledge graph and large language model, characterized in that: The device comprises: The medical visual question answering classification unit uses a trained medical visual question answering classification model to process medical images, knowledge graphs, and question texts to obtain predicted answers. The medical visual question answering classification model uses a multimodal encoder to extract features from medical images, knowledge graphs, and question texts, and then fuses the extracted features in multiple stages using a fusion method based on an improved MCAN (Multi-Mode Canonical Detection) using attention compression and gating mechanisms to generate predicted answers based on the fused features. A knowledge graph retrieval enhancement unit is configured to input the knowledge graph and the question text into a knowledge graph retrieval enhancement generation module, and obtain structured prompt words by searching the knowledge graph; the knowledge graph retrieval enhancement generation module is configured to dynamically search the knowledge graph based on the question text and generate structured prompt words containing node attributes related to the question and knowledge graph relationship edges; The question-answering unit is used to integrate the predicted answer, the structured prompt words, and the basic prompts including role positioning and question-answering requirements using LLM to obtain a final medical answer.

Images