An adaptive medical visual question answering method, device and medium

Abstract

The present application relates to the field of artificial intelligence and medical information processing technology, and specifically discloses a self-adaptive medical visual question and answer method, device and medium, wherein the present application method realizes knowledge-driven reasoning through a graph-to-chain medical reasoning module, optimizes multi-modal alignment in combination with a medical knowledge enhancement hybrid module, and finally generates an interpretable medical answer by adjusting model parameters efficiently and adaptively in a low rank, thereby solving the reasoning illusion caused by the complexity of medical knowledge and the modal alignment difficulty problem caused by data scarcity in the existing medical visual question and answer system, and significantly improving the interpretability, reasoning accuracy and clinical practicability of the medical visual question and answer system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and medical information processing technology, in particular to a self-adaptive medical visual question answering method, device and medium. BACKGROUND

[0002] With the in-depth application of artificial intelligence technology in the medical field, medical visual question answering (Med-VQA) has gradually become an important tool for assisting clinical diagnosis. This task requires the model to combine medical images (such as X-ray films, CT scans) and natural language questions to generate accurate and explainable answers. Existing technologies mainly realize end-to-end training through multi-modal large language models (MLLMs), use attention mechanisms to locate key image areas, or introduce external knowledge graphs (KGs) to enhance the understanding of entities such as diseases and symptoms. For example, some methods retrieve disease-symptom relationships in the knowledge graph to assist the model in generating answers consistent with medical logic. However, these methods still face significant challenges in practical applications.

[0003] Although general large language models (LLMs) perform well in natural language processing, their pre-training data lacks professional medical knowledge, leading to the generation of "hallucinations" when answering open clinical questions, i.e., generating answers that are unrelated or even contradictory to the image content. For example, when analyzing chest X-rays, the model may mistakenly attribute tracheal deviation to patient posture rather than pathological factors. In addition, the analysis of medical images relies heavily on the accurate identification of subtle features such as lesion morphology and anatomical structure abnormalities. However, existing models have limitations in the deep fusion of visual and textual modalities, making it difficult to effectively align image local features with question semantics, causing modality interference. This problem is particularly prominent in complex cases, such as pathological judgments that require the combination of multiple image regions.

[0004] On the other hand, the scarcity of medical annotation data severely restricts model performance. Existing public datasets (such as VQA-RAD, SLAKE) are limited in size, and the annotation process relies on domain experts, which is costly. Traditional data augmentation methods (such as random cropping and rotation) can expand training samples, but they cannot capture the semantic association between medical images and text descriptions, limiting the improvement of model generalization ability. More importantly, existing Med-VQA systems rely on black-box reasoning mechanisms and lack logical tracing of the answer generation process. For example, the model may correctly judge "cortical atrophy", but it cannot clearly explain the image features (such as widened sulci and flattened gyri) that it relies on, which reduces the trust of clinicians in the results. The above defects collectively limit the reliability and practicality of existing technologies in real medical scenarios.

[0005] In summary, there is an urgent need for a self-adaptive medical visual question answering method, device and medium to solve the problems in the existing technology. SUMMARY

[0006] The application aims to provide an adaptive medical visual question answering method, device and medium, and the specific technical solutions are as follows:

[0007] An adaptive medical visual question answering method comprises the following steps:

[0008] Step S100: extracting core entities from a medical image and generating a knowledge chain prompt through a graph-to-chain medical reasoning module;

[0009] Step S200: fusing medical image and text features to generate a semantically enhanced hybrid image through a medical knowledge enhancement hybrid module;

[0010] Step S300: training a multi-modal large language model based on the enhanced dataset, optimizing model parameters through a joint loss function, and obtaining a medical visual question answering model;

[0011] Step S400: inputting the medical image and question text into the medical visual question answering model, and outputting an answer and corresponding visual explanation.

[0012] Optionally, in step S100, the knowledge chain prompt is generated, comprising the following steps:

[0013] Step S101: driving medical image analysis by using a structured prompt to extract a core entity set wherein, is the total number of entities, and each element in the set represents a medical entity;

[0014] Step S102: retrieving entities and relationships related to from a knowledge graph to generate a knowledge chain wherein, is the path length, represents the relationship between entities.

[0015] Optionally, in step S102, the entities and relationships related to are retrieved from the knowledge graph, and the process is as follows:

[0016] Constructing an entity relationship matrix of the knowledge graph represents the relationship between entity and entity , is the total number of relationship types;

[0017] Calculating entity state transition weights through an attention mechanism , and the expression is:

[0018] ;​

[0019] wherein, denotes the state transition weight from the th entity to the th entity in the th reasoning step, is the score of the relation , is an indicator function, which is 1 when , otherwise 0;

[0020] iteratively update the entity states , and finally aggregate to obtain , wherein, denotes the number of reasoning steps, is the step attention weight, is the total number of reasoning steps.

[0021] Optionally, in step S100, the knowledge graph is PrimeKG, and in the knowledge chain generation process, a path scoring mechanism is used to screen Top-K candidate entities;

[0022] Path scoring is the average value of the relation score in the path, and the expression is as follows:

[0023] ;

[0024] wherein, is the path length, and the path scoring is sorted to finally generate the knowledge chain most relevant to the input query.

[0025] Optionally, in step S200, a semantic-enhanced hybrid image is generated, including the following steps:

[0026] Step S201: divide the medical image and into non-overlapping image blocks, and extract visual features and , wherein ;

[0027] Step S202: calculate the semantic relevance score of each image block and the text feature, and the expression is as follows:

[0028] ;

[0029] wherein, and denote the semantic relevance score, is the text embedding, denotes the splicing operation, is a multi-layer perception;

[0030] Step S203: cropping the high-relevance region of the medical image according to the semantic relevance score and replacing the low-relevance region of the medical image to generate a mixed image . .

[0031] Optionally, in step S203, the replaced image region is: , i.e., when the coordinate is located in the maximum relevance region of the medical image , the remaining region retains the original content, the size of the cropped region is controlled by the hyperparameter , and the expression of the convolution kernel size is , , and are the image height and width, is the block size.

[0032] Optionally, in step S300, the joint loss function is expressed as follows:

[0033] ;

[0034] ;

[0035] ;

[0036] wherein and are weight coefficients, is the total token number of the answer sequence, represents the th token in the th sample answer sequence, represents the model prediction target token is the conditional probability of the next output token, represents the medical image of the th sample, is the question token sequence from the beginning to the current time , is the answer token sequence generated from the beginning to the previous time, represents the th visual feature in the medical image of the th sample; represents the corresponding visual explanation label; is the model parameter, ​to explain the total number of visual elements, denotes the text generation loss, denotes the visual alignment loss.

[0037] Optionally, in step S300, the generation process of the enhanced data set is as follows:

[0038] The knowledge chain is serialized into a text form and spliced with the original question as an enhanced input, in the format of:

[0039] ;

[0040] wherein, is the enhanced question of the entity, is the visual explanation, denotes the generated knowledge chain, denotes the target answer text corresponding to the enhanced input, is a separator.

[0041] In addition, the present application also includes a computer device comprising a memory and a processor;

[0042] The memory is used to store a computer program that can run on the processor;

[0043] The processor is used to implement the steps of the adaptive medical visual question and answer method described above when the computer program is executed.

[0044] In addition, the present application also includes a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the adaptive medical visual question and answer method described above.

[0045] The technical scheme of the present application has the following beneficial effects:

[0046] The present application proposes an adaptive medical visual question and answer method based on a large language model and a knowledge graph. The method of the present application trains a model using an enhanced data set, solves the reasoning illusion caused by the complexity of medical knowledge in the existing medical visual question and answer (Med-VQA) system, and solves the difficulty of modality alignment caused by data scarcity. The method of the present application also fuses medical image and text features through a medical knowledge enhancement hybrid module, significantly improves the explainability, reasoning accuracy and clinical practicability of the medical visual question and answer system.

[0047] In addition to the purposes, features and advantages described above, the present application has other purposes, features and advantages. The present application will be further described in detail below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the steps of the adaptive medical visual question answering method in a preferred embodiment of the present invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example:

[0052] like Figure 1 As shown, this embodiment provides an adaptive medical visual question answering method based on a large-scale language model and knowledge graph, including the following steps (S100 to S400):

[0053] Step S100: Extract core entities from medical images and generate knowledge chain hints using the graph-to-chain medical reasoning module.

[0054] Specifically, generating knowledge chain hints includes the following steps:

[0055] Step S101: Utilize structured prompts (MIP, Med-Insight Prompt) to drive medical image analysis and extract the core entity set. ,in, The total number of entities, and each element in the set Each represents a medical entity; this embodiment uses a multimodal large language model (MLLM) to obtain medical entities; in addition, this embodiment also aligns entity names through a medical ontology library (UMLS) to ensure consistency with the terminology in the knowledge graph;

[0056] Step S102: Retrieve information from the knowledge graph related to... Generate a knowledge chain by identifying related entities and relationships. ,in, For path length, It represents the relationship between entities.

[0057] Furthermore, retrieving data from the knowledge graph related to... The related entities and relationships are as follows:

[0058] The entity relationship matrix of the knowledge graph is constructed The entity is represented The relationship between the entity is , is the total number of relationship types;

[0059] The entity state transition weight is calculated through an attention mechanism , and the expression is as follows:

[0060] ;

[0061] wherein, represents the state transition weight of the i-th entity to the j-th entity in the k-th reasoning step, is the score of the relationship is an indicator function, and the value of the indicator function is 1 when , otherwise 0; the attention mechanism is generated by a multi-layer perception (MLP) and a Sigmoid function; The entity state is iteratively updated , and finally aggregated to obtain , wherein, represents the number of reasoning steps,

[0062] is the step attention weight, is the total number of reasoning steps. In the present embodiment, the knowledge graph preferably adopts PrimeKG (PrimeKG contains 17,080 disease entities and 4,050,249 relationships), and in the knowledge chain generation process, Top-K candidate entities are screened through a path scoring mechanism, the path score is the average value of the relationship scores in the path, and finally the knowledge chain most relevant to the input query is generated; The path score is the average value of the relationship scores in the path, and the expression is as follows:

[0063]

[0064] ; wherein, is the path length, and the knowledge chain most relevant to the input query is finally generated according to the path score ranking.

[0065]

[0066]

[0067] ​​​​​Step S200: Fuse the medical image and the text feature by the medical knowledge enhanced hybrid module to generate a semantic enhanced hybrid image.

[0068] Specifically, the semantic enhanced hybrid image is generated, including the following steps:

[0069] Step S201: Divide the medical image into non-overlapping image blocks, and extract visual features and , wherein ; it is to be noted that the PMC-CLIP encoder is used to extract the visual features in the embodiment;

[0070] Step S202: Calculate the semantic relevance score of each image block and the text feature, and the expression is as follows:

[0071] ;

[0072] wherein and represent the semantic relevance score, is the text embedding, represents the splicing operation, is a multi-layer perception machine; the dimension D of the text embedding in the embodiment is 768;

[0073] Step S203: According to the semantic relevance score, crop the high relevance area of the medical image and replace the low relevance area of the medical image to generate a hybrid image .

[0074] Further, the replaced image area is: satisfying , that is, when the coordinates are located in the maximum relevance area of the medical image , the remaining area retains the original content of , and the size of the cropped area is controlled by the hyperparameter ( ), and the expression of the convolution kernel size is , and are the image height and width, is the block size.

[0075] Step S300: Train the multi-modal large language model based on the enhanced data set, optimize the model parameters through the joint loss function, and obtain the medical visual question and answer model.

[0076] Specifically, the joint loss function​ The expression is as follows:

[0077] ;

[0078] ;

[0079] ;

[0080] wherein, and are weight coefficients, is the total number of tokens of the answer sequence, represents the token in the sample answer sequence, represents the model prediction target token is the conditional probability of the next output token, represents the medical image of the sample, the question token sequence from the beginning to the current time , is the answer token sequence generated from the beginning to the previous time, represents the visual feature in the medical image of the sample; represents the corresponding visual explanation label; is a model parameter, is the total number of visual explanation elements, represents a text generation loss, and the embodiment adopts a cross-entropy loss to maximize the generation probability of the answer sequence, represents a visual alignment loss, and the embodiment adopts a mean square error calculation to constrain the generated visual explanation to be consistent with the label.

[0081] Further, the generation process of the enhanced data set is as follows:

[0082] The knowledge chain is sequenced into a text form and spliced with the original question as an enhanced input, and the format is as follows:

[0083] ;

[0084] wherein, is the entity-enhanced question, is the visual explanation, represents the generated knowledge chain, represents the target answer text corresponding to the enhanced input, is a separator.

[0085] Further, the embodiment adopts a low-rank adaptation (LoRA) method to update only the low-rank matrix parameters in the model, significantly reducing the training cost and adapting to the medical field knowledge. The specific process of the low-rank adaptation (LoRA) method is as follows:

[0086] Inserting a low-rank matrix in the attention layer of the pre-trained model (rank r = 8), only updating and , the parameter quantity is reduced to 0.1% of the original model, using the RAdam optimizer, the learning rate is 2x10-5, and the global batch size is 128. Among them, represents a dimension reduction matrix, which is used to map the input feature from dimension to a low-rank subspace; represents a dimension increase matrix, which is used to map the low-rank feature back to the original dimension , represents the input feature dimension, which is taken as = 768 in the embodiment.

[0087] Step S400: input the medical image and the question text into the medical visual question answering model to output the answer and the corresponding visual explanation.

[0088] Specifically, in the embodiment, the knowledge chain is serialized into a text form, such as , and then input into the medical visual question answering model.

[0089] Examples of medical visual question answering models applying the method of the embodiment are as follows:

[0090] Example 1:

[0091] Question: Is there a <entity> abdominal aorta < / entity> visualization?

[0092] Knowledge chain 1: abdominal aorta → imaging technology → enhanced CT scan → imaging findings → visualization confirmation.

[0093] Answer: This is an abdominal axial enhanced CT scan image, which shows that the abdominal aorta is filled with contrast agent, confirming that the blood vessel is visualized. Yes.

[0094] Example 2:

[0095] Question: Is there <entity> calcification < / entity>?

[0096] Knowledge chain 1: calcification → imaging findings → high-density lesions → anatomical sites → abdominal aorta and iliac arteries.

[0097] Answer: There are high-density lesions in the abdominal aorta and iliac artery region that meet the characteristics of calcification. Yes.

[0098] Example 3:

[0099] Question: Is there a <entity>blunting of costophrenic angle< / entity>?

[0100] Knowledge chain 1: costophrenic angle -> imaging findings -> clear and sharp -> pathological correlation -> no pleural effusion.

[0101] Answer: Bilateral costophrenic angles are clear and sharp, indicating no blunting signs and no pleural effusion. No.

[0102] The embodiment of the application further discloses a computer device comprising a memory and a processor.

[0103] The memory is used to store a computer program which can run on the processor.

[0104] The processor is used to execute the computer program to realize the steps of the legal knowledge intelligent question-answering method and the adaptive medical visual question-answering method.

[0105] Illustratively, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device.

[0106] The computer device can be a mobile phone, a desktop computer, a notebook computer, a palm computer and a cloud server, etc. The computer device can include, but is not limited to, a processor, a memory. For example, the computer device can also include an input / output device, a network access device, a bus, etc.

[0107] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0108] The memory can be used to store the computer program and / or modules, and the processor realizes the computer program by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0109] The modules / units integrated in the computer device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can realize the steps of the above-mentioned various method embodiments when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the computer readable medium can include or exclude contents according to the requirements of legislation and patent practice in a jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0110] In addition, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the adaptive medical visual question and answer method.

[0111] The embodiment provides a self-adaptive medical visual question and answer method, knowledge-driven reasoning is realized through a graph-to-chain medical reasoning module, multi-modal alignment is optimized in combination with a medical knowledge enhancement hybrid module, model parameters are efficiently fine-tuned through low-rank adaptation (LoRA), and finally an interpretable medical answer is generated. The embodiment method solves the problems of reasoning illusion caused by the complexity of medical knowledge, difficulty in modal alignment caused by data scarcity, and insufficient accuracy of open question answers in the existing medical visual question and answer (Med-VQA) system, and significantly improves the interpretability, reasoning accuracy and clinical practicability of the medical visual question and answer system.

[0112] It should be noted that the apparatus embodiments described above are only illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided by the present application indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0113] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An adaptive medical visual question answering method, characterized in that, Includes the following steps: Step S100: Extract core entities from medical images and generate knowledge chain hints using the graph-to-chain medical reasoning module; Step S200: The medical image and text features are fused through the medical knowledge enhancement hybrid module to generate a semantically enhanced hybrid image; Step S300: Train a multimodal large-scale language model based on the enhanced dataset, optimize the model parameters through a joint loss function, and obtain a medical visual question answering model; Step S400: Input the medical image and question text into the medical visual question answering model, and output the answer and the corresponding visual explanation; In step S200, a semantically enhanced hybrid image is generated, including the following steps: Step S201: Transfer medical images and Divided into Extract visual features from non-overlapping image patches. and ,in ; Step S202: Calculate the semantic relevance score between each image patch and the text features, as shown in the following expression: ； in, and Indicates semantic relevance score, For text embedding, This indicates a splicing operation. It is a multilayer perceptron; Step S203: Crop the medical image based on the semantic relevance score. Highly relevant regions and replacement of medical images Low-correlation regions are used to generate mixed images. .

2. The adaptive medical visual question answering method according to claim 1, characterized in that, In step S100, knowledge chain hints are generated, including the following steps: Step S101: Utilize structured prompts to drive medical image analysis and extract the core entity set. in, The total number of entities, and each element in the set Each represents a medical entity; Step S102: Retrieve information from the knowledge graph related to... Generate a knowledge chain by identifying related entities and relationships. ,in, For path length, It represents the relationship between entities.

3. The adaptive medical visual question answering method according to claim 2, characterized in that, In step S102, the relevant information is retrieved from the knowledge graph. The relevant entities and relationships, and the process are as follows: Constructing the entity relation matrix of a knowledge graph Representing entities With entity The relationship between them is , This represents the total number of relation types. Calculate entity state transition weights using an attention mechanism. Its expression is: ； in, Indicates the first In the first step of reasoning The entity to the first State transition weights for each entity For relationship The score, For indicator functions, when The value of the indicator function is 1 if the time condition is met, and 0 otherwise. Iteratively update entity state Finally, the aggregation yields ,in, Indicates the number of reasoning steps. For attention weights in the steps, This represents the total number of reasoning steps.

4. The adaptive medical visual question answering method according to claim 3, characterized in that, In step S100, the knowledge graph is PrimeKG, and during the knowledge chain generation process, Top-K candidate entities are selected through a path scoring mechanism. Path scoring Score the relationship within the path The average value is expressed as follows: ； in, Given the path length, the paths are sorted according to their scores, and finally, the knowledge chain most relevant to the input query is generated.

5. The adaptive medical visual question answering method according to claim 4, characterized in that, In step S203, the image region to be replaced is: satisfying That is, when coordinates Located in medical images When the region of maximum relevance is identified, the remaining regions are retained. The original content, the size of the cropped area is determined by the hyperparameter. Control, kernel size The expression is , and For the image height and width, This refers to the block size.

6. The adaptive medical visual question answering method according to claim 5, characterized in that, In step S300, the joint loss function The expression is as follows: ； ； ； in, and These are the weighting coefficients. The total number of lexical units in the answer sequence. Indicates the first The first sample answer sequence in the nth sample answer sequence Each word element, The model predicts the target word. The conditional probability of the next output word. Indicates the first Medical images of a sample, From the beginning to the present moment Problem word sequence, The sequence of answer words generated from the beginning to the previous time step. Indicates the first In the medical images of the sample, the first One visual feature; This indicates the corresponding visual explanation label; For model parameters, To visually interpret the total number of elements, Indicates the text generation loss. This indicates visual alignment loss.

7. The adaptive medical visual question answering method according to claim 6, characterized in that, In step S300, the process of generating the enhanced dataset is as follows: The knowledge chain is serialized into text format and concatenated with the original question to form an enhanced input, in the following format: ； in, For the problems after entity enhancement, For visual explanation, This represents the generated knowledge chain. This represents the target answer text corresponding to the enhanced input. This is a separator.

8. A computer device, characterized in that, Including memory and processor; The memory is used to store computer programs that can run on the processor; When the processor executes the computer program, it implements the steps of the adaptive medical visual question answering method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the adaptive medical visual question-answering method according to any one of claims 1 to 7.

Images