Medical visual question and answer method and system based on knowledge graph

Abstract

The invention provides a medical visual question-answering method and system based on a knowledge graph, and relates to the technical field of data processing, and the method comprises the steps: obtaining question information inputted by a user; extracting image entity features from the image information through an image recognition model, and extracting text entity features and questioning intentions from the text information through a natural language recognition model; matching the image entity feature with a medical entity in a pre-constructed knowledge graph to obtain a first matching result; matching the text entity features with extended information of medical entities in a pre-constructed knowledge graph to obtain a second matching result; determining a medical entity corresponding to the question information based on the first matching result and the second matching result; and obtaining a medical answer corresponding to the question information according to the question intention and the determined medical entity. According to the method, the identification of different expression modes of the user in different scenes can be enhanced, so that the problem information of the user is analyzed more accurately, and the occurrence of entity ambiguity is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a medical visual question answering method and system based on a knowledge graph. Background Art

[0002] Medical Visual Question Answering (MedVQA) is an application that combines medical knowledge and computer vision technology. It allows users to ask questions by uploading medical images (such as X-rays, CT scans, MRI images, etc.), and the system analyzes the images and gives diagnostic suggestions or related information. This technology has potential application value in auxiliary diagnosis, medical education, and patient communication. Medical visual question answering based on knowledge graph is an advanced application form that not only relies on image recognition and natural language processing technology, but also uses structured medical knowledge graphs to enhance the system's understanding and reasoning capabilities. This method can more accurately analyze medical images and provide more accurate answers to specific questions. However, in actual use, it is found that since non-medical users usually lack relevant professional knowledge, the system cannot accurately identify the questions that the user actually wants to express through the information they input, which reduces the accuracy of the feedback.

[0003] Therefore, how to provide a medical visual question answering method and system with higher accuracy and practicality is a problem that needs to be solved urgently. Summary of the invention

[0004] In order to improve the above problems, the present invention provides a medical visual question answering method and system based on knowledge graph.

[0005] A first aspect of an embodiment of the present invention provides a medical visual question answering method based on a knowledge graph, the method comprising: Acquire question information input by a user, wherein the question information includes image information and text information; Extracting image entity features from the image information by using an image recognition model, and extracting text entity features and question intention from the text information by using a natural language recognition model; Matching the image entity features with medical entities in a pre-constructed knowledge graph to obtain a first matching result; Matching the text entity feature with the extended information of the medical entity in the pre-constructed knowledge graph to obtain a second matching result; Determine a medical entity corresponding to the question information based on the first matching result and the second matching result; A medical answer corresponding to the question information is obtained according to the question intention and the determined medical entity.

[0006] Optionally, the method for constructing the knowledge graph specifically includes: Obtain medical image data and annotate the images; Acquire medical text data, annotate medical entities in the medical text data, and establish an association relationship between the medical entities and medical image data based on the annotations; Extract medical entities and their relationships from medical text data; Extended information is added to each medical entity according to the medical image data associated with the medical entity and the context in the medical text data.

[0007] Optionally, the step of adding extended information to each medical entity according to the medical image data associated with the medical entity and the context in the medical text data specifically includes: Determine a current context corresponding to the medical text data according to the context of the medical entity in the medical text data; Obtaining an extended word of the medical entity in the current context, wherein the extended word includes at least one of a synonym, an antonym, and an associated medical entity; The extended term is added as extended information of the medical entity.

[0008] Optionally, the step of matching the image entity features with medical entities in a pre-constructed knowledge graph to obtain a first matching result specifically includes: Calculate entity similarity between the image entity features and the embedding vectors of each medical entity in the knowledge graph; According to the calculation results, multiple medical entities whose entity similarity is greater than a preset first similarity threshold are selected; The selected multiple medical entities are sorted according to entity similarity as the first matching result.

[0009] Optionally, the step of matching the text entity feature with the extended information of the medical entity in the pre-constructed knowledge graph to obtain a second matching result specifically includes: Respectively obtain extended information of each medical entity in the first matching result; Calculating information similarity between the text entity features and the extended information of each medical entity in turn; According to the information similarity calculation result, the multiple medical entities are sorted according to the information similarity as the second matching result.

[0010] Optionally, the step of determining the medical entity corresponding to the question information based on the first matching result and the second matching result specifically includes: According to the sorting order of the second matching results, a similarity correction value for the first matching result is obtained according to a preset correction weight ratio; Based on the first matching result, the entity similarity of each medical entity is corrected by respectively adding the similarity correction value; The medical entity with the highest corrected entity similarity is used as the medical entity corresponding to the question information.

[0011] Optionally, the step of determining the medical entity corresponding to the question information based on the first matching result and the second matching result further includes: If the corrected entity similarities are all less than a preset second similarity threshold, generating a guidance prompt according to the text entity feature; Acquire supplementary text information input by the user based on the guidance prompt; extracting supplementary text features from the supplementary text information; Matching the supplementary text feature with the extended information of each medical entity in the first matching result to obtain a third matching result; The entity similarity of each medical entity in the first matching result is supplemented and corrected by the third matching result.

[0012] Optionally, the step of determining the medical entity corresponding to the question information based on the first matching result and the second matching result further includes: Get the user's case information; Extract case entity features from case information; Matching the case entity feature with the extended information of each medical entity in the first matching result to obtain a fourth matching result; The entity similarity of each medical entity in the first matching result is supplemented and corrected through the fourth matching result.

[0013] Optionally, the step of extracting image entity features from the image information by using an image recognition model specifically includes: When there are multiple image entity features extracted from the image information by the image recognition model, generating a confirmation prompt according to the image entity features; Acquire confirmation information input by the user based on the confirmation prompt; The image entity features confirmed by the user are screened out from a plurality of image entity features according to the confirmation information.

[0014] A second aspect of an embodiment of the present invention provides a medical visual question answering system based on a knowledge graph, including: An information acquisition unit, used to acquire question information input by a user, wherein the question information includes image information and text information; A feature extraction unit, used to extract image entity features from the image information through an image recognition model, and to extract text entity features and question intention from the text information through a natural language recognition model; A feature matching unit, used to match the image entity feature with the medical entity in the pre-constructed knowledge graph to obtain a first matching result; The feature matching unit is further used to match the text entity feature with the extended information of the medical entity in the pre-constructed knowledge graph to obtain a second matching result; An entity confirmation unit, configured to determine a medical entity corresponding to the question information based on the first matching result and the second matching result; The answer generation unit is used to obtain a medical answer corresponding to the question information according to the question intention and the determined medical entity.

[0015] In summary, the present invention provides a medical visual question-answering method and system based on knowledge graph, which can enhance the recognition of different expressions of users in different scenarios during the question-answering process, and then more accurately identify and parse the user's question information, reduce the occurrence of entity ambiguity, and effectively improve the overall performance and user experience of the medical visual question-answering system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A method flow chart of a medical visual question answering method based on a knowledge graph according to an embodiment of the present invention; Figure 2 This is a functional module block diagram of a medical visual question answering system based on a knowledge graph according to an embodiment of the present invention.

[0018] Reference numerals: Information acquisition unit 110; feature extraction unit 120; feature matching unit 130; entity confirmation unit 140; answer generation unit 150. DETAILED DESCRIPTION

[0019] Medical Visual Question Answering (MedVQA) is an application that combines medical knowledge and computer vision technology. It allows users to ask questions by uploading medical images (such as X-rays, CT scans, MRI images, etc.), and the system analyzes the images and gives diagnostic suggestions or related information. This technology has potential application value in auxiliary diagnosis, medical education, and patient communication. Medical visual question answering based on knowledge graph is an advanced application form that not only relies on image recognition and natural language processing technology, but also uses structured medical knowledge graphs to enhance the system's understanding and reasoning capabilities. This method can more accurately analyze medical images and provide more accurate answers to specific questions. However, in actual use, it is found that since non-medical users usually lack relevant professional knowledge, the system cannot accurately identify the questions that the user actually wants to express through the information they input, which reduces the accuracy of the feedback.

[0020] Therefore, how to provide a medical visual question answering method and system with higher accuracy and practicality is a problem that needs to be solved urgently.

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0024] In the description of the present invention, the terms “first”, “second”, etc. are only used to distinguish the descriptions and should not be understood as indicating or implying relative importance.

[0025] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0026] The following is a detailed description of a medical visual question answering method based on a knowledge graph provided in this embodiment.

[0027] See also Figure 1 , this embodiment provides a medical visual question answering method based on a knowledge graph, the method comprising: Step S101, obtaining question information input by a user, wherein the question information includes image information and text information.

[0028] Through user interface design, image uploading and text input functions can be realized. Both web applications and mobile applications can support these operations. The image information is the image file uploaded by the user that needs to be recognized, and the text information is the text description of the user's specific question.

[0029] It should be noted that the input of text information can be done by the user directly entering the text content, or by voice input, that is, the user uploads voice information, and the text content is obtained by recognizing the voice information, and then after modification or confirmation by the user, it is used as the text information input.

[0030] Step S102: extracting image entity features from the image information through an image recognition model, and extracting text entity features and question intention from the text information through a natural language recognition model.

[0031] The image information uploaded by users is usually of common medical image types, such as X-rays, CT scans, and MRI images. In some scenarios, the uploaded image information may also be photos taken by the user himself. Therefore, it is necessary to select a suitable image recognition model for the image information uploaded by the user. First, determine whether the uploaded image information belongs to the medical image type. If so, use the image recognition model for medical images. If it is a photo, use the image recognition model for pictures. Using pre-trained deep learning models (such as ResNet, DenseNet, or Transformer architecture) can effectively extract features from medical images.

[0032] It should be noted that the image information uploaded by the user may include relatively rich content, and the image entity features extracted by the image recognition model may include multiple ones. When the user asks a question, he or she is mainly targeting one or a category of them. Therefore, further confirmation is required to clarify which identified one the user is mainly targeting in the current question.

[0033] Specifically, as a preferred embodiment of the present invention, step S102 specifically includes: When there are multiple image entity features extracted from the image information by the image recognition model, generating a confirmation prompt according to the image entity features; Acquire confirmation information input by the user based on the confirmation prompt; The image entity features confirmed by the user are screened out from a plurality of image entity features according to the confirmation information.

[0034] When the image recognition model outputs multiple image entity features, generating a confirmation prompt can help the system ensure that the most relevant image entity feature is selected. The content of the confirmation prompt is related to the output result of the image recognition model. The confirmation prompt can be output in text form. For example, if multiple lesion areas are detected, the user can be prompted: "Please confirm where the shadow you are concerned about is located?". It can also be combined with visual assistance, such as providing image annotation or highlighting in the confirmation prompt to help users more intuitively select the correct image entity feature. Specifically, different lesion areas are marked in the image, and the user is asked to select the most relevant one.

[0035] The confirmation information entered by the user corresponds to the content of the confirmation prompt. It should be noted that the user may enter useless information at this time. Therefore, the confirmation information entered by the user needs to be identified to ensure that the confirmation information provided by the user is valuable. Information identified as useless will not be used as confirmation information.

[0036] As a preferred implementation, during the process of the user inputting confirmation information, a real-time feedback mechanism can be provided to dynamically generate relevant questions or options based on the user's initial input to further guide the user to provide more information, or to help the user better understand how to provide useful information through simple prompts or examples.

[0037] When the user provides available confirmation information, the most relevant image entity features are screened out according to the user's confirmation information, and used as the image entity features extracted from the image information in the current problem to participate in subsequent comparisons. For example, the user selects one specific area from multiple marks.

[0038] The extraction of text entity features and question intentions can be achieved through advanced NLP models (such as BERT or other bidirectional Transformer architectures), which can effectively parse user text input. It should be noted that different models can be used to implement text entity features and question intentions. For text entity features, an entity realization model is required. Common models include BiLSTM-CRF: combining bidirectional LSTM and conditional random fields, suitable for sequence labeling tasks; BERT: a pre-trained language model suitable for entity recognition in the medical field; BioBERT: a BERT model optimized for biomedical texts. For question intentions, an intent recognition model is required. Common models include BERT: suitable for text classification tasks; GPT: a generative pre-trained model suitable for understanding complex intentions; ClinicalBERT: a BERT model optimized for clinical texts.

[0039] In this embodiment, the medical entities in the constructed knowledge graph are all associated with rich extended information, which helps to more accurately match the medical entities and the objects described in the user question information. The following is an explanation of the method for constructing the knowledge graph.

[0040] As a preferred embodiment of the present invention, the method for constructing the knowledge graph specifically includes: Obtain medical image data and annotate the images; Acquire medical text data, annotate medical entities in the medical text data, and establish an association relationship between the medical entities and medical image data based on the annotations; Extract medical entities and their relationships from medical text data; Extended information is added to each medical entity according to the medical image data associated with the medical entity and the context in the medical text data.

[0041] Medical image data (such as X-rays, CT scans, etc.) and medical text data (including medical literature, electronic health records, clinical guidelines and drug instructions, hospital question-and-answer sessions, etc.) can be obtained through legal and regulatory channels in cooperation with medical institutions or professional data providers, and de-identification technology can be used to protect personal privacy. The annotation process for medical image data can be completed manually with the assistance of machine learning algorithms.

[0042] Establishing the association between text and image data needs to be implemented in steps. First, apply existing medical NLP tools (such as cTAKES, MedTagger) for preprocessing and annotation, and match and associate the annotation results of text data with the annotation of medical image data. Among them, the preprocessing part includes removing irrelevant characters, punctuation marks, digital conversion, etc., and then performing operations such as word segmentation, part-of-speech tagging, and named entity recognition (NER). When annotating, you can use existing medical knowledge graphs or professional dictionaries to assist in identifying medical terms, disease names, drug information, etc. in the text. At the same time, you can use machine learning models or deep learning frameworks (such as BERT, BioBERT) to improve the accuracy of entity recognition, and finally use experts to manually correct the results of automatic annotation to ensure high accuracy. When associating, the disease type is usually used as the association factor. By using natural language processing technology, information such as disease type and condition description is extracted from the text, and it is associated with the abnormal part on the image of the same type of disease.

[0043] In this embodiment, the operation of extracting medical entities and their relationships is performed in the same way as other knowledge graphs, and the main difference is reflected in the subsequent addition of extended information. The specific operation process includes two parts: medical entity recognition and relationship extraction. First, the medical entities in the text are identified through existing medical named entity recognition tools or models (such as cTAKES, MetaMap, BioBERT), and then dependency syntax analysis tools (such as Stanford NLP, SpaCy) are used to parse the sentence structure and identify the relationship between entities. Based on known relationship patterns (such as diseases and symptoms, drugs and side effects, etc.), relationships are extracted through regular expressions or template matching methods.

[0044] When adding extended information, the influence of the associated medical image data and the context in the medical text data should be fully considered. Specifically, as a preferred embodiment of the present invention, the step of adding extended information to each medical entity according to the medical image data associated with the medical entity and the context in the medical text data specifically includes: Determine a current context corresponding to the medical text data according to the context of the medical entity in the medical text data; Obtaining an extended word of the medical entity in the current context according to the associated medical image data, wherein the extended word includes at least one of a synonym, an antonym, and an associated medical entity; The extended term is added as extended information of the medical entity.

[0045] First, natural language processing techniques (such as dependency parsing and coreference resolution) are used to understand the contextual relationship of medical entities in the sentence, and then the current context is determined based on the contextual information. For example, if "chest pain" appears in the context of describing nocturnal attacks and dyspnea, its context can be determined as "chest pain that occurs at night, accompanied by dyspnea".

[0046] Taking into account the differences in the description methods of the same medical entity in different contexts, combined with the annotation information of the associated image data, the extended terms of the medical entity in each context in which it appeared can be generated.

[0047] For example, suppose there are two pieces of medical text data and the associated medical image data: Text 1: "The patient has a long history of smoking and recently developed chest pain symptoms. Chest CT showed a shadow in the right upper lobe of the lung." Image 1: The shaded area of ​​the right upper lobe is marked, which is approximately 2 cm in size and irregular in shape.

[0048] Text 2: "The patient recently experienced chest pain during the day, accompanied by cough, and no obvious abnormalities were found in chest X-ray." Image 2: Chest X-ray, no obvious abnormalities were seen.

[0049] The medical entity is identified as "chest pain", and two contexts are obtained: Context 1: "Chest pain that occurs at night, accompanied by dyspnea".

[0050] Context 2: "Chest pain that occurs during the day, accompanied by cough".

[0051] For context 1, the extended words of the current context are generated, including synonyms - "chest discomfort" and "dyspnea", and related medical entities - "angina pectoris" and "myocardial infarction". For context 2, the extended words of the current context are generated, including synonyms - "chest discomfort" and "cough", and related medical entities - "angina pectoris" and "gastroesophageal reflux disease".

[0052] For the same extended words that appear in different contexts, duplicate removal is performed.

[0053] Through the above process, extended information can be added to each medical entity in the knowledge graph, thereby completing the construction of the knowledge graph in the embodiment of the present invention.

[0054] Step S103, matching the image entity features with the medical entities in the pre-constructed knowledge graph to obtain a first matching result.

[0055] Matching image entity features with medical entities in a pre-built knowledge graph can effectively find the most relevant medical entity. This is a common operation method in existing medical visual question-answering systems based on knowledge graphs. However, simply matching image entity features with medical entities in knowledge graphs is prone to entity ambiguity. Therefore, in this embodiment, matching image entity features with medical entities in a pre-built knowledge graph is the first step in confirming entities in user text.

[0056] Specifically, as a preferred method of the embodiment of the present invention, step S103 specifically includes: Calculate entity similarity between the image entity features and the embedding vectors of each medical entity in the knowledge graph; According to the calculation results, multiple medical entities whose entity similarity is greater than a preset first similarity threshold are selected; The selected multiple medical entities are sorted according to entity similarity as the first matching result.

[0057] Pre-trained deep learning models (such as ResNet, DenseNet, or Transformer architectures) can be used to extract image entity features and convert them into embedded vectors. Medical entities in the knowledge graph can also be embedded in a similar way. The similarity between the image entity features and the embedded vectors of the medical entities in the knowledge graph is calculated. Common similarity calculation methods include cosine similarity, Euclidean distance, etc.

[0058] By setting the first similarity threshold, irrelevant medical entities can be filtered out and those entities that are highly relevant to the image features can be retained. This method helps to improve the accuracy of the system. The specific value of the first similarity threshold is determined based on the current user's usage or user group. For example, if the user generally has a poor medical foundation and the quality of the image data provided is not high, then the value of the first similarity threshold can be set lower.

[0059] Sorting the matching results of medical entities according to entity similarity can intuitively display the most relevant medical entities, facilitating subsequent processing and generating answers.

[0060] For example, suppose a user uploads a chest CT image and enters the question: "There are shadows in the lungs, what does this mean?" The first similarity threshold is set to 0.8. After similarity calculation, multiple medical entities with entity similarity greater than the threshold are screened out. Suppose the screened results include "benign nodules" (entity similarity 0.9), "malignant tumors" (entity similarity 0.82), and "inflammation" (entity similarity 0.85). Finally, the screened medical entities are sorted according to entity similarity, and the first matching result is obtained: "benign nodules" (entity similarity 0.9), "inflammation" (entity similarity 0.85), and "malignant tumors" (entity similarity 0.82).

[0061] Step S104, matching the text entity feature with the extended information of the medical entity in the pre-constructed knowledge graph to obtain a second matching result.

[0062] Here, the matching object of the text entity feature is the extended information of the medical entity rather than the medical entity itself. The main consideration is that the text entity feature is extracted from the text information uploaded by the user. The text information uploaded by the user is usually greatly affected by the user's current environment and the user's own mastery of medical-related professional knowledge. Compared with professional expressions, the content of its expression may deviate to a large extent. If the text entity feature is directly matched with the medical entity, the deviation between the matching result and the user's actual expression object will also be large, which will lead to the system being unable to accurately identify the problem that the user actually wants to express through the input information. Accordingly, when the extended information of the medical entity is generated, the different contexts are fully considered, which can better cover the user's non-professional expression. Therefore, matching the text entity feature with the extended information of the medical entity in the pre-constructed knowledge graph can achieve better matching results.

[0063] Specifically, as a preferred method of the embodiment of the present invention, step S104 specifically includes: Respectively obtain extended information of each medical entity in the first matching result; Calculating information similarity between the text entity features and the extended information of each medical entity in turn; According to the information similarity calculation result, the multiple medical entities are sorted according to the information similarity as the second matching result.

[0064] Through natural language processing technology, the similarity between text entity features and the extended information of medical entities can be calculated. Commonly used methods include word vector models (such as Word2Vec, GloVe), pre-trained language models (such as BERT), and TF-IDF.

[0065] Continuing with the above example: For the first matching result: "benign nodule" (entity similarity 0.9), "inflammation" (entity similarity 0.85), and "malignant tumor" (entity similarity 0.82), the extended information is obtained respectively, and then the information similarity between the text entity features provided by the user and the extended information of each medical entity is calculated, and the medical entities are sorted according to the information similarity score to obtain the second matching result: "inflammation" (information similarity 0.88), "benign nodule" (information similarity 0.85), and "malignant tumor" (information similarity 0.79).

[0066] Step S105: determining a medical entity corresponding to the question information based on the first matching result and the second matching result.

[0067] By comprehensively considering image and text information, the most likely medical entity can be determined more accurately. On this basis, a weighted fusion strategy is used to adjust the weight of the matching results according to the confidence of image and text information.

[0068] Specifically, the above step S105 specifically includes: According to the sorting order of the second matching results, a similarity correction value for the first matching result is obtained according to a preset correction weight ratio; Based on the first matching result, the entity similarity of each medical entity is corrected by respectively adding the similarity correction value; The medical entity with the highest corrected entity similarity is used as the medical entity corresponding to the question information.

[0069] When adjusting the entity similarity of the first matching result by setting the correction weight ratio, different weights are set according to the importance of the second matching result (i.e., the matching of the text entity feature and the extended information). It should be noted that the correction weight strategy needs to be set reasonably to ensure that it can effectively improve the accuracy of the matching results without introducing bias.

[0070] The calculation of the similarity correction value may adopt different correction weight ratios. For example, according to the sorting order of the second matching results, a fixed similarity correction value may be assigned to each information similarity in the order from high to low. It is also possible to assign a weight coefficient to each information similarity in the order from high to low according to the sorting order of the second matching results, and multiply the information similarity by the weight coefficient to obtain the similarity correction value corresponding to each information similarity. It should be noted that the correction weight ratio may be dynamically adjusted according to the specific application scenario and the characteristics of the data set.

[0071] When superimposing the similarity correction value on the entity similarity, you can choose the simplest direct summation, or you can use the weighted summation method to determine the weighted ratio of the entity similarity and the similarity correction value according to the specific application scenario and the characteristics of the data set.

[0072] Continuing with the above example: Calculate the first matching result: Benign nodules (entity similarity 0.9), inflammation (entity similarity 0.85), and malignant tumors (entity similarity 0.82).

[0073] The second matching result is calculated: Inflammation (information similarity 0.88), benign nodules (information similarity 0.85), and malignant tumors (information similarity 0.79).

[0074] Assuming that the information similarity is directly used as the similarity correction value at this time, the weighted ratio of the two when superimposed is: entity similarity weight 0.6, similarity correction value 0.4.

[0075] The calculation of superposition is as follows: 1. For "benign nodules": Entity similarity: 0.9; Similarity correction value: 0.85; Corrected entity similarity = 0.6 * 0.9 + 0.4 * 0.85 = 0.88; 2. For "inflammation": Entity similarity: 0.85; Similarity correction value: 0.88; Corrected entity similarity = 0.6 * 0.85 + 0.4 * 0.88 = 0.862; 3. For “malignant tumors”: Entity similarity: 0.82; Similarity correction value: 0.79; Corrected entity similarity = 0.6 * 0.82 + 0.4 * 0.79 = 0.808; Select final answer: According to the corrected entity similarity scores, the medical entity with the highest corrected similarity was selected: “benign nodule” (similarity 0.88).

[0076] On the basis of the above scheme, some special cases need to be considered, that is, the corrected entity similarities are not high, and other methods are needed to improve the accuracy of entity matching. Specifically, as a preferred implementation, if the corrected entity similarities are less than a preset second similarity threshold, a guide prompt is generated according to the text entity feature; Acquire supplementary text information input by the user based on the guidance prompt; extracting supplementary text features from the supplementary text information; Matching the supplementary text feature with the extended information of each medical entity in the first matching result to obtain a third matching result; The entity similarity of each medical entity in the first matching result is supplemented and corrected by the third matching result.

[0077] Based on the requirements for the overall accuracy of the system, the second similarity threshold can be set to a relatively high value, and the determination of this value can be based on the actual situation of multiple uses during the execution of this method. When the corrected entity similarities are less than the preset second similarity threshold, it means that the existing recognition accuracy does not meet the requirements. At this time, the user needs to be guided to enter more useful information. The content of the guidance prompt is generated based on the text entity features. For example, the text entity feature has the term "shadow", which can prompt the user: "Please describe the specific location and shape of the shadow." Multiple rounds of dialogue can also be introduced on this basis to gradually guide the user to provide more information to ensure that there is enough background data for accurate matching.

[0078] The method of extracting supplementary text features from the supplementary text information is the same as the method of extracting text entity features from text information, and will not be repeated here. The method of matching the extended information of each medical entity in the first matching result of the supplementary text features is also the same as matching the text entity features with the extended information of the medical entity in the pre-constructed knowledge graph to obtain the second matching result. Because the supplementary text information is a supplement to the text information in the question information, the same feature extraction and matching methods can be used for both.

[0079] It should be noted that the operation of supplementing and correcting the entity similarity of each medical entity in the first matching result through the third matching result is based on the result of the last supplementary correction. Therefore, the correction weight ratio, weighted ratio, and correction value calculation method used in the supplementary correction can be the same as those used for correction based on the second matching result. As other implementations, different correction weight ratios can also be used for calculation.

[0080] Continuing with the above example: Assume that the preset second similarity threshold is 0.90, and all corrected entity similarities are lower than the threshold.

[0081] Then do the following: 1. Generate boot prompt: Guidance prompt: "Please describe the exact location and shape of the shadow." 2. Obtain user supplementary text information: User input: "The shadow is located in the right upper lobe of the lung, approximately 2 cm in size and irregular in shape." 3. Extracting supplementary text features: Use a pre-trained medical language model to extract supplementary text features: “right upper lobe”, “2 cm”, “irregular shape”.

[0082] 4. Calculate the third matching result: The supplementary text features are matched with the extended information of each medical entity in the first matching result to obtain the result: benign nodules (similarity 0.85); inflammation (similarity 0.87); Malignant tumor (similarity 0.82); 5. Supplementary amendments: Select a weight ratio of 0.1 to calculate the correction value, and choose the direct summation method to make additional corrections.

[0083] The corrected weighted similarity is supplemented and corrected according to the third matching result: Benign nodules (supplemented corrected entity similarity 0.88 + 0.1*0.85 = 0.965); Inflammation (supplemented corrected entity similarity 0.862 + 0.1*0.87 = 0.949); Malignant tumor (supplemented corrected entity similarity 0.808 + 0.1*0.82 = 0.89); According to the results of supplementary correction, the medical entity with the highest entity similarity after supplementary correction was selected: “benign nodule” (supplementary and corrected entity similarity 0.965).

[0084] As a preferred implementation, in order to further improve the accuracy of the method, the embodiment of the present invention also provides a parallel optimization method with the above supplementary correction. Specifically, the method includes: Get the user's case information; Extract case entity features from case information; Matching the case entity feature with the extended information of each medical entity in the first matching result to obtain a fourth matching result; The entity similarity of each medical entity in the first matching result is supplemented and corrected through the fourth matching result.

[0085] The features of the user's case information are very likely to be associated with the user's current question content. Therefore, extracting case entity features based on the user's case information and adding them to the correction of matching results can effectively improve accuracy.

[0086] The calculation method and supplementary correction method of the fourth matching result are the same as those of the third matching result mentioned above and will not be repeated here. The main difference between the two lies in the source of feature extraction, one is based on the user's supplementary information and the other is based on the user's case information.

[0087] It should be noted that when correcting the entity similarity of each medical entity in the first matching result, whether through the second matching result, the third matching result or the fourth matching result, the same correction weight ratio, weighted ratio, and correction value calculation method can be used at the same time, or different strategies or values ​​can be selected respectively.

[0088] Continuing with the above example: Assume that the entity similarity corrected by the second matching result is as follows: Benign nodules (corrected entity similarity 0.88); Inflammation (corrected entity similarity 0.862); Malignant tumor (corrected entity similarity 0.808); Next, obtain the user's case information and extract case entity features from it: 1. Get the user's case information: User case information: "The patient has a long history of smoking and recently developed chest pain symptoms. Chest CT showed a shadow in the right upper lobe of the lung." 2. Extract case entity features from case information: Use pre-trained medical language models (such as BioBERT) to extract case entity features: "long-term smoking history", "chest pain", "right upper lobe of the lung", "shadow".

[0089] 3. Calculate the fourth matching result: Match the case entity features with the extended information of each medical entity in the first matching result: benign nodules (similarity 0.85); inflammation (similarity 0.87); Malignant tumor (similarity 0.89); 4. Supplementary amendments: The corrected entity similarity is supplemented and corrected according to the fourth matching result: benign nodules (final similarity 0.88 + 0.85 = 1.72); inflammation (final similarity 0.862 + 0.87 = 1.732); Malignant tumor (final similarity 0.808 + 0.89 = 1.617); Select final answer: According to the corrected similarity score, the medical entity with the highest corrected similarity was selected: “inflammation” (final similarity 1.732).

[0090] Step S106: obtaining a medical answer corresponding to the question information according to the question intention and the determined medical entity.

[0091] Based on the constructed knowledge graph, combined with the question intention and the matching results, a detailed medical answer can be generated. As a preferred implementation, a template generation technology can be used to generate structured answers according to different question intentions.

[0092] Continuing with the above example: Based on the above matching results and the question intention, a detailed medical answer is generated: "Based on the symptoms and imaging features you provided, inflammation is most likely. It is recommended that you consult a doctor for further examination (such as blood tests, PET-CT, etc.) to confirm the specific situation." The above is a medical visual question-answering method based on knowledge graph provided by an embodiment of the present invention, which uses confirmation prompts and user feedback to accurately screen out the most relevant image entity features. This method not only improves the accuracy of the system, but also enhances the user experience. Compared with the existing medical visual question-answering solution based on knowledge graph, it has the following advantages: 1. Enhanced accuracy and relevance; By generating confirmation prompts and obtaining direct feedback from users, the system can more accurately understand the user's concerns and provide more precise answers and services. This approach avoids the problem of misjudgment or incomplete matching that may be caused by relying solely on algorithms, especially when dealing with complex medical images.

[0093] 2. Improve user experience; The user participation process makes the entire interactive process more transparent and intuitive, increasing the user's trust in the system. Providing visual aids (such as highlighting, marking, etc.) allows users to more clearly see and select the specific areas they care about, further improving the user experience.

[0094] 3. Flexibility and adaptability; The system can flexibly adjust its workflow according to different input situations, whether it is a simple single feature or a complex multi-feature scenario. Through multi-round dialogue management and real-time feedback mechanism, the system can dynamically adjust questions or options based on the information provided by the user to better meet user needs.

[0095] 4. Reduce data dependence; Compared with the knowledge graph method that relies on large-scale annotated data, this solution reduces the reliance on pre-built large databases, especially in specific fields or rare diseases. Using data from users' instant feedback can make up for the problem of insufficient data to a certain extent.

[0096] In summary, the medical visual question-answering method based on knowledge graph provided by the implementation of the present invention can enhance the recognition of different expressions of users in different scenarios during the question-answering process, thereby more accurately identifying and parsing user question information, reducing the occurrence of entity ambiguity, and effectively improving the overall performance and user experience of the medical visual question-answering system.

[0097] like Figure 2 As shown, the medical visual question answering system based on knowledge graph provided by the present invention comprises: An information acquisition unit 110 is used to acquire question information input by a user, wherein the question information includes image information and text information; A feature extraction unit 120, configured to extract image entity features from the image information by using an image recognition model, and to extract text entity features and question intention from the text information by using a natural language recognition model; A feature matching unit 130, configured to match the image entity feature with a medical entity in a pre-constructed knowledge graph to obtain a first matching result; The feature matching unit 130 is further used to match the text entity feature with the extended information of the medical entity in the pre-constructed knowledge graph to obtain a second matching result; An entity confirmation unit 140, configured to determine a medical entity corresponding to the question information based on the first matching result and the second matching result; The answer generation unit 150 is used to obtain a medical answer corresponding to the question information according to the question intention and the determined medical entity.

[0098] The medical visual question answering system based on knowledge graph provided in an embodiment of the present invention is used to implement the above-mentioned medical visual question answering method based on knowledge graph. Therefore, the specific implementation method is the same as the above-mentioned method and will not be repeated here.

[0099] In summary, the present invention provides a medical visual question-answering method and system based on knowledge graph, which can enhance the recognition of different expressions of users in different scenarios during the question-answering process, and then more accurately identify and parse the user's question information, reduce the occurrence of entity ambiguity, and effectively improve the overall performance and user experience of the medical visual question-answering system.

[0100] In several embodiments disclosed in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0102] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A medical visual question answering method based on knowledge graph, characterized in that: The method comprises: Acquire question information input by a user, wherein the question information includes image information and text information; Extracting image entity features from the image information by using an image recognition model, and extracting text entity features and question intention from the text information by using a natural language recognition model; Matching the image entity features with medical entities in a pre-constructed knowledge graph to obtain a first matching result; Matching the text entity feature with the extended information of the medical entity in the pre-constructed knowledge graph to obtain a second matching result; Determine a medical entity corresponding to the question information based on the first matching result and the second matching result; A medical answer corresponding to the question information is obtained according to the question intention and the determined medical entity.

2. The medical visual question answering method based on knowledge graph according to claim 1, characterized in that: The method for constructing the knowledge graph specifically includes: Obtain medical image data and annotate the images; Acquire medical text data, annotate medical entities in the medical text data, and establish an association relationship between the medical entities and medical image data based on the annotations; Extract medical entities and their relationships from medical text data; Extended information is added to each medical entity according to the medical image data associated with the medical entity and the context in the medical text data.

3. The medical visual question answering method based on knowledge graph according to claim 2, characterized in that: The step of adding extended information to each medical entity according to the medical image data associated with the medical entity and the context in the medical text data specifically includes: Determine a current context corresponding to the medical text data according to the context of the medical entity in the medical text data; Obtaining an extended word of the medical entity in the current context, wherein the extended word includes at least one of a synonym, an antonym, and an associated medical entity; The extended term is added as extended information of the medical entity.

4. The medical visual question answering method based on knowledge graph according to claim 3 is characterized in that: The step of matching the image entity features with the medical entities in the pre-constructed knowledge graph to obtain a first matching result specifically includes: Calculate entity similarity between the image entity features and the embedding vectors of each medical entity in the knowledge graph; According to the calculation results, multiple medical entities whose entity similarity is greater than a preset first similarity threshold are selected; The selected multiple medical entities are sorted according to entity similarity as the first matching result.

5. The medical visual question answering method based on knowledge graph according to claim 4, characterized in that: The step of matching the text entity feature with the extended information of the medical entity in the pre-constructed knowledge graph to obtain a second matching result specifically includes: Respectively obtain extended information of each medical entity in the first matching result; Calculating information similarity between the text entity features and the extended information of each medical entity in turn; According to the information similarity calculation result, the multiple medical entities are sorted according to the information similarity as the second matching result.

6. The medical visual question answering method based on knowledge graph according to claim 5, characterized in that: The step of determining the medical entity corresponding to the question information based on the first matching result and the second matching result specifically includes: According to the sorting order of the second matching results, a similarity correction value for the first matching result is obtained according to a preset correction weight ratio; Based on the first matching result, the entity similarity of each medical entity is corrected by respectively adding the similarity correction value; The medical entity with the highest corrected entity similarity is used as the medical entity corresponding to the question information.

7. The medical visual question answering method based on knowledge graph according to claim 6, characterized in that: The step of determining the medical entity corresponding to the question information based on the first matching result and the second matching result specifically includes: If the corrected entity similarities are all less than a preset second similarity threshold, generating a guidance prompt according to the text entity feature; Acquire supplementary text information input by the user based on the guidance prompt; extracting supplementary text features from the supplementary text information; Matching the supplementary text feature with the extended information of each medical entity in the first matching result to obtain a third matching result; The entity similarity of each medical entity in the first matching result is supplemented and corrected by the third matching result.

8. The medical visual question answering method based on knowledge graph according to claim 6, characterized in that: The step of determining the medical entity corresponding to the question information based on the first matching result and the second matching result specifically includes: Get the user's case information; Extract case entity features from case information; Matching the case entity feature with the extended information of each medical entity in the first matching result to obtain a fourth matching result; The entity similarity of each medical entity in the first matching result is supplemented and corrected through the fourth matching result.

9. The medical visual question answering method based on knowledge graph according to claim 1, characterized in that: The step of extracting image entity features from the image information by using an image recognition model specifically includes: When there are multiple image entity features extracted from the image information by the image recognition model, generating a confirmation prompt according to the image entity features; Acquire confirmation information input by the user based on the confirmation prompt; The image entity features confirmed by the user are screened out from a plurality of image entity features according to the confirmation information.

10. A medical visual question answering system based on knowledge graph, characterized in that: include: An information acquisition unit, used to acquire question information input by a user, wherein the question information includes image information and text information; A feature extraction unit, used to extract image entity features from the image information through an image recognition model, and to extract text entity features and question intention from the text information through a natural language recognition model; A feature matching unit, used to match the image entity feature with the medical entity in the pre-constructed knowledge graph to obtain a first matching result; The feature matching unit is further used to match the text entity feature with the extended information of the medical entity in the pre-constructed knowledge graph to obtain a second matching result; An entity confirmation unit, configured to determine a medical entity corresponding to the question information based on the first matching result and the second matching result; The answer generation unit is used to obtain a medical answer corresponding to the question information according to the question intention and the determined medical entity.

Images