A medical visual question answering method based on type inference and semantic constraint

Abstract

The application discloses a medical visual question answering method based on type reasoning and semantic constraint. The medical visual question answering model can be roughly divided into four modules of picture feature extraction, question feature extraction, multi-modal feature fusion and loss function design, and the modules are interrelated and closely combined. On the basis of the existing medical visual question answering model, the application proposes a new medical visual question answering model based on type reasoning and answer semantic constraint, takes question type reasoning as the basis, constructs a joint semantic space as the core, designs a new loss function to constrain the model for training, and finally completes the prediction of the answer. The application designs an advanced medical visual question answering model based on advanced artificial intelligence technology, the characteristics of the medical background and the actual application requirements. The method has broad application prospects in medical intelligent inquiry, auxiliary clinical decision and image training.

Description

TECHNICAL FIELD

[0001] The application designs a medical visual question answering method based on type inference and semantic constraints, which relates to the cross field of image processing and natural language processing. BACKGROUND

[0002] With the rapid development of artificial intelligence (AI), clinical decision support has ushered in new opportunities. Currently, the main tasks of image diagnosis are image recognition, lesion detection and benign and malignant judgment. Medical visual question answering (Med-VQA) is relatively less studied due to the lack of detailed medical image annotation dataset. Med-VQA is a special domain problem, which aims to retrieve the correct answer by considering radiological images and clinical questions. The application aims to design a high-reliability medical visual question answering system, which can not only support clinical teaching and provide decision support for clinicians, but also help patients better understand their own health status according to medical image reports. From the perspective of the combination of clinical medicine and artificial intelligence, medical visual question answering has broad application prospects. However, due to the limitations of professional knowledge and data resources, the related technology of medical visual question answering is still in the initial stage of development. Therefore, we can draw lessons from the advanced solutions in the general visual question answering task.

[0003] In recent years, general VQA has attracted attention in the field of cross-media expression and interaction, attracting many experts and scholars in the fields of computer vision and natural language processing. The goal of general visual question answering is to answer questions related to real images. For example, given a picture, there are several people on the boat, and students and doctors can quickly answer "how many people are on the boat?" Generally, the general visual question answering task contains simple questions such as "what color are the leaves?" and some challenging questions such as "what can the purple object in the image be used for?". As can be seen, for different types of questions, the visual question answering model needs different reasoning modules to automatically reason and accurately answer.

[0004] Compared with the visual question answering task in the general field, medical visual question answering is more challenging. The Med-VQA model we designed is used to solve health and safety problems, so it requires higher accuracy and safety. However, due to privacy issues, a large amount of medical data is usually difficult to obtain. Secondly, medical visual question answering involves multi-modal images (MRI, X-Ray, CT, etc.) and diversified clinical questions, such as Figure 1Therefore, it is necessary to have professional medical image reading knowledge to make accurate inferences and give correct answers. In addition, training an efficient network model using deep learning requires a large amount of accurate labeled data. Medical visual question answering data also requires accurate annotation of medical professional knowledge, so obtaining medical data sets requires time, effort and money. According to the existing research work, there is only one manually annotated public medical visual question answering data set VQA-RAD. Compared with medical visual question answering data sets in general fields, medical visual question answering data sets are very limited. Therefore, we cannot directly use some advanced VQA models to process medical visual question answering problems.

[0005] Therefore, a large number of scholars have proposed advanced picture feature extraction modules, question feature extraction modules and multi-modal feature fusion modules for the visual question answering task in the medical field. These methods mainly improve different mode feature extraction methods and do not pay attention to the important modal data-rich information in the answer. In addition, they also ignore an important problem, that is, different types of questions focus on different information. In view of this problem, the present application proposes a medical visual question answering framework based on type reasoning and semantic space constraint. First, according to the question and answer pair type, a question classifier is trained in the training data set, and it is used to classify all questions. Second, the prediction result of the medical visual question answering model is mapped to the same semantic space as the reference answer. Then, we calculate the cosine distance between them to constrain the model for training. SUMMARY

[0006] The present application proposes a medical visual question answering method based on type reasoning and semantic constraint for the medical visual question answering task. The method mainly includes four modules: picture feature extraction, question feature extraction, multi-modal feature fusion and loss function design. The present application uses the corresponding ResNet-34 network in the multi-task pre-training model (MTPT) for picture feature extraction. In the question feature extraction module, the present application designs a type reasoning framework, which extracts features through a 1024-dimensional LSTM while classifying clinical questions into closed and open types. Then, the corresponding multi-modal features are fused through the multi-modal fusion feature module and input into the classifier for classification. In the loss function design stage, the present application constructs a joint semantic space based on the candidate answer set, and designs the loss function of the model through the semantic loss function and the cross-entropy loss function to guide the model for training.

[0007] The VQA-RAD dataset is a publicly available dataset with manual annotations by professional medical personnel. In the field of medical visual question answering, most advanced models use this dataset to validate their performance. This dataset contains 351 radiological images (head MRI, chest X-ray, and abdominal CT) and 3,515 related clinical questions. Each image has an average of 10 related questions. All questions are mainly divided into closed-ended and open-ended types. Our model achieved good experimental results on the VQA-RAD dataset, and the rationality of our method was verified through ablation experiments on the type reasoning and semantic constraint modules.

[0008] The present invention achieves the above objectives through the following technical solutions:

[0009] 1. In step (i), the input radiology images are classified by the multi-task pre-trained model MTPT, and then input into the corresponding ResNet-34 network for image feature extraction to obtain the image feature vector V. i For use by subsequent modules;

[0010] 2. In steps (ii) and (iii), the clinical question is word-embedded using the BioWordVec pre-trained model to obtain a 200-dimensional question word embedding Q. emb Then, the input is fed into a 1024-dimensional LSTM to obtain the problem feature vector V. q The calculation formula is as follows:

[0011] Q emb = BioWordVec(q) (1)

[0012] To improve the model's reasoning ability, such as Figure 2 As shown, we will get Inputting the data into a 1024-dimensional LSTM to obtain the feature vector of the problem. An attention mechanism is introduced to assign different weights to different words in the question. The specific calculation formula is as follows:

[0013]

[0014] In the formula, W1, W2, W a The weights are trained and then passed through a binary classification network to obtain the classification of the problem.

[0015] 3. In step (iv), this invention uses a cross-modal self-attention method to fuse multimodal features. To better understand the image regions related to the problem, an 8-dimensional spatial feature V is introduced. sThe picture features, the question features and the space features are first classified into closed type and open type through a binary classification network of question type, then the multi-modal feature fusion of the picture features, the question features and the space features is carried out respectively according to different types, and finally, answer classification is carried out.

[0016] 4、In steps (five) and (six), first, we calculate the loss between the predicted answer and the reference answer through the binary cross-entropy (BCE) loss, and the calculation formula is as follows:

[0017]

[0018] Then the candidate answer a i is input into the pre-trained model BioWordVec to obtain the word embedding B(i) of the answer, and then the Top-K predicted values of the medical visual question answering model and the reference answers in the data set are mapped to the same semantic space, and the calculation formula is as follows:

[0019]

[0020] F(g * )=P i B(i) (9)

[0021] In the formula, P i is the prediction probability of the model, and then the cosine similarity between the answers is calculated, and the calculation formula is as follows:

[0022]

[0023] Then the semantic loss function of the model can be represented as follows:

[0024] L s =1-S(F(P),F(g * )) (11)

[0025] Finally, the loss function is designed to constrain the model for training, and the loss calculation method is as follows:

[0026] Loss=L CE / BCE +λL S (12)

[0027] In order to ensure that the question type reasoning module and the semantic constraint module have the same effect on the model, the default value of λ in the formula is set to 0.5. DETAILED DESCRIPTION

[0028] Figure 1 is a medical visual question answering instance display.

[0029] Figure 2is a medical visual question answering model structure. DETAILED DESCRIPTION

[0030] The application will be further described below with reference to the drawings:

[0031] Figure 1 is a medical visual question answering model structure. The model can be roughly divided into four modules: picture feature extraction, question feature extraction, multi-modal feature fusion and loss function design. Each module is interrelated and cooperatively trained to ultimately realize the answer prediction task of medical visual question answering.

[0032] The purpose of the picture feature extraction module is to classify the picture shape through the multi-task pre-training model MTPT, and then input it into the corresponding ResNet-34 network for feature extraction, and finally obtain the picture feature vector V i for subsequent modules.

[0033] The role of the question feature extraction module is to map the clinical question to a medical semantic space to obtain a question vector as the basic data of the type reasoning module. As shown in Figure 2 , first, the question is embedded by the BioWordVec pre-training model to obtain a 200-dimensional question embedding Q emb , and then input into the 1024-dimensional LSTM to obtain the question feature vector V q . At the same time, in order to improve the reasoning ability of the model, the question feature vector and the answer type label are input into a binary classification network to train a question type classification network, which classifies the question into open-ended and closed-ended questions.

[0034] The main purpose of the multi-modal feature fusion module is to learn the mutual association information between different modalities. In order to better understand the picture area related to the question, an 8-dimensional spatial feature is introduced. Through the question type binary classification network, the picture feature, the question feature and the spatial feature are first divided into closed and open types, and then the multi-modal feature fusion of the picture feature, the question feature and the spatial feature is performed for different types. Finally, the answer classification is performed.

[0035] The loss function module mainly guides the model to learn the relationship between different modalities and the hidden rules in the question and answer process. As shown in Figure 2 , first, we calculate the loss between the predicted answer and the reference answer through the binary cross-entropy (BCE) loss. Then all the answers in the candidate answer set are mapped to the joint semantic space through the BioWordVec pre-training model to calculate the semantic similarity between the answers to define the semantic loss function. Finally, the cross-entropy loss and the semantic loss are combined into the loss function of the model by using the hyperparameter λ to constrain the model for training.

[0036] In order to embody the practical application ability of medical visual question answering, the present application uses the currently only public hand-labeled data set VQA-RAD to verify the performance of the model. At the same time, the existing method is also compared with the existing method, and the specific experimental results are shown in the following figure:

[0037] Table 1 The experimental comparison results of the network model of the present application and other existing models on the VQA-RAD data set

[0038]

[0039] As shown in the experimental results in Table 1, the medical visual question answering method based on type reasoning and semantic constraint proposed by the present application is superior to the existing best experimental results in overall performance index and closed question performance index on the public VQA-RAD data set.

[0040] In order to further verify the effectiveness of the type reasoning module and the semantic constraint module proposed by the present application, the present application carries out detailed ablation experiments, and cancels the type reasoning module and the semantic constraint module respectively under the condition that other conditions remain unchanged. In order to ensure the fairness of the comparative experiment, the present application trains the model in the same experimental environment, and the detailed results of the ablation experiment are shown in Table 2:

[0041] Table 1 Ablation experiment results of different modules of the network model of the present application

[0042]

[0043]

[0044] As can be seen from the experimental results in the table, both modules of the present application are beneficial to the improvement of the model performance, and compared with the type reasoning module, the effect of the semantic constraint module is better. The experimental results again verify the effectiveness of the method of the present application.

Claims

1. A medical visual question answering method based on type inference and semantic constraints, characterized in that Comprising the following steps: Step one: classify the picture shape through the multi-task pre-training model MTPT, and then input the picture into the corresponding ResNet-34 network to extract the picture feature, and obtain the picture feature vector V i ; Step two: obtain question word embedding Q using BioWordVec pre-training model emb Then input the question word embedding into 1024-dimensional LSTM to obtain question feature vector V q ; the question word embedding Q is obtained by using the BioWordVec pre-training model emb It includes encoding the question through the BioWordVec pre-training model, and the calculation method is as follows: Q emb = BioWordVec(q) (1) Obtained The question feature vector is obtained by inputting the question into a 1024-dimensional LSTM N is the length of the question. The length of the question is uniformly clipped to 12, and the length less than 12 is padded with the size of the dictionary. Then, an attention mechanism is introduced to assign different weights to different words in the question. The specific calculation method is as follows: a = softmax((W a G) Τ )(6) where W1, W2, W a are the training weight values; Step three: input the question feature vector V q and answer type label into a binary classification network, train a question type classification network to classify the question into open question and closed question; Step four: In order to better understand the picture area related to the question, an 8-dimensional spatial feature V is introduced s The picture feature, the question feature and the spatial feature are classified into two categories by a binary classification network of question type, then the multi-modal feature fusion of the picture feature, the question feature and the spatial feature is carried out for different types, and finally the answer classification is carried out. Step five: build an answer semantic space, map all answers in the candidate answer set A to the same semantic space through the BioWordVec pre-training model, calculate the semantic similarity between answers to design a semantic loss function; The constructing an answer semantic space, mapping all answers in the candidate answer set A to the same semantic space through the BioWordVec pre-training model includes firstly mapping the candidate answer a i Inputting into the pre-training model BioWordVec to obtain the word embedding B(i) of the answer, and then mapping the Top-K prediction values of the medical visual question answering model and the reference answers in the data set to the same semantic space, and the calculation formula is as shown in the following formula: F(g * ) = P i B(i)(8) where P i is the predicted probability of the model, and the semantic loss function is designed by calculating the semantic similarity between the answers, and the cosine similarity between the answers is calculated as follows: Then the semantic loss function of the model can be expressed as follows: L s = 1 - S(F(P), F(g * ))(10) Get the semantic loss function, which lays the foundation for designing the loss function of the model in the next step; Step six: combine the classification loss function and the semantic loss function into a combined loss function to constrain the model to complete training; The classification loss function includes calculating the loss between the predicted answer and the reference answer through the binary cross-entropy loss, and the calculation formula is as follows: Finally, design the loss function to constrain the model to train, and the loss calculation method is as follows: Loss = L BCE + λL S (12) In order to ensure that the question type reasoning module and the semantic constraint module have the same effect on the model, the default value of λ in the formula is set to 0.5.

Images