A medical visual question answering method based on corresponding feature fusion

Abstract

The application discloses a medical visual question answering method based on corresponding feature fusion. The model can be roughly divided into five modules, namely, category classification, visual feature extraction, semantic feature extraction, feature fusion and answer prediction. Considering the connection between specific medical images and corresponding problems, the application extracts specific feature representations for fusion after classifying the input data through a category classifier, and simultaneously combines a semantic attention module to predict answers, so that an end-to-end medical visual question answering process is realized. In recent years, medical resources are increasingly scarce, and the voice of medical empowerment is increasing. How to help patients efficiently and accurately understand their own conditions and assist doctors in diagnosing diseases has become crucial to improving the current medical efficiency and quality. In combination with the characteristics and demands of the medical background, the medical visual question answering method is researched, and has broad application prospects in intelligent inquiry and auxiliary clinical diagnosis.

Description

Technical Field

[0001] This invention designs a medical visual question answering method based on corresponding feature fusion, which involves the fields of deep learning technology, computer vision technology, and natural language processing technology. Background Technology

[0002] In recent years, visual question answering (VQA) has gained increasing attention. As a challenging multimodal task, VQA combines computer vision and natural language processing, taking an image and a question based on that image as input, and then processing the image through a series of steps to output the correct answer. Most VQA methods are based on supervised learning frameworks, requiring large-scale, well-labeled multimodal data for model training. In the medical field, Med-VQA aims to improve the quality and efficiency of current medical diagnosis, alleviating the pressure on healthcare resources under the current strain. On one hand, for patients, Med-VQA can help them identify potential abnormalities in images and understand their condition in a timely manner; on the other hand, for medical staff, it can assist doctors in clinical diagnosis and highlight potentially overlooked abnormalities in images. However, compared to general domains, medical VQA faces the problem of insufficient training data for models. On the one hand, there are few ways to obtain labeled image data, and the labeling of images is difficult, requiring the cooperation of professional and experienced doctors. On the other hand, the medical field has high requirements for the accuracy and professionalism of the labeled datasets, and different doctors have different questioning styles and wording habits, which brings considerable challenges to the production of medical visual question answering datasets.

[0003] In Med-VQA, meta-learning, transfer learning, and conditional reasoning are applied to address the data scarcity problem in current medical visual question answering methods. Pre-training the model with a large amount of external unlabeled medical data enables it to adapt to feature extraction on small datasets; alternatively, further refining the Med-VQA task classification improves the model structure and enhances its prediction accuracy in small sample tasks.

[0004] As AI technology continues to be widely adopted, the call for its application in healthcare is growing stronger. In today's context of an aging population and a global pandemic, medical resources are increasingly strained. Helping patients efficiently and accurately understand their conditions, assisting doctors in disease diagnosis, and improving the efficiency and quality of current medical care have become crucial. Summary of the Invention

[0005] In this paper, considering the relationship between specific types of images and questions in Med-VQA, we classify images and questions using a pre-trained classifier and then perform multimodal feature fusion on the corresponding image-question pairs. Furthermore, considering the specific semantic information contained in different types of questions, we propose a semantic attention module to help the model continuously learn these specific semantic features during training, deepening the model's understanding of question semantics.

[0006] We conducted experiments on the benchmark dataset VQA-RAD from previous years, as well as on a recently proposed bilingual dataset SLAKE, to verify the effectiveness of the method of classifying and fusing multimodal data and the semantic attention module. Compared with the five best methods in the past, our method outperformed the other methods in terms of accuracy, achieving a new best model.

[0007] The present invention achieves the above objectives through the following technical solutions:

[0008] 1. In step (one), the input problem and image are preprocessed, as shown below. For use by subsequent modules;

[0009] 2. Step (II) classifies the input question q and image I, and the final classifier outputs the category prediction score S. k Where k∈{abdomen, brain, chest}, represents the score for each of the three categories, and the classified medical images are obtained simultaneously. and the problem

[0010] 3. Step (III) Classify the images I k The images are fed into a pre-trained ResNet-34 database containing corresponding external image databases for brain MRI, chest X-ray, and abdominal CT scans, respectively, to extract specific information from different types of images and output visual feature representations for the corresponding brain, chest, and abdomen.

[0011] 4. Step (IV) Use 200-dimensional BioWordVec to classify the problems. Perform word embedding to obtain Next, it is fed into a 1024-dimensional Long Short-Term Memory (LSTM) network to extract text features and obtain preliminary text feature representations for various types of questions.

[0012] 5. Text feature representation of various types of questions in step (V) Q k Further processing is required:

[0013] F k =MaxPooling(Q k (1)

[0014] f k =AvgPooling(Q k (2)

[0015] f k =MLP(f k (3)

[0016] f k” =AvgPooling(f k ') (4)

[0017]

[0018] a k =Sigmoid(conv(F k ')) (6)

[0019] The global features of different types of problems are initially obtained by using a max pooling layer (1) and a mean pooling layer (2). Next, the global feature f obtained through mean pooling is... k The data is fed into a three-layer MLP for nonlinear transformation (3), where ReLU is used as the activation function, and then the global features of the problem are compressed into a mean pooling layer (4). Simultaneously referencing the residual connections, the global features F initially obtained through the max pooling layer are... k and f k” spliced ​​as in This represents tensor concatenation; ultimately, we use a 2×1×1 convolution kernel to concatenate F. k After fusion and compression, the semantic attention weights for the entire problem are obtained through a Sigmoid layer. The value of this weight determines which semantic features the model should focus on and which unimportant information should be ignored. Finally, we use a... k Multiply by the text feature representation Q of various problems k :

[0020]

[0021] Here, "⊙" represents dot product, and the final semantic feature representations for various problems are obtained. Where k∈{abdomen, brain, chest}, corresponding to the previously extracted visual features;

[0022] 6. In step (six), V k and the corresponding After fusing the data (k∈{abdomen, brain, chest}), the results are fed into a classifier for answer prediction.

[0023]

[0024] Next, cross-entropy is used to calculate the loss for the model's predicted answer:

[0025]

[0026] Where p k This represents the true answer label corresponding to different types of questions, where n represents the number of categories, i.e., the total number of candidate answers, and m represents the number of samples in each type of batch. Next, we sum the losses of the three categories {abdomen, brain, chest}:

[0027] L pred =L Abdomen +L Brain +L Chest (10)

[0028] Simultaneously, we also need to calculate the loss (11) of our designed type classifier initially classifying the image and the problem. We set the true class label of the input image as y, and simultaneously use the previously obtained class scores S k The image is fed into a Sigmoid layer, and the maximum value is taken as the image category predicted by the classifier.

[0029]

[0030] Where m' represents the size of the unclassified batch, and n' represents the number of categories of the image to be classified. Finally, we combine the losses calculated above to obtain the final loss of the model, which is used for backpropagation during model training.

[0031] L final =λL pred +(1-λ)L cls (12)

[0032] Where λ is a hyperparameter used to balance the loss. Attached Figure Description

[0033] Figure 1 It is a medical visual question-answering model structure.

[0034] Figure 2 It is a semantic attention module. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings:

[0036] Figure 1 This describes the structure of a medical visual question answering model. The model can be broadly divided into input image-question classification, visual feature extraction, semantic feature extraction, feature fusion, and answer prediction. These modules are organically linked and, combined with semantic attention, work together to complete the task of medical visual question answering.

[0037] The goal of the image-question classification module is to classify the input image I and question q, and obtain the image I of the corresponding category. k And question q k , where k∈{abdomen, brain, chest}, is used for specific types of feature extraction in later modules.

[0038] The visual feature extraction module will classify the images I k The images are fed into pre-trained ResNet-34 databases containing corresponding brain MRI, chest X-ray, and abdominal CT images to extract specific information from different types of images and output visual feature representations V for the brain, chest, and abdomen, respectively. k .

[0039] In the semantic feature extraction module, a 200-dimensional BioWordVec is first used to analyze the classified question q. k Perform word embedding to obtain Next, it is fed into a 1024-dimensional Long Short-Term Memory (LSTM) network to extract text features and obtain preliminary text feature representations Q for various types of questions. k It is then fed into the semantic attention (SA) module for processing. Figure 2 This is the model structure of the semantic attention module. First, a max pooling layer and a mean pooling layer are used to initially obtain the Q values ​​for different categories of questions. k global features F k ,f k Next, the global feature f obtained through mean pooling is... k The data is fed into a three-layer fully connected layer for nonlinear transformation, using ReLU as the activation function. Then, an average pooling layer is used to compress the global features of the problem into f. k” Simultaneously, referencing the residual connections, the global features F initially obtained through the max pooling layer are... k and f k spliced ​​into F k Finally, a 2×1×1 convolution kernel is used to apply F. k After fusion and compression, the semantic attention weights 'a' for the entire problem are obtained through a Sigmoid layer. k Finally, use a kMultiply by the text feature representation Q of various problems k To obtain the final semantic feature representation of various problems

[0040] The feature fusion and answer prediction module uses visual features V k and corresponding semantic features After fusing the data (k∈{abdomen, brain, chest}), the results are fed into a classifier for answer prediction. The cross-loss function is used to calculate the loss of the model's answer prediction. Next, the prediction losses for the three types of questions {abdomen, brain, chest} are summed. Then, the classification loss L of the type classifier is calculated. cls Finally, by combining the first two, the final loss L of the model was obtained. final Used to train models for answer prediction.

[0041] Table 1

[0042]

[0043] As shown in Table 1, we validated our proposed model against five other state-of-the-art methods from different time periods on the VQA-RAD and SLAKE datasets ("*" indicates the version we selected). Experimental results show that our model (CFF+SA), after feature fusion with CMSA, achieves competitive results on both datasets. On the VQA-RAD dataset, compared to the current state-of-the-art model, it achieves improvements of 1.1% and 3.0% in prediction accuracy for "Open" and "Closed" type questions, respectively. On the SLAKE dataset, it achieves improvements of 0.5% and 0.4% in prediction accuracy for "Overall" and "Open" type questions, respectively, with only a slight decrease in prediction accuracy for "Closed" type questions. This fully demonstrates the effectiveness of our proposed model in medical visual question answering tasks.

[0044] Table 2

[0045]

[0046]

[0047] Table 3

[0048]

[0049] To validate our proposed model, we conducted experiments on the VQA-RAD dataset. First, for fairness, we replaced CMSA with BAN as the multimodal feature fusion method and compared it with the current state-of-the-art method that also uses BAN for feature fusion. As shown in Table 2, our method outperforms all other methods in answer prediction accuracy on "Open," improves upon the current state-of-the-art method by 1.8% on "Closed," and is slightly lower than the current state-of-the-art model by 0.2% on "Overall." The experimental results objectively verify that our proposed method still achieves good performance when combined with BAN for feature fusion.

[0050] Secondly, as shown in Table 3, we ablated the model, and to more clearly present the analysis results, we also calculated the prediction accuracy for the three types of questions: {abdomen, brain, chest}. Simultaneously, we compared the model's applicability by combining BAN and CMSA. Firstly, the table shows that without using the proposed semantic attention (SA) module, our proposed method of feature fusion of corresponding type multimodal inputs (CFF), combined with BAN, significantly improved the original model's answer prediction accuracy. Furthermore, after combining CMSA, the model also improved the answer prediction accuracy for the "Overall" and "Closed" question types. With the addition of the semantic attention (SA) module, the model showed a certain degree of improvement in answer prediction performance for almost all question types compared to the previous methods, confirming the positive impact of introducing the semantic attention (SA) module. Moreover, the comparison between BAN and CMSA shows that when our model combines CMSA, it achieves higher accuracy in answer prediction for all types of questions. In summary, through experiments, we have demonstrated that in Med-VQA, the method of feature fusion (CFF) for corresponding types of image question pairs and our designed semantic attention (SA) module can help the model achieve better results in answer prediction.

Claims

1. A medical visual question answering method based on corresponding feature fusion, characterized in that... comprising the steps of: Step 1: Preprocess the input medical images and questions. Set the image size and number of channels to 3×224×224. Then, set the maximum length of each question to 12, and zero-padding is applied to questions shorter than 12 to ensure that the tensor dimensions are the same in subsequent model calculations. Set the batch size to B, and the input images in one batch are... The input question is Step 2: First, visual features of the input image are extracted using two convolutional pooling layers. Then, the extracted visual features are fed into a three-layer fully connected layer for nonlinear transformation. Finally, the classifier outputs a class prediction score S. k Where k∈{abdomen, brain, chest}, represents the score for each of the three categories, and the classified medical images are obtained simultaneously. and the problem Step 3: Classify the images I k The images are fed into a pre-trained ResNet-34 database containing corresponding external image databases for brain MRI, chest X-ray, and abdominal CT scans, respectively, to extract specific information from different types of images and output visual feature representations for the corresponding brain, chest, and abdomen. Step 4: Select to use 200-dimensional BioWordVec to classify the problem. Perform word embedding to obtain Next, it is fed into a 1024-dimensional Long Short-Term Memory (LSTM) network to extract text features and obtain preliminary text feature representations for various types of questions. Step 5: Use a max pooling layer and an average pooling layer to initially obtain the different categories of questions Q. k global features Next, the global feature f obtained through mean pooling is... k The data is fed into a three-layer fully connected layer for nonlinear transformation, using ReLU as the activation function, and then compressed into a mean pooling layer to achieve the desired global features. Simultaneously referencing the residual connections, the global features F initially obtained through the max pooling layer are... k and f k” spliced ​​as Then, a 2×1×1 convolution kernel is used on F. k' After fusion and compression, the semantic attention weights for the entire problem are obtained through a Sigmoid layer. Finally, use a k Multiply by the text feature representation Q of various problems k To obtain the final semantic feature representation of various problems Step Six: Visual Feature V k and corresponding semantic features After fusion, where k∈{abdomen, brain, chest}, the data is fed into the classifier for answer prediction. The cross-loss function is used to calculate the loss of the model's answer prediction. Next, the three class losses {abdomen, brain, chest} are summed, and then the classification loss L of the designed type classifier is calculated. cls Finally, by combining the losses calculated above, the final loss L of the model is obtained. final Used for training models. 2.The medical visual question answering method based on corresponding feature fusion according to claim 1, characterized in that In step two, a category classifier module is proposed. Visual features of the input image I are extracted using convolutional and pooling layers, then fed into a three-layer fully connected layer for nonlinear transformation, ultimately outputting prediction scores S for different categories. k The category with the highest score is used as the category of the input image and the question. 3.The medical visual question answering method based on corresponding feature fusion according to claim 1, characterized in that In step five, a semantic attention module is proposed to address the text feature representation Q. k Further processing is performed to obtain its semantic attention weight 'a'. k Then use a k Multiply by the text feature representation Q of various problems k To obtain the final semantic feature representation of various problems 4.The medical visual question answering method based on corresponding feature fusion according to claim 1, characterized in that In step six, the corresponding visual features V k and semantic features are fused and sent to the classifier for answer prediction. The sum of the answer prediction loss and the type classification loss is calculated for backpropagation during model training.

Images