Medical visual question and answer method based on global visual information intervention

Abstract

The invention provides a medical visual question and answer method based on global visual information intervention, and belongs to the technical field of computer vision and medical artificial intelligence. According to the method, the global visual information missing from the main branch is supplemented through the global visual information branch, and the contribution of the image to the answer prediction score is enhanced, so that the model pays more attention to the characteristics of the medical image when predicting the answer and does not excessively depend on the language characteristics in the corresponding question; a forward compensation branch is proposed to enhance the contribution of multi-modal fusion information in a main branch to an answer prediction score, so that the influence caused by interference additionally introduced by a global visual information branch is inhibited; besides, a nonlinear fusion variant is used for fusing prediction scores of a plurality of branches to serve as a final score, and a multi-branch loss fusion module is combined, so that punishment on a global visual information branch and a forward compensation branch is increased, and therefore, the subject status of a main branch during answer prediction is ensured, and the accuracy of answer prediction is improved. And the accuracy of the whole model is effectively improved.

Description

technical field [0001] The invention belongs to the technical field of computer vision and medical artificial intelligence, in particular to a medical visual question answering method based on global visual information intervention. Background technique [0002] Clinics receive, store and examine large volumes of medical images, such as radiology and pathology images, on a daily basis, putting increasing pressure on doctors to analyze the images. Medical Visual Question Answering (VQA) systems aim to jointly analyze multimodal content from medical images and natural language and provide correct answers for a given clinical question corresponding to medical images. Not only can it provide doctors with valuable additional insights to reduce misdiagnosis, it can also help patients understand their medical images without a doctor. [0003] Although this field has great potential in the medical industry, it is still in its infancy and there is relatively little research related ...

AI Technical Summary

Problems solved by technology

[0005] The above studies can guarantee the generalization ability of the model well in the case of a small sample size of the data set, but most of these studies still have the problem of overfitting, which affects the accuracy of the model on the test set. negative impact

In addition, their proposed models tend to rely heavily on linguistic bias to answer questions (linguistic bias, that is, the model overuses the superficial correlation between the question and answer words in the training set to generate answers), regardless of the image, which is harmful to the model. The robustness of the negative impact

Images