Medical visual question and answer system

Abstract

The invention discloses a medical visual question and answer system, and relates to the field of medical knowledge question and answer, and the system comprises a data generation module which is used for obtaining a plurality of medical electronic medical records, and constructing a medical visual question and answer sample set in a semi-automatic manner; the model training evaluation module is used for calling one or more deep learning models from the model storage module according to the model training request; based on the model training configuration instruction, training one or more deep learning models by adopting a medical visual question and answer sample set; evaluating the plurality of trained deep learning models according to a preset evaluation index; performing priority ranking on the plurality of trained deep learning models according to an evaluation result to obtain a model sequence; and the question and answer module is used for inputting the to-be-processed patient picture and the corresponding question text into the trained deep learning model to obtain a corresponding answer text. According to the method, medical questions and answers which are more convenient, more accurate and wider in universality are realized.

Description

Technical Field

[0001] The present application relates to the field of medical knowledge question answering, and in particular to a medical visualization question answering system. Background Art

[0002] Medical visual question answering (Med-VQA) is a challenging task in the medical industry. Given a medical image, Med-VQA can efficiently answer medical questions related to the image. Currently, the main challenges facing Med-VQA include the following two aspects: first, there is a lack of high-quality manually annotated datasets; second, there is no evaluation in a unified experimental setting, and it is impossible to know the performance of the obtained model. Summary of the invention

[0003] The purpose of this application is to provide a medical visual question-answering system to achieve more convenient, more accurate, and more versatile medical question-answering.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] This application provides a medical visualization question answering system, including:

[0006] A data generation module is used to: obtain multiple medical electronic medical records and construct a medical visual question answering sample set in a semi-automatic manner; the medical electronic medical records contain patient images;

[0007] Model storage module, used to store multiple deep learning models;

[0008] A model training evaluation module is used to: obtain a model training request and a model training configuration instruction; call one or more of the deep learning models from the model storage module according to the model training request; train one or more of the deep learning models using the medical visual question answering sample set based on the model training configuration instruction; obtain preset evaluation indicators, and evaluate the multiple deep learning models after training according to the preset evaluation indicators to obtain corresponding evaluation results; prioritize the multiple deep learning models after training according to the evaluation results to obtain a model sequence;

[0009] The model storage module is also used to: store the deep learning model after training or store the model sequence;

[0010] The question-answering module is used to: receive the patient image to be processed and the corresponding question text; call the deep learning model after training from the model storage module; input the patient image to be processed and the corresponding question text into the deep learning model after training to obtain the corresponding answer text.

[0011] According to the specific embodiments provided by the present application, the present application discloses the following technical effects: The present application provides a medical visualization question-and-answer system, which uses a semi-automatic method to construct a medical visual question-and-answer sample set in the data generation module. Compared with the method of all computer annotation in the prior art, it can increase high-quality artificial data information and provide a more accurate data basis for subsequent model training and evaluation. In the model training evaluation module, the deep learning model is called from the model storage module according to the model training request, and then the training is performed based on the model training configuration instruction. After the training is completed, the deep learning model after training is evaluated according to the preset evaluation index to obtain the corresponding evaluation result. So far, the present application has realized the evaluation of multiple deep learning models under a unified configuration in the model training evaluation module, and then prioritized according to the evaluation results, so as to obtain an intuitive performance advantage and disadvantage situation between different deep learning models, which is convenient for users to choose in subsequent applications. Finally, the question-and-answer module is used to input the patient image to be processed and the corresponding question text into the deep learning model after training called from the model storage module to obtain the corresponding answer text. Since the trained deep learning model has been obtained in the previous article, it can be directly called when the question-answering module is working, which greatly saves time, improves efficiency, and is more convenient. Furthermore, the application system includes both a model training and evaluation module and a directly applicable question-answering module, which can be provided to various types of users such as medical researchers, doctors or patients for application, with convenient operation and wider versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 This is a schematic diagram of the structure of a medical visualization question-answering system in one embodiment of the present application;

[0014] Figure 2 A schematic diagram of an algorithm framework of a medical visual question answering model in one embodiment of the present application;

[0015] Figure 3 This is a workflow diagram of a one-stop guiding robot in one embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] In order to make the purpose, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] In an exemplary embodiment, Figure 1 As shown, a medical visualization question-answering system is provided, including a data generation module, a model storage module, a model training and evaluation module, a question-answering module, a medical record storage module, a retrieval module and a guidance module.

[0019] The data generation module is used to obtain multiple medical electronic medical records and construct a medical visual question and answer sample set in a semi-automatic manner; wherein the medical electronic medical records contain patient images (such as CT images) and may also contain clinical records (such as patient symptoms, diagnosis information, treatment information) and other related information. Specifically, the data generation module includes an image extraction unit and a question and answer construction unit.

[0020] The image extraction unit is used to extract patient images from the medical electronic medical record and semi-automatically annotate the patient images to obtain corresponding image labels; the image labels include the organ type (such as chest), two-dimensional plane shape (such as coronary) and image type (such as CT image) in the image.

[0021] The question and answer construction unit is used to: 1) construct a question template based on a preset high-frequency medical question set using placeholders; the placeholders are used to represent medical entities and the relationships between medical entities; 2) extract text information from the medical electronic medical records and the image labels based on the question template, that is, extract text medical entities and their relationships from the text content such as the medical electronic medical records and image labels; 3) construct a medical visual question and answer sample based on the patient image and the corresponding text information; a plurality of medical visual question and answer samples corresponding to the medical electronic medical records constitute a medical visual question and answer sample set.

[0022] In a specific application example, the multiple medical electronic medical records obtained in the data generation module are one or more of the medical electronic medical record set collected by the user, the ROCO data set, the MIMIC-CXR data set, and the MedICaT data set. When it is a medical electronic medical record set collected by the user, the self-collected medical electronic medical record set needs to be uploaded to the data generation module through a preset interface, and the various sensitive information that may be contained in the medical electronic medical record can be de-privatized as needed, so that it only displays patient symptom information, diagnosis information, treatment information, CT or X-ray scan images. Then, the patient's picture is extracted as an auxiliary mark to obtain the specific content of various preset labels (for example, when the required label is the picture type or the organ type in the picture, the corresponding label content is extracted from the patient's picture), and the required VQA data set is generated. Through the above processing, the implicit information in the real clinical samples can be mined, and semi-automatic annotation can be assisted, successfully helping users to efficiently and systematically obtain a customized, high-quality medical corpus that meets the requirements of the visual question answering task.

[0023] The model storage module is used to store a variety of deep learning models. The model storage module can be regarded as an integrated library, which stores a variety of advanced neural network models, allowing users to easily configure hyperparameters and then apply them to question-answering tasks. The various deep learning models stored in the model storage module include: deep learning models based on attention mechanisms, deep learning models based on encoder-decoders, joint embedding models, and large language models.

[0024] Among them, the deep learning model based on the attention mechanism: reproduced MMBERT that uses a BERT-like model to process features, CMSA (cross modal self-attention) that effectively captures the long-distance dependency of visual and linguistic features, and the PTUnifier model that uses an attention-based medical vision-language pre-trained multimodal interaction model for module fusion.

[0025] The encoder-decoder based deep learning model: extract image features through the image encoder, encode question features through the question encoder, fuse multimodal features in the feature fusion layer; generate answers based on the fused features through the decoder. In an application example, an alignment module is added to the TCL model for cross-modal and intra-modal representation learning.

[0026] Joint embedding model: connect image embedding and text embedding as a hybrid embedding input to the classifier for classification. In an application example, MEVF uses model-agnostic meta-learning (MAML) and convolutional denoising autoencoder to initialize the model weights for image feature extraction; in addition, PubMed CLIP retains the same question encoding, multimodal fusion and objective function of MEVF, only replacing the visual encoder; CR proposes QCR (question-conditional reasoning) and TCR (type-conditional reasoning) modules to further obtain key information in the question.

[0027] Large language model: A deep learning model trained on a large amount of text data that can help interpret complex and detailed information in medical images. In one application example, the models used in this application include the medical visual instruction fine-tuning model MedVInT, which uses a large language model to generate language representations of questions, and the open source chatbot MiniGPT-4.

[0028] In a specific application example, the model storage module also includes a model visualization unit and a download export interface; wherein the model visualization unit allows users (such as medical researchers) to view the model description and the source code of the relevant model; the download export interface allows users to export the source code of the relevant model. That is, the model storage module can be regarded as a deep learning model library, which contains a variety of the latest advanced models of medical field question-answering models, general field question-answering models, and some multimodal large language models, which are convenient for users to apply to corresponding question-answering tasks.

[0029] The model training evaluation module is used to: obtain a model training request and a model training configuration instruction; call one or more of the deep learning models from the model storage module according to the model training request; based on the model training configuration instruction, use the medical visual question and answer sample set to train one or more of the deep learning models; obtain preset evaluation indicators, and evaluate the multiple deep learning models after training according to the preset evaluation indicators to obtain corresponding evaluation results; and prioritize the multiple deep learning models after training according to the evaluation results to obtain a model sequence.

[0030] In an actual application, the user can determine the model training request and model training configuration instructions through touch screen input, keyboard input, etc. Model training requests can be used to select a specific deep learning model; model training configuration instructions are used to configure the learning rate, number of iterations, type of RNN network, etc. during the training process to ensure the uniform configuration of each deep learning network in the same batch. The model training evaluation module is also used to: generate an evaluation report based on the evaluation results of multiple deep learning models after training, and display it to the user; mark the deep learning model with the best evaluation result after training, so as to display it to the user first.

[0031] In another practical application, the model training and evaluation module supports pre-training and fine-tuning of deep learning models. In terms of pre-training, users are allowed to select self-collected data sets for separate learning or multiple data sets for joint pre-training to effectively alleviate the problem of insufficient data volume; for example, the three major public data sets ROCO data set, MIMIC-CXR data set and MedICaT data set are combined for joint training. In terms of fine-tuning, the user first selects a benchmark data set (such as the VQA-RAD data set, which is a medical visual question-answering data set of 315 medical images collected from MedPix), a deep learning model (such as MEVF) and preset evaluation indicators (such as ACC) for simple configuration and evaluation. This application also supports users to adjust the hyperparameters of the model themselves, and the performance changes of the model under different parameters will be shown to the user. After training and evaluating multiple deep learning models, the evaluation report is used to display the comparison results of various models, and the best performing model can be intuitively seen based on the comparison results. At the same time, this application system supports downloading and dumping the evaluation result file. On this basis, the model training evaluation module also includes an evaluation display unit, which is used to display the evaluation results corresponding to the multiple deep learning models after the training is completed in the form of a bar graph.

[0032] In another practical application, during the training of the deep learning model, the model training evaluation module includes an image preprocessing unit, a picture feature extraction unit, a text feature extraction unit, a feature fusion unit, and a prediction evaluation unit. The medical visual question answering model algorithm framework composed of the above multiple units can be applied to four deep learning networks: a deep learning model based on an attention mechanism, a deep learning model based on an encoder-decoder, a joint embedding model, and a large language model. The algorithm framework can be used as follows: Figure 2 shown.

[0033] The image preprocessing unit is used to: convert the patient image into an image format; specifically, convert the patient image in DICOM format into a standard image format, such as JPG.

[0034] The image feature extraction unit is used to: perform image encoding on the patient image after image format conversion to obtain an image feature vector; specifically, a convolutional neural network or a pre-trained visual model can be used to extract the image feature vector.

[0035] The text feature extraction unit is used to: perform word segmentation, part-of-speech tagging, syntactic analysis, and encoding processing on the question text in the text information in sequence to obtain a text feature vector; wherein word segmentation, part-of-speech tagging, syntactic analysis, and other processing can facilitate the computer to understand the meaning of the question; then, a recurrent neural network or a pre-trained language model is used to extract the text feature vector.

[0036] The feature fusion unit is used to fuse the image feature vector with the text feature vector to obtain a multimodal feature vector, thereby facilitating the computer to jointly understand the medical image and the question.

[0037] A prediction evaluation unit is used to: determine a predicted answer text based on the multimodal feature vector using a classification model or a regression model, and perform prediction performance evaluation in combination with the predicted answer text and the answer text in the text information.

[0038] It should be noted that the above-mentioned medical visual question answering model algorithm framework has not completed the training process, but is only a prediction process. If the prediction result meets the preset requirements, the corresponding deep learning model can be used as the deep learning model after training; otherwise, the next medical visual question answering sample needs to be processed until it meets the preset requirements.

[0039] The model storage module is also used to: store the deep learning model after training or store the model sequence.

[0040] The question-answering module is used to: receive the patient image to be processed and the corresponding question text; call the trained deep learning model from the model storage module; input the patient image to be processed and the corresponding question text into the trained deep learning model to obtain the corresponding answer text. The question-answering module is also used to: obtain the model sequence; input the patient image to be processed and the corresponding question text into any trained deep learning model in the model sequence to obtain multiple answer texts; sort the multiple answer texts according to the sorting priority in the model sequence and display them visually.

[0041] The medical record storage module is connected to the data generation module, and the medical record storage module is used to store all the medical electronic medical records.

[0042] The retrieval module is connected to the medical record storage module, and the retrieval module is used to: receive similar medical record retrieval requests, and obtain the medical electronic medical records to be retrieved corresponding to the similar medical record retrieval requests; based on the BFMatcher algorithm and the TF-IDF algorithm, filter the medical electronic medical records with the largest similarity value from the medical record storage module according to the medical electronic medical records to be retrieved, and perform visual display.

[0043] The guidance module is used to: interact with the user in the form of online chat, guide the user to input multiple medical electronic medical records into the data generation module, guide the user to input the model training request, the model training configuration instructions and the preset evaluation indicators into the model training evaluation module, guide the user to input the patient image to be processed and the corresponding question text into the question and answer module, and guide the user to input the similar medical record retrieval request and the corresponding medical electronic medical record to be retrieved into the retrieval module.

[0044] In an exemplary application, the system of the present application provides a one-stop guidance robot, which includes the functions of model training and evaluation module, question and answer module, retrieval module and guidance module, to assist non-professional users to conduct data set evaluation more conveniently. Figure 3 As shown, it is a workflow diagram of the one-stop guidance robot, and the specific process is as follows: professional users (researchers in the medical field) or non-professional users (doctors or patients) set it up according to their own needs; when a search is required, input a similar medical record search request and the corresponding medical electronic medical record to be searched, and the one-stop guidance robot will display the medical electronic medical record with the greatest similarity value; when a question and answer is required, input the patient image to be processed and the corresponding question text, and the one-stop guidance robot will display the corresponding answer text; when a model performance comparison is required, input multiple medical electronic medical records, model training requests, model training configuration instructions and preset evaluation indicators, and the one-stop guidance robot will display the corresponding evaluation report or model sequence.

[0045] The system of this application can be used by professional users or non-professional users, providing a unified evaluation framework to facilitate the deployment and evaluation of various multimodal medical models in different environments, helping non-professional users to avoid the trouble of setting up the environment and selecting parameters. Only simple configuration is required to implement various complex models, and various models are comprehensively experimented on the benchmark data set, while supporting users to select functions such as medical auxiliary diagnosis. Specifically, the data set collected by the user is input into the data generation component, and an annotation tool for marking the data is provided to generate medical visual question and answer samples; then it is input into the trained deep learning model and evaluated, so that a high-quality medical visual question and answer training data set that has been inspected can be presented. At the same time, users are allowed to submit any medical images and questions to chat with the one-stop guide robot of this application, and the test results of the selected model are given; for example, the user uploads a chest CT scan image and a medical question text, and the robot returns the test results of the most advanced models such as METER, PTUnifier, MMBert, MiniGPT-4, etc.; for the electronic medical record report retrieval function display, by searching the database and calculating the similarity, the robot displays the patient's electronic medical record report that best matches the medical image, as well as the best recommended medical insurance list that the user can choose. Users can have a basic understanding of their condition based on the corresponding images, and choose recommended medical insurance services according to their needs.

[0046] In summary, for the system of this application, users only need to select the benchmark data set, deep learning model and preset evaluation indicators before model practice to complete a simple configuration. Then the system can conduct extensive experiments on the selected model and report a table of the model's accuracy on closed, open and overall problems. For a more in-depth comparison, this application also provides a bar chart to compare the models, so that the comparison between models is more intuitive and convincing.

[0047] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0048] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A medical visualization question-answering system, characterized in that: The medical visualization question answering system comprises: A data generation module is used to: obtain multiple medical electronic medical records and construct a medical visual question answering sample set in a semi-automatic manner; the medical electronic medical records contain patient images; Model storage module, used to store multiple deep learning models; A model training evaluation module is used to: obtain a model training request and a model training configuration instruction; call one or more of the deep learning models from the model storage module according to the model training request; train one or more of the deep learning models using the medical visual question answering sample set based on the model training configuration instruction; obtain preset evaluation indicators, and evaluate the multiple deep learning models after training according to the preset evaluation indicators to obtain corresponding evaluation results; prioritize the multiple deep learning models after training according to the evaluation results to obtain a model sequence; The model storage module is also used to: store the deep learning model after training or store the model sequence; The question-answering module is used to: receive the patient image to be processed and the corresponding question text; call the deep learning model after training from the model storage module; input the patient image to be processed and the corresponding question text into the deep learning model after training to obtain the corresponding answer text.

2. The medical visualization question answering system according to claim 1, characterized in that: The medical visualization question-answering system also includes a medical record storage module and a retrieval module; The medical record storage module is connected to the data generation module, and the medical record storage module is used to: store all the medical electronic medical records; The retrieval module is connected to the medical record storage module, and the retrieval module is used to: receive similar medical record retrieval requests, and obtain the medical electronic medical records to be retrieved corresponding to the similar medical record retrieval requests; based on the BFMatcher algorithm and the TF-IDF algorithm, filter the medical electronic medical records with the largest similarity value from the medical record storage module according to the medical electronic medical records to be retrieved, and perform visual display.

3. The medical visualization question answering system according to claim 2, characterized in that: The medical visualization question-answering system further includes a guidance module; The guidance module is used to: interact with the user in the form of online chat, guide the user to input multiple medical electronic medical records into the data generation module, guide the user to input the model training request, the model training configuration instructions and the preset evaluation indicators into the model training evaluation module, guide the user to input the patient image to be processed and the corresponding question text into the question and answer module, and guide the user to input the similar medical record retrieval request and the corresponding medical electronic medical record to be retrieved into the retrieval module.

4. The medical visualization question answering system according to claim 1, characterized in that: The question-answering module is also used to: obtain the model sequence; input the patient image to be processed and the corresponding question text into any deep learning model after training in the model sequence, so as to obtain multiple answer texts; According to the sorting priority in the model sequence, the multiple answer texts are sorted and visually displayed.

5. The medical visualization question answering system according to claim 1, characterized in that: The data generation module includes a picture extraction unit and a question and answer construction unit; The image extraction unit is used to extract patient images from the medical electronic medical record and semi-automatically annotate the patient images to obtain corresponding image labels; the image labels include organ types, two-dimensional plane shapes and image types in the images; The question-answer construction unit is used to: construct a question template using placeholders based on a preset set of high-frequency medical questions; Extracting text information from the medical electronic medical record and the image label based on the question template; constructing a medical visual question answering sample based on the patient image and the corresponding text information; The medical visual question and answer samples corresponding to the multiple medical electronic medical records constitute a medical visual question and answer sample set; the placeholders are used to represent medical entities and the relationships between medical entities.

6. The medical visualization question answering system according to claim 1, characterized in that: The various deep learning models stored in the model storage module include: a deep learning model based on an attention mechanism, a deep learning model based on an encoder-decoder, a joint embedding model, and a large language model.

7. The medical visualization question answering system according to claim 5, characterized in that: In the process of training the deep learning model, the model training evaluation module includes: An image preprocessing unit, used to: convert the patient image into an image format; A picture feature extraction unit, used to: perform picture encoding on the patient picture after picture format conversion to obtain a picture feature vector; A text feature extraction unit is used to perform word segmentation processing, part-of-speech tagging processing, syntactic analysis processing, and encoding processing on the question text in the text information in sequence to obtain a text feature vector; A feature fusion unit, used to: fuse the image feature vector with the text feature vector to obtain a multimodal feature vector; A prediction evaluation unit is used to: determine a predicted answer text based on the multimodal feature vector using a classification model or a regression model, and perform prediction performance evaluation in combination with the predicted answer text and the answer text in the text information.

8. The medical visualization question answering system according to claim 1, characterized in that: The model training evaluation module also includes an evaluation display unit, which is used to display the evaluation results corresponding to the multiple deep learning models after training in the form of a bar graph.

9. The medical visualization question answering system according to claim 1, characterized in that: The multiple medical electronic medical records obtained in the data generation module are one or more of a medical electronic medical record set collected by a user, a ROCO data set, a MIMIC-CXR data set, and a MedICaT data set.

Images