Multi-modal question and answer model training method and device for medical image, medium and equipment

Abstract

The invention discloses a multi-modal question and answer model training method and device for a medical image, a medium and equipment, and belongs to the technical field of deep learning. Creating a multi-modal question and answer model; obtaining a first training set and a second training set; fixing the weights of the CLIP model, the SAM and the LLM, and training the weights of the first visual language connector and the second visual language connector by using the first training sample; fixing the weight of the CLIP model and the weight of the SAM, and training the weight of the first visual language connector, the weight of the second visual language connector and the weight of the LLM by using the first training sample; fixing the weights of the CLIP model and the SAM, and training the weights of the first visual language connector, the second visual language connector and the LLM by using a second training sample; and loading the trained weight into the question and answer model to obtain a trained multi-modal question and answer model. The multi-mode question and answer model can process fine and complex medical images, and the accuracy and fineness of answers are improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method, device, medium and equipment for training a multimodal question-answering model for medical images. Background Art

[0002] The input of the question-answering model is a medical image and a question about the medical image, and the output is the answer to the question based on the medical image. The question-answering model can help patients get feedback on their condition and help doctors generate diagnostic opinions.

[0003] The question-answering model in the related technology is a multimodal model, which includes a Contrastive Language-Image Pre-Training (CLIP) model, a visual-language connector and a large language model (LLM). The CLIP model is responsible for converting medical images into feature vectors, the visual-language connector is responsible for performing visual-language space conversion on the feature vectors, and the LLM is responsible for generating answers based on the converted feature vectors and the text features of the question.

[0004] Although the question-answering model can achieve a multimodal combination of visual and language understanding, its application in the medical field has the shortcomings of insufficient visual perception and lack of medical knowledge, resulting in low accuracy of the answers generated by the question-answering model. Summary of the invention

[0005] The present application provides a method, device, medium and equipment for training a multimodal question-answering model for medical images, which is used to solve the problems of insufficient visual perception and lack of medical knowledge in the application of question-answering models in the medical field. The technical solution is as follows:

[0006] According to a first aspect of the present application, a method for training a multimodal question-answering model for medical images is provided, the method comprising:

[0007] Creating a multimodal question-answering model, the multimodal question-answering model comprising a contrastive language-image pre-trained CLIP model, a segmentation model SAM, a first visual language connector, a second visual language connector, and a large language model LLM, the CLIP model being connected to the LLM via the first visual language connector, and the SAM being connected to the LLM via the second visual language connector;

[0008] Obtaining a first training set and a second training set, wherein the first training samples in the first training set include medical images, image annotations, simple question-and-answer texts with image annotations, and simple question-and-answer texts without image annotations, and the second training samples in the second training set include medical images, image annotations, detailed question-and-answer texts with image annotations, and detailed question-and-answer texts without image annotations;

[0009] Fixing the weights of the CLIP model, the SAM, and the LLM, and using the first training sample to train the weights of the first visual-linguistic connector and the second visual-linguistic connector;

[0010] Fixing the weights of the CLIP model and the SAM, and using the first training sample to train the weights of the first visual-linguistic connector, the second language-visual connector, and the LLM;

[0011] Fixing the weights of the CLIP model and the SAM, and using the second training sample to train the weights of the first visual-linguistic connector, the second language-visual connector, and the LLM;

[0012] The trained weights are loaded into the multimodal question-answering model to obtain a trained multimodal question-answering model.

[0013] In a possible implementation, the method further includes:

[0014] Processing the test sample using the multimodal question-answering model to obtain an answer output by the multimodal question-answering model;

[0015] Obtaining user feedback information on the answer;

[0016] The multimodal question-answering model is optimized using a direct preference optimization (DPO) algorithm and the feedback information.

[0017] In a possible implementation, fixing the weights of the CLIP model, the SAM, and the LLM, and using the first training sample to train the weights of the first visual language connector and the second language visual connector includes:

[0018] Using a CLIP model with fixed weights to extract features from the medical images in the first training sample to obtain first image features;

[0019] Encoding the first image feature using the first visual language connector to obtain a first image code;

[0020] When the simple question-answer text without image annotations in the first training sample is used for training, the medical image is feature extracted using a SAM with a fixed weight to obtain a second image feature; the second image feature is encoded using the second visual language connector to obtain a second image encoding; the simple question-answer text without image annotations is randomly masked and encoded using a word segmentation and an embedding layer to obtain a first text encoding;

[0021] When the image annotations and the simple question-answer text with image annotations in the first training sample are used for training, the medical image and the image annotation are extracted using a fixed-weight SAM to obtain a second image feature; the second image feature is encoded using the second visual language connector to obtain a second image encoding; the simple question-answer text with image annotations is encoded after random masking using a word segmentation and an embedding layer to obtain a first text encoding;

[0022] Using a fixed-weight LLM to process the first image code, the second image code, and the first text code to obtain a predicted text of the simple question-and-answer text;

[0023] The weights of the first visual-language connector and the second visual-language connector are updated according to the simple question-answer text and the predicted text.

[0024] In a possible implementation, fixing the weights of the CLIP model and the SAM, and using the first training sample to train the weights of the first visual language connector, the second language visual connector, and the LLM, includes:

[0025] Using a CLIP model with fixed weights to extract features from the medical images in the first training sample, to obtain third image features;

[0026] encoding the third image feature by using the first visual language connector to obtain a third image code;

[0027] When the simple question-answer text without image annotations in the first training sample is used for training, the medical image is feature extracted by using the SAM with fixed weights to obtain a fourth image feature; the fourth image feature is encoded by using the second visual language connector to obtain a fourth image encoding; the simple question-answer text without image annotations is randomly masked and encoded by using the word segmentation and embedding layer to obtain a second text encoding;

[0028] When the image annotations and the simple question-answer text with image annotations in the first training sample are used for training, the medical image and the image annotation are feature extracted using a fixed-weight SAM to obtain a fourth image feature; the fourth image feature is encoded using the second visual language connector to obtain a fourth image encoding; the simple question-answer text with image annotations is randomly masked and encoded using a word segmentation and an embedding layer to obtain a second text encoding;

[0029] Using the LLM to process the third image code, the fourth image code and the second text code to obtain a predicted text of the simple question and answer text;

[0030] The weights of the first visual-language connector, the second language-visual connector, and the LLM are updated according to the simple question-answer text and the predicted text.

[0031] In a possible implementation, fixing the weights of the CLIP model and the SAM, and using the second training sample to train the weights of the first visual language connector, the second language visual connector, and the LLM, includes:

[0032] Using a CLIP model with fixed weights to extract features from the medical images in the second training sample, to obtain fifth image features;

[0033] encoding the fifth image feature by using the first visual language connector to obtain a fifth image code;

[0034] When the detailed question-and-answer text without image annotations in the second training sample is used for training, the medical image is feature extracted using a SAM with a fixed weight to obtain a sixth image feature; the sixth image feature is encoded using the second visual language connector to obtain a sixth image encoding; the answer in the detailed question-and-answer text without image annotations is masked and encoded using a word segmentation and an embedding layer to obtain a third text encoding;

[0035] When the image annotations and the detailed question-and-answer text with the image annotations in the second training sample are used for training, the medical image and the image annotation are feature extracted using a SAM with fixed weights to obtain a sixth image feature; the sixth image feature is encoded using the second visual language connector to obtain a sixth image encoding; the answer in the detailed question-and-answer text with the image annotations is masked and encoded using a word segmentation and an embedding layer to obtain a third text encoding;

[0036] Processing the fifth image code, the sixth image code and the third text code by using the LLM to obtain a predicted text of the answer;

[0037] The weights of the first visual-linguistic connector, the second linguistic-visual connector, and the LLM are updated according to the answer and the predicted text.

[0038] In a possible implementation, creating a multimodal question-answering model includes:

[0039] Acquire a multimodal model, and pre-train the multimodal model using a third training set, wherein the multimodal model includes a CLIP model, a first visual language connector, and an LLM;

[0040] Obtaining a SAM, and fine-tuning the SAM using a fourth training set;

[0041] The multimodal question answering model is created based on the pre-trained multimodal model and the fine-tuned SAM model.

[0042] In a possible implementation, the method further includes:

[0043] Using the CLIP model to perform feature extraction on the medical image to be processed, to obtain a seventh image feature;

[0044] encoding the seventh image feature by using the first visual language connector to obtain a seventh image code;

[0045] When the medical image has no image annotation, extracting features from the medical image using the SAM to obtain an eighth image feature; encoding the eighth image feature using the second visual language connector to obtain an eighth image code; encoding the problem to be processed using a word segmentation and an embedding layer to obtain a fourth text code;

[0046] When the medical image has an image annotation, the SAM is used to extract features from the medical image and the image annotation to obtain an eighth image feature; the second visual language connector is used to encode the eighth image feature to obtain an eighth image code; and the word segmentation and embedding layer are used to encode the problem to be processed to obtain a fourth text code;

[0047] The seventh image code, the eighth image code and the fourth text code are processed by using the LLM to obtain an answer output by the multimodal question-answering model.

[0048] According to a second aspect of the present application, a multimodal question-answering model training device for medical images is provided, the device comprising:

[0049] A creation module, used to create a multimodal question-answering model, wherein the multimodal question-answering model includes a contrastive language-image pre-trained CLIP model, a segmentation model SAM, a first visual language connector, a second visual language connector, and a large language model LLM, wherein the CLIP model is connected to the LLM via the first visual language connector, and the SAM is connected to the LLM via the second visual language connector;

[0050] an acquisition module, configured to acquire a first training set and a second training set, wherein the first training samples in the first training set include medical images, image annotations, simple question-and-answer texts with image annotations, and simple question-and-answer texts without image annotations, and the second training samples in the second training set include medical images, image annotations, detailed question-and-answer texts with image annotations, and detailed question-and-answer texts without image annotations;

[0051] A training module, used for fixing the weights of the CLIP model, the SAM and the LLM, and training the weights of the first visual-linguistic connector and the second visual-linguistic connector using the first training sample;

[0052] The training module is further used to fix the weights of the CLIP model and the SAM, and train the weights of the first visual language connector, the second language visual connector, and the LLM using the first training sample;

[0053] The training module is further used to fix the weights of the CLIP model and the SAM, and train the weights of the first visual language connector, the second language visual connector, and the LLM using the second training sample;

[0054] A loading module is used to load the trained weights into the multimodal question-answering model to obtain a trained multimodal question-answering model.

[0055] According to a third aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the multimodal question-answering model training method for medical images as described above.

[0056] According to a fourth aspect of the present application, a computer device is provided, which includes the above-mentioned multimodal question-answering model training device for medical images.

[0057] The beneficial effects of the technical solution provided by this application include at least:

[0058] By training the multimodal question answering model with a large number of first training sets and second training sets, medical knowledge can be injected into the multimodal question answering model, thereby improving the professional ability of the multimodal question answering model to understand and process medical images and texts.

[0059] By processing medical images through pixel-level SAM, it is possible to provide fine-grained and dense visual perception capabilities for the multimodal question-answering model, improving its ability to process delicate and complex medical images.

[0060] By adding image annotations of regions of interest in medical images, the focus of user questions can be narrowed from the entire image to the region of interest, thereby improving the accuracy and precision of the answers.

[0061] Through the DPO algorithm, we can collect users' feedback on the answers output by the multimodal question-answering model, and perform value alignment, preference alignment, and expertise alignment based on the feedback information to improve the security, accuracy, and professionalism of the answers output by the multimodal question-answering model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 is a structural diagram of a multimodal question-answering model provided in an embodiment of the present application;

[0064] Figure 2 is a flowchart of a multimodal question-answering model training method for medical images provided by an embodiment of the present application;

[0065] Figure 3 is a flowchart of a multimodal question-answering model training method for medical images provided by an embodiment of the present application;

[0066] Figure 4 is a flowchart of a method for using a multimodal question-answering model for medical images provided by an embodiment of the present application;

[0067] Figure 5 This is a structural block diagram of a multimodal question-answering model training device for medical images provided by an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0069] like Figure 1As shown, the multimodal question-answering model includes a CLIP model, a SAM, a first visual language connector (VL-Connector1), a second visual language connector (VL-Connector2) and an LLM. The CLIP model is connected to the LLM through the first visual language connector (VL-Connector1), and the SAM is connected to the LLM through the second visual language connector (VL-Connector2). The multimodal question-answering model also includes a tokenizer and embedding layer, and the tokenizer and embedding layer are connected to the LLM. Among them, the CLIP model and SAM are used to process medical images, and output the image encoding to the LLM through their respective connectors. The tokenizer and embedding layer are used to encode questions and output the text encoding to the LLM. The LLM processes the image encoding and the text encoding and outputs the answer.

[0070] like Figure 2 As shown, it shows a method flow chart of a multimodal question-answering model training method for medical images provided by an embodiment of the present application, and the multimodal question-answering model training method for medical images can be applied to a computer device. The multimodal question-answering model training method for medical images may include:

[0071] Step 201, creating a multimodal question-answering model.

[0072] Among them, the structure of the multimodal question-answering model is as follows Figure 1 shown.

[0073] The multimodal question-answering model is formed by adding a visual-language connector and a SAM to a multimodal model. We can pre-train the multimodal model and fine-tune the SAM so that the multimodal model and SAM have their own initial weights.

[0074] Step 202, obtaining a first training set and a second training set, wherein the first training samples in the first training set include medical images, image annotations, simple question and answer texts with image annotations, and simple question and answer texts without image annotations, and the second training samples in the second training set include medical images, image annotations, detailed question and answer texts with image annotations, and detailed question and answer texts without image annotations.

[0075] Medical images are images output after examining body parts using medical equipment, including but not limited to X-rays, B-ultrasound images, computed tomography (CT) scan images, and nuclear magnetic resonance imaging (MRI) images.

[0076] Image annotations are used to indicate the region of interest selected by the user in a medical image. Image annotations can be points, boxes, contours, masks, etc.

[0077] According to the complexity and sophistication of the question-answer text, we divide the question-answer text into two categories: simple question-answer text and detailed question-answer text. The classification rules can be set according to the content, word count, etc., which are not limited here. Among them, simple question-answer text is used in the pre-training stage to enable the multimodal question-answering model to simply describe the general content and rough content of the medical image; detailed question-answer text is used in the fine-tuning stage to enable the multimodal question-answering model to describe the content of the medical image in more detail, or to generate answers with reasoning nature.

[0078] In this embodiment, the first training set and the second training set are used to supplement professional medical knowledge for the multimodal question-answering model.

[0079] Step 203, fixing the weights of the CLIP model, SAM and LLM, and using the first training sample to train the weights of the first visual language connector and the second language visual connector.

[0080] When training just starts, medical images and simple question-answering texts without image annotations in the first training sample can be used for training; after the first visual-language connector and the second language visual connector gradually stabilize, medical images, image annotations, and simple question-answering texts with image annotations in the first training sample can be used for training with a certain probability, so that the multimodal question-answering model supports both question-answering without image annotations and question-answering with image annotations.

[0081] Step 204 , fix the weights of the CLIP model and the SAM, and train the weights of the first visual-linguistic connector, the second language visual connector, and the LLM using the first training sample.

[0082] Similarly, when training begins, the medical images and simple question-and-answer texts without image annotations in the first training sample can be used for training; subsequently, the medical images, image annotations, and simple question-and-answer texts with image annotations in the first training sample can be used for training with a certain probability.

[0083] Step 205 , fix the weights of the CLIP model and the SAM, and use the second training sample to train the weights of the first visual-linguistic connector, the second language-visual connector, and the LLM.

[0084] Similarly, at the beginning of training, the medical images and detailed question-and-answer texts without image annotations in the second training sample can be used for training; subsequently, the medical images, image annotations, and detailed question-and-answer texts with image annotations in the second training sample can be used for training with a certain probability.

[0085] Step 206, loading the trained weights into the multimodal question-answering model to obtain a trained multimodal question-answering model.

[0086] To summarize, the multimodal question-answering model training method for medical images provided in the embodiments of the present application trains the multimodal question-answering model by using a large number of first training sets and second training sets, thereby injecting medical knowledge into the multimodal question-answering model and improving the professional ability of the multimodal question-answering model to understand and process medical images and texts.

[0087] Processing medical images through pixel-level SAM can provide fine-grained and dense visual perception capabilities for the multimodal question-answering model, improving its ability to process delicate and complex medical images.

[0088] By adding image annotations of regions of interest in medical images, the focus of user questions can be narrowed from the entire image to the region of interest, thereby improving the accuracy and precision of the answers.

[0089] like Figure 3 As shown, it shows a flowchart of a multimodal question-answering model training method for medical images provided by an embodiment of the present application, and the multimodal question-answering model training method for medical images can be applied to a computer device. The multimodal question-answering model training method for medical images may include:

[0090] Step 301, creating a multimodal question-answering model.

[0091] Specifically, creating a multimodal question-answering model may include:

[0092] (1) Obtain a multimodal model and use the third training set to pre-train the multimodal model, where the multimodal model includes a CLIP model, a first visual language connector, and an LLM.

[0093] The multimodal models include the CLIP model, the first visual-linguistic connector, and the LLM.

[0094] The training samples in the third training set include medical images and question-answering text of at least one round of question-answering conducted on the medical images.

[0095] After pre-training the multimodal model based on the image-text pairs in the third training set, the multimodal model can learn the correlation between images and texts, and can effectively extract and fuse the information of these two modalities, so that the multimodal model has strong cross-modal understanding capabilities. It can perform well in a variety of tasks, including but not limited to image description generation, visual question answering, multimodal dialogue, etc.

[0096] In order to make the multimodal model better serve the medical industry, it is necessary to further fine-tune it using specialized medical data. This fine-tuning can not only enhance the multimodal model's ability to understand medical images, but also help the multimodal model master more professional medical knowledge. Specifically, the fine-tuned multimodal model can: improve the ability to understand professional vocabulary, better understand and generate medical terms; improve disease recognition accuracy, more accurately identify and classify diseases; enhance medical decision support, and provide more accurate diagnostic advice or treatment plans. Multimodal models can retain their general capabilities while acquiring expertise specific to the medical industry, thereby playing a greater role in practical applications. For example, they can demonstrate higher efficiency in auxiliary diagnosis, case analysis, medical education, and other aspects.

[0097] (2) Obtain SAM and fine-tune SAM using the fourth training set.

[0098] The CLIP model is a powerful image encoder that performs well in processing a wide range of image types, but its performance may be limited when processing complex images in specific fields, such as medical images. Because medical images often contain a lot of details and specific anatomical structures, this places higher demands on the model's visual understanding ability. For example, subtle differences in X-rays, CT scan images, or MRI images may be crucial to the diagnosis of a disease, but these details may not be easily captured by the CLIP model. In addition, medical images often require accurate semantic segmentation to distinguish different tissue types or lesion areas. Although the CLIP model can handle general object recognition tasks well, it may be insufficient in the fine analysis of medical images. Therefore, relying solely on the CLIP model may not be sufficient to meet the high-precision requirements of medical image processing. In order to overcome the above limitations, an advanced semantic segmentation model such as SAM is introduced in this application to enhance the visual perception ability of the multimodal model.

[0099] SAM can automatically segment regions of interest from images and assign a category label to each pixel. This capability is particularly useful for medical images because it can help multimodal models more accurately identify and locate important anatomical structures or lesions. By integrating SAM into multimodal models, we can achieve the following goals:

[0100] ① Enhanced image understanding capabilities: SAM can provide more detailed image segmentation results, which helps multimodal models to more deeply understand the structures and details in medical images;

[0101] ② Improved lesion detection accuracy: When processing medical images, SAM can highlight key areas such as tumors, fractures, or other abnormalities, thereby improving the detection accuracy of multimodal models for these areas;

[0102] ③ Improve the overall performance of the model: By combining the segmentation capability of SAM and the general image understanding capability of CLIP, the multimodal model can show better performance on medical image processing tasks.

[0103] In this embodiment, the fourth training set may be used to fine-tune the SAM. The training samples in the fourth training set include medical images and masks, where the masks represent pixel-level annotations of objects to be identified in the medical images.

[0104] (3) Create a multimodal question-answering model based on the pre-trained multimodal model and the fine-tuned SAM model.

[0105] In specific implementation, the output of SAM can be used as additional image features to input into the multimodal model, or the multimodal model structure can be directly modified to integrate the function of SAM. In this way, the multimodal model can use the additional information provided by SAM to improve its understanding and processing of medical images, and obtain the final multimodal question-answering model.

[0106] In an optional embodiment, a second visual language connector is used to connect SAM to the multimodal large model to obtain a multimodal question-answering model.

[0107] After pre-training the multimodal model and fine-tuning SAM, both the multimodal model and SAM have their own initial weights.

[0108] Step 302, obtaining a first training set and a second training set, wherein the first training samples in the first training set include medical images, image annotations, simple question and answer texts with image annotations, and simple question and answer texts without image annotations, and the second training samples in the second training set include medical images, image annotations, detailed question and answer texts with image annotations, and detailed question and answer texts without image annotations.

[0109] The first and second training sets are crucial for building a multimodal question-answering model that can understand and process medical images and text. These training sets usually contain a large number of medical images and related clinical reports, diagnostic instructions, or surgical guidelines, providing rich training materials for the multimodal question-answering model, enabling it to learn image and text features that are unique to the medical field. For example, a multimodal question-answering model can identify specific types of fractures or lesions from X-rays and understand the corresponding medical terms.

[0110] Step 303, fixing the weights of the CLIP model, SAM and LLM, and using the first training sample to train the weights of the first visual-linguistic connector and the second visual-linguistic connector.

[0111] Specifically, fixing the weights of the CLIP model, the SAM, and the LLM, and using the first training sample to train the weights of the first visual language connector and the second language visual connector may include:

[0112] (1) Using the CLIP model with fixed weights, feature extraction is performed on the medical image in the first training sample to obtain the first image feature.

[0113] (2) Encode the first image feature using the first visual language connector to obtain a first image code.

[0114] (3) When the simple question-answer text without image annotations in the first training sample is used for training, the medical image is feature extracted using a fixed-weight SAM to obtain a second image feature; the second image feature is encoded using a second visual language connector to obtain a second image encoding; the simple question-answer text without image annotations is randomly masked and encoded using a word segmentation and embedding layer to obtain a first text encoding.

[0115] Randomly masking the simple question and answer text refers to randomly selecting some words from the simple question and answer text for masking.

[0116] (4) When the image annotations and the simple question-answer text with image annotations in the first training sample are used for training, the medical image and the image annotation are feature extracted using a fixed-weight SAM to obtain a second image feature; the second image feature is encoded using a second visual language connector to obtain a second image encoding; the simple question-answer text with image annotations is randomly masked and encoded using a word segmentation and embedding layer to obtain a first text encoding.

[0117] (5) The first image code, the second image code and the first text code are processed using a fixed-weight LLM to obtain a predicted text of a simple question-and-answer text.

[0118] In this embodiment, the first image code, the second picture code and the first text code may be concatenated, and then the concatenated code may be processed using LLM to obtain the predicted text.

[0119] The predicted text is the text obtained by recovering the simple question-answer text after being randomly masked.

[0120] (6) Update the weights of the first visual language connector and the second language visual connector according to the simple question and answer text and the predicted text.

[0121] In this embodiment, the simple question and answer text can be compared with the predicted text to determine whether the randomly masked words are the same as the restored words, and then the weights of the first visual language connector and the second language visual connector are updated based on the comparison result.

[0122] Step 304 , fix the weights of the CLIP model and the SAM, and train the weights of the first visual-linguistic connector, the second language visual connector, and the LLM using the first training sample.

[0123] Specifically, fixing the weights of the CLIP model and the SAM, and using the first training sample to train the weights of the first visual language connector, the second language visual connector, and the LLM may include:

[0124] (1) Using the CLIP model with fixed weights, feature extraction is performed on the medical images in the first training sample to obtain the third image features.

[0125] (2) Encode the third image feature using the first visual language connector to obtain a third image code.

[0126] (3) When the simple question-answer text without image annotations in the first training sample is used for training, the medical image is feature extracted using a fixed-weight SAM to obtain a fourth image feature; the fourth image feature is encoded using a second visual language connector to obtain a fourth image encoding; the simple question-answer text without image annotations is randomly masked and encoded using a word segmentation and embedding layer to obtain a second text encoding.

[0127] (4) When the image annotations and the simple question-answer text with image annotations in the first training sample are used for training, the medical image and the image annotation are feature extracted using a fixed-weight SAM to obtain a fourth image feature; the fourth image feature is encoded using a second visual language connector to obtain a fourth image encoding; the simple question-answer text with image annotations is randomly masked and encoded using a word segmentation and embedding layer to obtain a second text encoding.

[0128] (5) The third image code, the fourth image code and the second text code are processed using LLM to obtain a predicted text of the simple question-answer text.

[0129] (6) Update the weights of the first visual language connector, the second language visual connector and LLM according to the simple question and answer text and the predicted text.

[0130] Step 305 , fix the weights of the CLIP model and the SAM, and use the second training sample to train the weights of the first visual-linguistic connector, the second language-visual connector, and the LLM.

[0131] Specifically, fixing the weights of the CLIP model and the SAM, and using the second training sample to train the weights of the first visual language connector, the second language visual connector, and the LLM may include:

[0132] (1) Using the CLIP model with fixed weights, feature extraction is performed on the medical images in the second training sample to obtain the fifth image feature.

[0133] (2) Encode the fifth image feature using the first visual language connector to obtain a fifth image code.

[0134] (3) When the detailed question-and-answer text without image annotations in the second training sample is used for training, the medical image is feature extracted using a fixed-weight SAM to obtain a sixth image feature; the sixth image feature is encoded using a second visual language connector to obtain a sixth image encoding; the answers in the detailed question-and-answer text without image annotations are masked and encoded using a word segmentation and embedding layer to obtain a third text encoding.

[0135] Masking the answers in the detailed question and answer text means completely masking the answers and only keeping the questions.

[0136] (4) When the image annotations and the detailed question-and-answer text with the image annotations in the second training sample are used for training, the medical image and the image annotation are feature extracted using a fixed-weight SAM to obtain a sixth image feature; the sixth image feature is encoded using a second visual language connector to obtain a sixth image encoding; the answers in the detailed question-and-answer text with the image annotations are masked and encoded using a word segmentation and embedding layer to obtain a third text encoding.

[0137] (5) The fifth image code, the sixth image code and the third text code are processed using LLM to obtain the predicted text of the answer.

[0138] Predicted text is the recovery of fully occluded responses.

[0139] (6) Update the weights of the first visual-linguistic connector, the second language visual connector, and the LLM according to the answer and predicted text.

[0140] In this embodiment, the answer can be compared with the predicted text to determine whether the completely covered answer is the same as the restored answer, and then the weights of the first visual language connector, the second language visual connector and the LLM are updated based on the comparison result.

[0141] Step 306, loading the trained weights into the multimodal question-answering model to obtain a trained multimodal question-answering model.

[0142] Step 307: Process the test sample using the multimodal question-answering model to obtain the answer output by the multimodal question-answering model; obtain user feedback information on the answer; and optimize the multimodal question-answering model using the DPO algorithm and the feedback information.

[0143] In this embodiment, the direct preference optimization (DPO) algorithm is used to optimize the multimodal question-answering model. The DPO algorithm is capable of learning human preference habits in medical scenarios. Traditional training methods are not sufficient to achieve this goal because they mainly rely on minimizing the loss function to optimize the model, which does not always ensure that the output of the model is consistent with human preference habits. The DPO algorithm is an effective alignment optimization technique that enables the model to learn how to adjust its behavior based on user feedback to meet user expectations while minimizing the gap between model behavior and target behavior. For a multimodal question-answering model, this means that the multimodal question-answering model must not only be able to understand text information, but also be able to process image data and synthesize this information to generate higher quality answers:

[0144] 1. Value alignment: The output of the multimodal question-answering model should be consistent with human values ​​and social ethical standards. In the medical field, the multimodal question-answering model needs to understand and comply with medical ethics guidelines, such as patient privacy protection and the principle of non-harm. By using the DPO algorithm, the multimodal question-answering model can be trained to learn and imitate the interaction process between doctors and patients, as well as the ethics and values ​​reflected in these processes. For example, the multimodal question-answering model can be taught not to disclose sensitive information without explicit consent.

[0145] 2. Preference alignment: The DPO algorithm needs to focus on how the multimodal question-answering model adapts to the specific preferences of different users. In medical consultation scenarios, different patients may have different preferences. For example, some people prefer to get concise answers, while others may want more detailed explanations. Using the DPO algorithm, we can collect user feedback on the output of the multimodal question-answering model. This feedback can include positive comments, negative comments, or specific suggestions for improvement. Based on this feedback, the multimodal question-answering model can gradually learn how to better meet the personalized needs of users.

[0146] 3. Professional alignment: Multimodal question-answering models not only need to provide accurate information, but also need to ensure that their outputs are consistent with the professional knowledge system in the medical field. This requires multimodal question-answering models to not only understand complex medical terms and concepts, but also to be able to give evidence-based advice. Through the DPO algorithm, multimodal question-answering models can continuously learn from professionals, such as improving the quality of their answers by simulating conversations between doctors and patients. In addition, medical experts can be introduced as supervisors who can provide professional feedback to help multimodal question-answering models further improve their professionalism.

[0147] Specifically, when using test samples to test the multimodal question-answering model, a certain amount of feedback information from user feedback can be collected, including the user's acceptance level of the results (for example, 0 means completely unacceptable, 5 means average, and 10 means very willing to accept) and the user's modifications to the results, etc.; then the DPO algorithm is used to align the output results of the multimodal question-answering model on the feedback information set of user feedback, thereby improving the generation performance of the multimodal question-answering model to meet user expectations and application scenarios.

[0148] To summarize, the multimodal question-answering model training method for medical images provided in the embodiments of the present application trains the multimodal question-answering model by using a large number of first training sets and second training sets, thereby injecting medical knowledge into the multimodal question-answering model and improving the professional ability of the multimodal question-answering model to understand and process medical images and texts.

[0149] Processing medical images through pixel-level SAM can provide fine-grained and dense visual perception capabilities for the multimodal question-answering model, improving its ability to process delicate and complex medical images.

[0150] By adding image annotations of regions of interest in medical images, the focus of user questions can be narrowed from the entire image to the region of interest, thereby improving the accuracy and precision of the answers.

[0151] Through the DPO algorithm, we can collect users' feedback on the answers output by the multimodal question-answering model, and perform value alignment, preference alignment, and expertise alignment based on the feedback information to improve the security, accuracy, and professionalism of the answers output by the multimodal question-answering model.

[0152] After obtaining the trained multimodal question-answering model, users can use the multimodal question-answering model to consult medical questions. Taking the example of a user consulting about a CT image of the lungs, the use of the multimodal question-answering model includes:

[0153] Step 401, using the CLIP model to extract features from the medical image to be processed, to obtain the seventh image feature.

[0154] The user uploads a CT image of the lungs, and optionally selects the image of interest (if not, the entire image is selected), and enters a consulting question (for example, what is wrong with the lung area in the image?).

[0155] The CLIP model extracts features from the CT image of the lungs to obtain the first image features.

[0156] Step 402: Encode the seventh image feature using the first visual language connector to obtain the seventh image code.

[0157] VL-Connector1 encodes the seventh image feature to obtain the seventh image code (image tokens1).

[0158] Step 403, when the medical image has no image annotation, use SAM to extract features of the medical image to obtain the eighth image feature; use the second visual language connector to encode the eighth image feature to obtain the eighth image code; use the word segmentation and embedding layer to encode the problem to be processed to obtain the fourth text code.

[0159] SAM performs feature extraction on the CT image of the lungs to obtain the eighth image feature.

[0160] VL-Connector2 encodes the eighth image feature to obtain the eighth image code (image tokens2).

[0161] Tokenizer&embedding encodes the consulting questions to obtain the fourth text encoding (text tokens).

[0162] Step 404, when the medical image has image annotations, use SAM to extract features of the medical image and the image annotations to obtain the eighth image feature; use the second visual language connector to encode the eighth image feature to obtain the eighth image code; use the word segmentation and embedding layer to encode the problem to be processed to obtain the fourth text code.

[0163] SAM performs feature extraction on the CT image and the region of interest of the lung to obtain the eighth image feature.

[0164] VL-Connector2 encodes the eighth image feature to obtain the eighth image code (image tokens2).

[0165] Tokenizer&embedding encodes the consulting questions to obtain the fourth text encoding (text tokens).

[0166] Step 405: Use LLM to process the seventh image code, the eighth image code and the fourth text code to obtain the answer output by the multimodal question-answering model.

[0167] The seventh image code (image tokens1), the eighth image code (image tokens2) and the fourth text code (text tokens) are concatenated and fed into the LLM at the time step, and a professional answer is generated after analysis and reasoning.

[0168] For example, the answer generated by the multimodal question-answering model is: An air-fluid plane is visible in the right chest cavity, and the adjacent lung tissue is compressed and reduced in volume. There are a few blurred shadows in both lungs, and it is difficult to distinguish between solid shadows.

[0169] like Figure 5 As shown, it shows a structural block diagram of a multimodal question-answering model training device for medical images provided by an embodiment of the present application, and the multimodal question-answering model training device for medical images can be applied to a computer device. The multimodal question-answering model training device for medical images may include:

[0170] A creation module 510 is used to create a multimodal question-answering model, wherein the multimodal question-answering model includes a CLIP model, a SAM, a first visual language connector, a second visual language connector, and an LLM, wherein the CLIP model is connected to the LLM via the first visual language connector, and the SAM is connected to the LLM via the second visual language connector;

[0171] An acquisition module 520 is used to acquire a first training set and a second training set, wherein the first training samples in the first training set include medical images, image annotations, simple question-and-answer texts with image annotations, and simple question-and-answer texts without image annotations, and the second training samples in the second training set include medical images, image annotations, detailed question-and-answer texts with image annotations, and detailed question-and-answer texts without image annotations;

[0172] A training module 530 is used to fix the weights of the CLIP model, the SAM and the LLM, and train the weights of the first visual-linguistic connector and the second visual-linguistic connector using the first training sample;

[0173] The training module 530 is further used to fix the weights of the CLIP model and the SAM, and train the weights of the first visual-linguistic connector, the second-language visual connector, and the LLM using the first training sample;

[0174] The training module 530 is further used to fix the weights of the CLIP model and the SAM, and train the weights of the first visual-linguistic connector, the second language-visual connector, and the LLM using the second training sample;

[0175] The loading module 540 is used to load the trained weights into the multimodal question-answering model to obtain a trained multimodal question-answering model.

[0176] In an optional embodiment, the training module 530 is further configured to:

[0177] The test samples are processed using the multimodal question-answering model to obtain the answers output by the multimodal question-answering model;

[0178] Get user feedback on the answer;

[0179] The multimodal question answering model is optimized using the direct preference optimization (DPO) algorithm and feedback information.

[0180] In an optional embodiment, the training module 530 is further configured to:

[0181] Using a CLIP model with fixed weights to extract features from the medical image in the first training sample, to obtain first image features;

[0182] Encoding the first image feature using a first visual language connector to obtain a first image code;

[0183] When the simple question-answer text without image annotations in the first training sample is used for training, the medical image is feature extracted using the SAM with fixed weights to obtain the second image feature; the second image feature is encoded using the second visual language connector to obtain the second image encoding; the simple question-answer text without image annotations is randomly masked and encoded using the word segmentation and embedding layer to obtain the first text encoding;

[0184] When the image annotations and the simple question-answer text with image annotations in the first training sample are used for training, the medical image and the image annotation are feature extracted using the SAM with fixed weights to obtain the second image feature; the second image feature is encoded using the second visual language connector to obtain the second image encoding; the simple question-answer text with image annotations is randomly masked and encoded using the word segmentation and embedding layer to obtain the first text encoding;

[0185] The first image code, the second image code and the first text code are processed by using a fixed-weight LLM to obtain a predicted text of a simple question-answering text;

[0186] The weights of the first visual-language connector and the second visual-language connector are updated according to the simple question-answer text and the predicted text.

[0187] In an optional embodiment, the training module 530 is further configured to:

[0188] Using a CLIP model with fixed weights to extract features from the medical image in the first training sample, to obtain third image features;

[0189] encoding the third image feature using the first visual language connector to obtain a third image code;

[0190] When the simple question-answer text without image annotations in the first training sample is used for training, the medical image is feature extracted using the SAM with fixed weights to obtain a fourth image feature; the fourth image feature is encoded using the second visual language connector to obtain a fourth image encoding; the simple question-answer text without image annotations is randomly masked and encoded using the word segmentation and embedding layers to obtain a second text encoding;

[0191] When the image annotations and the simple question-answer text with image annotations in the first training sample are used for training, the medical image and the image annotation are feature extracted using the SAM with fixed weights to obtain the fourth image feature; the fourth image feature is encoded using the second visual language connector to obtain the fourth image encoding; the simple question-answer text with image annotations is randomly masked and encoded using the word segmentation and embedding layer to obtain the second text encoding;

[0192] Using LLM to process the third image code, the fourth image code and the second text code, to obtain a predicted text of the simple question-answering text;

[0193] The weights of the first visual-language connector, the second language visual connector, and the LLM are updated according to the simple question-answer text and the predicted text.

[0194] In an optional embodiment, the training module 530 is further configured to:

[0195] Using the CLIP model with fixed weights to extract features from the medical image in the second training sample, to obtain a fifth image feature;

[0196] encoding the fifth image feature using the first visual language connector to obtain a fifth image code;

[0197] When the detailed question-and-answer text without image annotations in the second training sample is used for training, the medical image is feature extracted using the SAM with fixed weights to obtain the sixth image feature; the sixth image feature is encoded using the second visual language connector to obtain the sixth image encoding; the answer in the detailed question-and-answer text without image annotations is masked and encoded using the word segmentation and embedding layer to obtain the third text encoding;

[0198] When the image annotations and the detailed question-and-answer text with the image annotations in the second training sample are used for training, the medical image and the image annotation are feature extracted using the SAM with fixed weights to obtain the sixth image feature; the sixth image feature is encoded using the second visual language connector to obtain the sixth image encoding; the answer in the detailed question-and-answer text with the image annotations is masked and encoded using the word segmentation and embedding layer to obtain the third text encoding;

[0199] Using LLM to process the fifth image code, the sixth image code and the third text code to obtain a predicted text of the answer;

[0200] Update the weights of the first visual-linguistic connector, the second language visual connector, and the LLM according to the answer and the predicted text.

[0201] In an optional embodiment, the creation module 510 is further used to:

[0202] Obtain a multimodal model, and use the third training set to pre-train the multimodal model, where the multimodal model includes a CLIP model, a first visual language connector, and an LLM;

[0203] Obtain SAM and fine-tune SAM using the fourth training set;

[0204] Create a multimodal question answering model based on the pre-trained multimodal model and the fine-tuned SAM model.

[0205] In an optional embodiment, the device further comprises:

[0206] An extraction module, used for performing feature extraction on the medical image to be processed by using the CLIP model to obtain a seventh image feature;

[0207] an encoding module, configured to encode the seventh image feature by using the first visual language connector to obtain a seventh image code;

[0208] The encoding module is further used for, when the medical image has no image annotation, extracting features from the medical image using SAM to obtain an eighth image feature; encoding the eighth image feature using the second visual language connector to obtain an eighth image encoding; encoding the problem to be processed using the word segmentation and embedding layer to obtain a fourth text encoding;

[0209] The encoding module is further used for, when the medical image has an image annotation, extracting features of the medical image and the image annotation using SAM to obtain an eighth image feature; encoding the eighth image feature using the second visual language connector to obtain an eighth image encoding; encoding the problem to be processed using the word segmentation and embedding layer to obtain a fourth text encoding;

[0210] The processing module is used to use LLM to process the seventh image code, the eighth image code and the fourth text code to obtain the answer output by the multimodal question-answering model.

[0211] To summarize, the multimodal question-answering model training device for medical images provided in the embodiment of the present application trains the multimodal question-answering model by using a large number of first training sets and second training sets, thereby injecting medical knowledge into the multimodal question-answering model and improving the professional ability of the multimodal question-answering model to understand and process medical images and texts.

[0212] Processing medical images through pixel-level SAM can provide fine-grained and dense visual perception capabilities for the multimodal question-answering model, improving its ability to process delicate and complex medical images.

[0213] By adding image annotations of regions of interest in medical images, the focus of user questions can be narrowed from the entire image to the region of interest, thereby improving the accuracy and precision of the answers.

[0214] Through the DPO algorithm, we can collect users' feedback on the answers output by the multimodal question-answering model, and perform value alignment, preference alignment, and expertise alignment based on the feedback information to improve the security, accuracy, and professionalism of the answers output by the multimodal question-answering model.

[0215] An embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the multimodal question-answering model training method for medical images as described above.

[0216] An embodiment of the present application provides a computer device, which includes the multimodal question-answering model training device for any medical image mentioned above.

[0217] It should be noted that: the multimodal question-answering model training device for medical images provided in the above embodiment only uses the division of the above functional modules as an example when training the multimodal question-answering model for medical images. In actual applications, the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the multimodal question-answering model training device for medical images is divided into different functional modules to complete all or part of the functions described above. In addition, the multimodal question-answering model training device for medical images provided in the above embodiment and the multimodal question-answering model training method embodiment for medical images belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0218] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0219] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A multimodal question-answering model training method for medical images, characterized in that: The method comprises: Creating a multimodal question-answering model, the multimodal question-answering model comprising a contrastive language-image pre-trained CLIP model, a segmentation model SAM, a first visual language connector, a second visual language connector, and a large language model LLM, the CLIP model being connected to the LLM via the first visual language connector, and the SAM being connected to the LLM via the second visual language connector; Obtaining a first training set and a second training set, wherein the first training samples in the first training set include medical images, image annotations, simple question-and-answer texts with image annotations, and simple question-and-answer texts without image annotations, and the second training samples in the second training set include medical images, image annotations, detailed question-and-answer texts with image annotations, and detailed question-and-answer texts without image annotations; Fixing the weights of the CLIP model, the SAM, and the LLM, and using the first training sample to train the weights of the first visual-linguistic connector and the second visual-linguistic connector; Fixing the weights of the CLIP model and the SAM, and using the first training sample to train the weights of the first visual-linguistic connector, the second language-visual connector, and the LLM; Fixing the weights of the CLIP model and the SAM, and using the second training sample to train the weights of the first visual-linguistic connector, the second language-visual connector, and the LLM; The trained weights are loaded into the multimodal question-answering model to obtain a trained multimodal question-answering model.

2. The multimodal question-answering model training method for medical images according to claim 1, characterized in that: The method further comprises: Processing the test sample using the multimodal question-answering model to obtain an answer output by the multimodal question-answering model; Obtaining user feedback information on the answer; The multimodal question-answering model is optimized using a direct preference optimization (DPO) algorithm and the feedback information.

3. The multimodal question-answering model training method for medical images according to claim 1, characterized in that: The step of fixing the weights of the CLIP model, the SAM, and the LLM, and using the first training sample to train the weights of the first visual language connector and the second language visual connector includes: Using a CLIP model with fixed weights to extract features from the medical images in the first training sample to obtain first image features; Encoding the first image feature using the first visual language connector to obtain a first image code; When the simple question-answer text without image annotations in the first training sample is used for training, the medical image is feature extracted using a SAM with a fixed weight to obtain a second image feature; the second image feature is encoded using the second visual language connector to obtain a second image encoding; the simple question-answer text without image annotations is randomly masked and encoded using a word segmentation and an embedding layer to obtain a first text encoding; When the image annotations and the simple question-answer text with image annotations in the first training sample are used for training, the medical image and the image annotation are extracted using a fixed-weight SAM to obtain a second image feature; the second image feature is encoded using the second visual language connector to obtain a second image encoding; the simple question-answer text with image annotations is encoded after random masking using a word segmentation and an embedding layer to obtain a first text encoding; Using a fixed-weight LLM to process the first image code, the second image code, and the first text code to obtain a predicted text of the simple question-and-answer text; The weights of the first visual-language connector and the second visual-language connector are updated according to the simple question-answer text and the predicted text.

4. The multimodal question-answering model training method for medical images according to claim 1, characterized in that: The step of fixing the weights of the CLIP model and the SAM, and using the first training sample to train the weights of the first visual language connector, the second language visual connector, and the LLM comprises: Using a CLIP model with fixed weights to extract features from the medical images in the first training sample, to obtain third image features; encoding the third image feature by using the first visual language connector to obtain a third image code; When the simple question-answer text without image annotations in the first training sample is used for training, the medical image is feature extracted by using the SAM with fixed weights to obtain a fourth image feature; the fourth image feature is encoded by using the second visual language connector to obtain a fourth image encoding; the simple question-answer text without image annotations is randomly masked and encoded by using the word segmentation and embedding layer to obtain a second text encoding; When the image annotations and the simple question-answer text with image annotations in the first training sample are used for training, the medical image and the image annotation are feature extracted using a fixed-weight SAM to obtain a fourth image feature; the fourth image feature is encoded using the second visual language connector to obtain a fourth image encoding; the simple question-answer text with image annotations is randomly masked and encoded using a word segmentation and an embedding layer to obtain a second text encoding; Using the LLM to process the third image code, the fourth image code and the second text code to obtain a predicted text of the simple question and answer text; The weights of the first visual-language connector, the second language-visual connector, and the LLM are updated according to the simple question-answer text and the predicted text.

5. The multimodal question-answering model training method for medical images according to claim 1, characterized in that: The step of fixing the weights of the CLIP model and the SAM, and using the second training sample to train the weights of the first visual language connector, the second language visual connector, and the LLM comprises: Using a CLIP model with fixed weights to extract features from the medical images in the second training sample, to obtain fifth image features; encoding the fifth image feature by using the first visual language connector to obtain a fifth image code; When the detailed question-and-answer text without image annotations in the second training sample is used for training, the medical image is feature extracted using a SAM with a fixed weight to obtain a sixth image feature; the sixth image feature is encoded using the second visual language connector to obtain a sixth image encoding; the answer in the detailed question-and-answer text without image annotations is masked and encoded using a word segmentation and an embedding layer to obtain a third text encoding; When the image annotations and the detailed question-and-answer text with the image annotations in the second training sample are used for training, the medical image and the image annotation are feature extracted using a SAM with fixed weights to obtain a sixth image feature; the sixth image feature is encoded using the second visual language connector to obtain a sixth image encoding; the answer in the detailed question-and-answer text with the image annotations is masked and encoded using a word segmentation and an embedding layer to obtain a third text encoding; Processing the fifth image code, the sixth image code and the third text code by using the LLM to obtain a predicted text of the answer; The weights of the first visual-linguistic connector, the second linguistic-visual connector, and the LLM are updated according to the answer and the predicted text.

6. The multimodal question-answering model training method for medical images according to claim 1, characterized in that: The creating of the multimodal question answering model comprises: Acquire a multimodal model, and pre-train the multimodal model using a third training set, wherein the multimodal model includes a CLIP model, a first visual language connector, and an LLM; Obtaining a SAM, and fine-tuning the SAM using a fourth training set; The multimodal question answering model is created based on the pre-trained multimodal model and the fine-tuned SAM model.

7. The multimodal question-answering model training method for medical images according to claim 1, characterized in that: The method further comprises: Using the CLIP model to perform feature extraction on the medical image to be processed, to obtain a seventh image feature; encoding the seventh image feature by using the first visual language connector to obtain a seventh image code; When the medical image has no image annotation, extracting features from the medical image using the SAM to obtain an eighth image feature; encoding the eighth image feature using the second visual language connector to obtain an eighth image code; encoding the problem to be processed using a word segmentation and an embedding layer to obtain a fourth text code; When the medical image has an image annotation, the SAM is used to extract features from the medical image and the image annotation to obtain an eighth image feature; the second visual language connector is used to encode the eighth image feature to obtain an eighth image code; and the word segmentation and embedding layer are used to encode the problem to be processed to obtain a fourth text code; The seventh image code, the eighth image code and the fourth text code are processed by using the LLM to obtain an answer output by the multimodal question-answering model.

8. A multimodal question-answering model training device for medical images, characterized in that: The device comprises: A creation module, used to create a multimodal question-answering model, wherein the multimodal question-answering model includes a contrastive language-image pre-trained CLIP model, a segmentation model SAM, a first visual language connector, a second visual language connector, and a large language model LLM, wherein the CLIP model is connected to the LLM via the first visual language connector, and the SAM is connected to the LLM via the second visual language connector; an acquisition module, configured to acquire a first training set and a second training set, wherein the first training samples in the first training set include medical images, image annotations, simple question-and-answer texts with image annotations, and simple question-and-answer texts without image annotations, and the second training samples in the second training set include medical images, image annotations, detailed question-and-answer texts with image annotations, and detailed question-and-answer texts without image annotations; A training module, used for fixing the weights of the CLIP model, the SAM and the LLM, and training the weights of the first visual-linguistic connector and the second visual-linguistic connector using the first training sample; The training module is further used to fix the weights of the CLIP model and the SAM, and train the weights of the first visual language connector, the second language visual connector, and the LLM using the first training sample; The training module is further used to fix the weights of the CLIP model and the SAM, and train the weights of the first visual language connector, the second language visual connector, and the LLM using the second training sample; A loading module is used to load the trained weights into the multimodal question-answering model to obtain a trained multimodal question-answering model.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the multimodal question-answering model training method for medical images as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device includes: the multimodal question-answering model training device for medical images as described in claim 8.

Images