Medical multi-modal model training method and device, electronic equipment and storage medium

Abstract

The invention relates to a medical multi-modal model training method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining multi-modal medical data, including medical image data and medical text data, and converting the medical image data and the medical text data into an image mark and a text mark respectively; and encoding the image marks and the text marks to extract image features and text features, and inputting the image features and the text features into a medical multi-modal model for information association. And based on the associated information of the image features and the text features, performing iterative training through the medical multi-modal model according to a preset task target to obtain a trained medical multi-modal model. The medical multi-modal model is composed of a medical image segmentation model, a medical image classification model, a medical visual question-answering model and a medical text classification model, and task targets are a medical image segmentation task, a medical image classification task, a medical visual question-answering task and a medical text classification task.

Description

Technical Field

[0001] This invention relates to the fields of medical assistance and artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for training a medical multimodal model. Background Technology

[0002] Currently, existing multimodal large model training schemes primarily use the LLava multimodal large model, whose main goal is to effectively utilize the capabilities of pre-trained large language and visual models. This scheme selects a pre-trained large language model as the language model and uses a pre-trained CLIP visual encoder to provide visual features from the input image. To integrate the image vector space into the text vector space, a simple linear layer is used to connect the image features to the word embedding space. Specifically, a trainable projection matrix is ​​used to convert the visual features into language embeddings, which have the same dimension as the word embedding space in the large language model, resulting in a series of visual tokens. While this simple projection scheme is lightweight and allows for rapid iteration of data-centric experiments, this pre-training method only pre-trains for image-to-text generation and fails to pre-train for text-to-image generation tasks.

[0003] Furthermore, existing multimodal model training schemes, in order to learn medical domain knowledge, select 600,000 image-text pairs from PMC-15M and convert these text pairs into instruction-following data through a simple expansion method. However, the instructions simply describe the task of the image. Finally, for each sample, given the language instruction and image input, the model is required to predict the original title. However, this scheme can only keep the visual encoder weights unchanged and continue to update the pre-trained weights of the projection layer and language model. In order to train the model to follow various instructions and complete tasks in a conversational manner, the model is fine-tuned on collected biomedical language-image instruction-following data.

[0004] However, the existing technologies mentioned above have not been able to be optimized by combining specific domain tasks and domain expert models during the training process of multimodal models. Furthermore, the existing multimodal model training architecture has not been able to make full use of image information, and only extracts image information through pre-trained visual models. The process of multimodal data learning and training needs to be further improved. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, electronic device, and storage medium for training a medical multimodal model that can fully learn and utilize image and text information in multimodal data to address the aforementioned technical problems.

[0006] This invention provides a method for training a medical multimodal model, the method comprising: Acquire multimodal medical data, which includes medical image data and medical text data, and convert the medical image data and medical text data into image tags and text tags, respectively; The image and text tags are encoded to extract image and text features, and the image and text features are then input into a medical multimodal model for information association. Based on the correlation information of the image features and text features, the medical multimodal model is iteratively trained according to the preset task objectives to obtain the trained medical multimodal model. The medical multimodal model is composed of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model. The task objectives are medical image segmentation, medical image classification, medical visual question answering, and medical text classification.

[0007] In one embodiment, acquiring multimodal medical data, which includes medical image data and medical text data, and converting the medical image data and medical text data into image tags and text tags, respectively, includes: The medical image data is converted into discrete image tag sequences by an image segmenter. The image tag sequences have image start tags and image end tags to characterize the start and end of the image tag sequence. The medical text data is input into a text segmenter, which then segments the medical text data into a sequence of text tags consisting of multiple text tags. The text tag sequence has a text start tag and a text end tag, which are used to characterize the start and end of the text tag sequence.

[0008] In one embodiment, the encoding process of the image tags and text tags to extract image features and text features, and the input of the image features and text features into a medical multimodal model for information association, includes: In the autoregressive model training framework, the image label sequence and text label sequence are encoded and decoded through an autoregressive mechanism to generate prediction results of the multimodal medical data and the probability of occurrence of the prediction results; The image markers are mapped to fixed-dimensional image embedding vectors by an image encoder, so that the image decoder is invoked during the decoding stage to convert the image embedding vectors into RGB format image pixel values; The text encoder maps the text tokens to fixed-dimensional text embedding vectors, so that the text decoder can be invoked during the decoding phase to map the text embedding vectors to the corresponding words and sub-word units in the vocabulary.

[0009] In one embodiment, the step of iteratively training the medical multimodal model based on the association information of the image features and text features according to a preset task objective to obtain the trained medical multimodal model includes: When the target task is the medical image segmentation task, the medical image data is input into the medical image segmentation model to call multiple convolutional layers and downsampling layers of the medical image segmentation model to extract image features from the medical image data, and the image features are compressed in the bottleneck layer; In the decoding stage, the image features are converted into the image size of the medical image data through multiple upsampling layers and convolutional layers, and the image features of the corresponding layers of the decoder and encoder are fused to output the image segmentation result.

[0010] In one embodiment, the step of obtaining a trained medical multimodal model by iteratively training the medical multimodal model based on the association information of the image features and text features according to a preset task objective further includes: When the target task is the medical image classification task, the medical image data is input into the medical image classification model to call the medical image classification model to segment the medical image data into multiple image blocks, and each image block is converted into a fixed-dimensional image block embedding vector through an image word segmenter. The image patch embedding vectors corresponding to all image patches are input into a neural network model with a self-attention mechanism to capture the dependencies between different image patches and generate image classification results.

[0011] In one embodiment, the step of obtaining a trained medical multimodal model by iteratively training the medical multimodal model based on the association information of the image features and text features according to a preset task objective further includes: When the target task is the medical visual question answering task, the medical image data and the medical text data associated with the medical image data are input into the medical visual question answering model, so as to call the neural network model to capture the dependency relationship between image tags and text tags in the medical image data and medical text data, and obtain fused visual-text features; The visual-text features are input into a text decoder to output medical visual question-and-answer text; When the target task is the medical text classification task, the medical text data is input into the medical text classification model based on the BERT model architecture, so that the medical text classification model can be invoked to output the text classification result according to the preset medical type based on the text tags corresponding to the medical text data.

[0012] In one embodiment, the method further comprises: Obtain the hidden layer results predicted by the medical multimodal model, which include output image data and output text data, and convert the output image data and output text data into corresponding output image label sequences and output text label sequences; The output image tag sequence and output text tag sequence are linearly transformed and feature extracted using a fusion network structure with a cross-attention mechanism, and the image features and text features corresponding to the hidden layer results are fused.

[0013] The present invention also provides a medical multimodal model training device for implementing the medical multimodal model training method described in any of the above claims, the device comprising: A data preprocessing module is used to acquire multimodal medical data, which includes medical image data and medical text data, and convert the medical image data and medical text data into image tags and text tags, respectively; The multimodal data association module is used to encode the image tags and text tags to extract image features and text features, and input the image features and text features into the medical multimodal model for information association; The multimodal model training module is used to iteratively train the medical multimodal model according to a preset task objective based on the correlation information between the image features and text features, so as to obtain the trained medical multimodal model. The medical multimodal model is composed of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model. The task objectives are medical image segmentation, medical image classification, medical visual question answering, and medical text classification.

[0014] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the medical multimodal model training method as described above.

[0015] The present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the medical multimodal model training method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the medical multimodal model training method as described above.

[0017] The aforementioned medical multimodal model training method, apparatus, electronic device, and storage medium acquire multimodal medical data containing medical image data and medical text data, and convert the medical image data and medical text data into image tags and text tags, respectively. The image tags and text tags are then encoded to extract image features and text features, which are then input into a medical multimodal model composed of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model for information association. Finally, based on the association information of image features and text features, the medical multimodal model is iteratively trained according to the task objectives of the medical image segmentation task, medical image classification task, medical visual question answering task, and medical text classification task to obtain the trained medical multimodal model. This invention, based on multimodal medical data fused from medical image data and medical text, combines task objective classification from multiple specific medical fields and performs targeted training and fusion of multiple expert models to obtain the desired medical multimodal model. This not only fully utilizes the association information between images and text in multimodal medical data but also improves the comprehensiveness and accuracy of the trained medical multimodal model in multimodal data analysis. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the medical multimodal model training method provided by the present invention; Figure 2 A schematic diagram of the multimodal model training architecture of the medical multimodal model training method provided in a specific embodiment of the present invention; Figure 3 A schematic diagram of the fusion network architecture of the medical multimodal model training method in a specific embodiment of the present invention; Figure 4 The second schematic diagram of the medical multimodal model training method provided by the present invention; Figure 5 The third flowchart illustrates the medical multimodal model training method provided by this invention. Figure 6 The fourth flowchart illustrates the medical multimodal model training method provided by this invention. Figure 7 The fifth flowchart illustrates the medical multimodal model training method provided by this invention. Figure 8 The sixth flowchart illustrating the medical multimodal model training method provided by this invention; Figure 9 The seventh flowchart illustrating the medical multimodal model training method provided by this invention; Figure 10 A schematic diagram of the structure of the medical multimodal model training device provided by the present invention; Figure 11 This is a diagram of the internal structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The following combination Figures 1 to 11 The present invention describes a medical multimodal model training method, apparatus, electronic device, and storage medium.

[0022] like Figure 1 As shown, in one embodiment, a medical multimodal model training method includes the following steps: Step S110: Obtain multimodal medical data, which includes medical image data and medical text data, and convert the medical image data and medical text data into image tags and text tags, respectively.

[0023] Specifically, the server acquires multimodal medical data consisting of a mixture of medical image data and medical text data, and converts the medical image data and medical text data into image tag sequences and text tag sequences, respectively, composed of image tags and text tags.

[0024] Combination Figure 2As shown in the specific embodiments, the medical multimodal model training method provided by this invention includes tasks such as medical image segmentation, medical image classification, medical VQA (medical visual question answering), and medical text classification. When a user performs a task, the medical intent recognition classifier needs to determine the specific task type of the user input. Intent recognition is a classifier for user input, specifically categorized into four types: medical segmentation, medical image classification, medical VQA, and medical text classification. These task types also correspond to four expert models (medical image segmentation model, medical image classification model, medical visual question answering model, and medical text classification model). The intent recognition classifier will provide predictions to one of the four expert models, ultimately obtaining the hidden layer result of the expert model. Regardless of the task type, predictions can be made for the medical multimodal model, and the hidden layer result of the multimodal model can be obtained. Finally, the hidden layer features of the expert model and the hidden layer features of the medical multimodal model are fused, and the final recognition result is output after passing through the fusion network.

[0025] In this embodiment, the pre-training process of the medical multimodal model involves medical image segmentation, medical image classification, and medical VQA tasks. These tasks use multimodal training data of images and text, and also include training tasks with medical plain text pairs, to enable the model to understand and associate information from multiple modalities. During model training, different data types are first input into the model. For image data, an Image Tokenizer is used to convert the image into a series of image tokens; for text data, a Text Tokenizer is used to convert it into text tokens. This process transforms the raw multimodal data into a form that the model can process.

[0026] Step S120: Encode the image tags and text tags to extract image features and text features, and input the image features and text features into the medical multimodal model for information association.

[0027] Specifically, the server encodes the image and text tags obtained in step S110 to extract the image and text features corresponding to the medical image data and medical text data, respectively. The extracted image and text features are then input into a medical multimodal model composed of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model for the association processing of image and text information.

[0028] Combination Figure 2As shown in the specific embodiment, the medical multimodal model training method provided by this invention converts image data and text data in medical multimodal data into image tokens and text tokens. These image tokens and text tokens are then processed by corresponding encoders. Image tokens are processed by an Image Decoder, while text tokens are processed by a Text Decoder. The encoder's role is to extract features and information from the input data for subsequent fusion and understanding. The image features and text features processed by the encoder are then fed into the medical multimodal model for further fusion and learning. At this stage, the model attempts to understand and associate data from different modalities, for example, combining visual information from images with semantic information from text to achieve a deeper level of understanding and analysis.

[0029] In this embodiment, during the encoding and decoding of medical image data, for example, if the original medical image is a microscope image of a tissue slice, the original image is input to an image tokenizer, which converts the image into a series of image tokens. These tokens typically represent different parts or features of the image. The image token sequence begins with "..."<IMG_S> "Start with "<IMG_E> "End" indicates the beginning and end of the image. Afterward, each image marker is mapped to a fixed-size embedding vector, the dimension of which is called the embedding size. During the decoding phase, the embedding vector is converted back to image pixel values, typically in RGB format, where each pixel is represented by three values ​​indicating the intensity of red, green, and blue colors. In the decoding phase, the image decoder converts the embedding vector sequence back to image pixel values, generating the final output image.

[0030] In the process of decoding and encoding medical text data, taking "What plane is this image?" as an example, the question text is input into a text tokenizer, which segments the text into a series of text tokens. These tokens are typically words or sub-word units, and the sequence of text tokens begins with "..."<TXT_S> "Start with "<TXT_E> "End" indicates the beginning and end of the text. Each text token is mapped to a fixed-size embedding vector, the dimension of which is also called the embedding size. During decoding, the embedding vector is mapped back to specific words or sub-words in the vocabulary, the size of which is called the vocabulary size. In the decoding phase, the text decoder converts the sequence of embedding vectors back into text, generating the final output text.

[0031] Step S130: Based on the correlation information between image features and text features, the medical multimodal model is iteratively trained according to the preset task objectives to obtain the trained medical multimodal model.

[0032] The medical multimodal model consists of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model. The objectives are medical image segmentation, medical image classification, medical visual question answering, and medical text classification.

[0033] Specifically, the server inputs the association information of image features and text features output in step S120 into the medical multimodal model, so that the medical multimodal model can be iteratively trained according to the task objectives of the medical image segmentation task, medical image classification task, medical visual question answering task and medical text classification task to learn the association information between images and text, and finally obtain the trained medical multimodal model.

[0034] Combination Figure 2 As shown in the specific embodiments, the medical multimodal model training method provided by this invention, during the model training process, trains and optimizes the model according to specific task objectives (such as classification, image and text generation, etc.). Through a large amount of multimodal data and its extracted features and iterative learning training, the model gradually improves its performance and accuracy in processing multimodal data. Ultimately, the fully trained medical multimodal model can be applied to various practical scenarios, such as medical image analysis, disease diagnosis assistance, and health consultation, providing more accurate and comprehensive medical services. The entire model training process aims to enable the model to effectively process and understand data from different modalities, thereby playing a greater role in the medical field.

[0035] In this embodiment, the pre-trained framework demonstrates an autoregressive model for processing joint representations of images and text. This model achieves cross-modal information fusion and generation by serializing images and text and encoding and decoding them within a unified framework. Within this framework, through...<IMG_S> and<IMG_E> Mark the beginning and end of the image sequence, and by...<TXT_S> and<TXT_E> The start and end points of the corresponding text sequences are marked. These markings help the model identify the type and boundaries of the input data, thereby correctly switching and adjusting its strategies during processing.

[0036] It should be noted that the core of an autoregressive model is an autoregressive mechanism. It predicts the next element based on previous context information, and the expression is:

[0037] In the formula, This represents the target element currently being predicted, while It is a sequence of elements that has already been processed; The model parameters are represented by the formula, which describes the probabilistic process of predicting and generating results based on the input. In practical applications, this autoregressive model can be used for tasks such as image caption generation and visual question answering, enabling the understanding and representation of complex scenes by learning the relationship between images and text.

[0038] In this embodiment, a medical image segmentation model is used to accurately segment medical images. For example, using an image of a skin lesion as input data, the model first employs an encoder consisting of multiple convolutional and downsampling layers to extract image features. The input image first passes through a series of convolutional and downsampling layers, progressively reducing the image's spatial size while increasing the abstraction of features. Subsequently, a bottleneck layer connects the encoder and decoder at the deepest level, further compressing features and extracting the most abstract and important information. In the decoder, multiple upsampling and convolutional layers progressively restore the feature maps to the original image size. Each layer of the decoder is fused with the feature maps of the corresponding layer in the encoder (indicated by red arrows) to retain more detailed information. Finally, the output is a segmentation result; for example, a white segmentation mask represents the segmentation result of the lesion region.

[0039] In this embodiment, the medical image classification model first takes a medical image (such as a pathological slide image) as input and segments it into multiple patches. For example, the image is segmented into 9 patches. These segments are then fed into an Image Tokenizer, which converts each patch into a fixed-length vector representation, for example, each patch is converted into a 768-dimensional vector (3×16×16). The vector sequence processed by the Image Tokenizer is then fed into a Transformer Encoder. The Transformer Encoder is a neural network structure based on a self-attention mechanism, capable of capturing global dependencies between different patches, thereby generating higher-level feature representations for image classification tasks. The medical image classification model design fully utilizes the advantages of the Transformer in processing sequential data and is suitable for complex medical image analysis tasks.

[0040] In this embodiment, the medical visual question-answering model aims to process and understand medical images and related text information to generate accurate answers or explanations. In the application, firstly, a medical image is input, which may be an X-ray, CT scan, or other types of medical imaging data. Simultaneously, there is a text input, which may be a clinical question or description related to the image. Then, the image and text are processed by their respective tokenizers. The Image Tokenizer converts the image into a series of image tokens, which can be understood and processed by the machine learning model; the Text Tokenizer segments the text into a series of words or phrases, i.e., text tokens, for subsequent processing. In the Transformer Encoder stage, the tokens from the image and text are fed into a Transformer Encoder to process sequential data and capture long-term dependencies in the data. In the medical visual question-answering model, the Transformer Encoder is responsible for integrating the information from the image and text to generate a representation that fuses visual and linguistic features. Finally, the fused feature representation is passed to the Text Decoder, which is responsible for generating the final output text, which can be a description of the image, an answer to a clinical question, or any meaningful text content based on the image and text input. The medical visual question-answering model is designed to provide intelligent visual question-answering services in the medical field, helping doctors, researchers, or patients to better understand and analyze medical images and related information.

[0041] In this embodiment, the medical text classification model adopts the BERT model architecture for medical text classification. At the embedding layer, the text tags corresponding to the input text data are classified according to preset types. For example, Tok 1 corresponds to E1, Tok 2 corresponds to E2, Tok 1 and Tok 2 represent text tags, and E1 and E2 represent preset types. Finally, the final classification result is output at the output layer.

[0042] In this embodiment, as Figure 3 As shown, the hidden layer results from different expert models and the hidden layer results from the general medical multimodal model need to be processed by a fusion network structure with a cross-attention mechanism to obtain the final output. First, the text and images corresponding to the hidden layer results from different expert models and the general medical multimodal model are converted into corresponding text tag sequences and image tag sequences. Starting from the input sequences corresponding to these two hidden layer results, linear transformation and feature extraction are performed on the input sequences corresponding to the two hidden layer results. In this process, one hidden layer result is used as the key and the other as the value. The key-value pair and the user's input query target are input into the cross-attention mechanism. In this process, each element in the query sequence is compared with all elements in the key sequence to determine their relevance or similarity. After processing by multiple cross-attention mechanisms, the preliminary fusion result is output. The result is then processed by residual connections and normalization (Add&Norm) and the FFN network structure to finally output the overall multimodal medical large model test results.

[0043] The aforementioned medical multimodal model training method acquires multimodal medical data containing medical image data and medical text data, and converts the medical image data and medical text data into image tags and text tags, respectively. The image tags and text tags are then encoded to extract image features and text features, which are then input into a medical multimodal model composed of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model for information association. Finally, based on the association information of image and text features, the medical multimodal model is iteratively trained according to the task objectives of the medical image segmentation task, medical image classification task, medical visual question answering task, and medical text classification task to obtain the trained medical multimodal model. This method, based on multimodal medical data fused from medical image data and medical text, combines multiple specific medical domain task objective classifications and conducts targeted training and fusion of multiple expert models to obtain the desired medical multimodal model. It not only fully utilizes the association information between images and text in multimodal medical data but also improves the comprehensiveness and accuracy of the trained medical multimodal model in multimodal data analysis.

[0044] like Figure 4 As shown, in one embodiment, the medical multimodal model training method provided by the present invention includes the following steps in step S110: Step S111: The medical image data is converted into a discrete image tag sequence by an image segmenter. The image tag sequence has an image start tag and an image end tag, which are used to characterize the start and end of the image tag sequence.

[0045] Step S112: Input the medical text data into the text segmenter to call the text segmenter to segment the medical text data into a text tag sequence composed of multiple text tags. The text tag sequence has a text start tag and a text end tag, which are used to represent the start and end of the text tag sequence.

[0046] like Figure 5 As shown, in one embodiment, the medical multimodal model training method provided by the present invention includes the following steps in step S120: Step S121: In the autoregressive model training framework, the image label sequence and text label sequence are encoded and decoded through the autoregressive mechanism to generate prediction results of multimodal medical data and the probability of occurrence of the prediction results.

[0047] Step S122: The image markers are mapped to fixed-dimensional image embedding vectors by the image encoder, so that the image decoder is called in the decoding stage to convert the image embedding vectors into image pixel values ​​in RGB format.

[0048] Step S123: The text encoder maps the text tokens to fixed-dimensional text embedding vectors so that the text decoder can be called during the decoding stage to map the text embedding vectors to the corresponding words and sub-word units in the vocabulary.

[0049] like Figure 6 As shown, in one embodiment, the medical multimodal model training method provided by the present invention includes the following steps in step S130: Step S131: When the target task is medical image segmentation, the medical image data is input into the medical image segmentation model to call multiple convolutional layers and downsampling layers of the medical image segmentation model to extract image features from the medical image data, and the image features are compressed in the bottleneck layer.

[0050] In step S132, during the decoding stage, the image features are converted into the image size of the medical image data through multiple upsampling layers and convolutional layers, and the image features of the corresponding layers of the decoder and encoder are fused to output the image segmentation result.

[0051] like Figure 7 As shown, in one embodiment, the medical multimodal model training method provided by the present invention further includes the following steps in step S130: Step S133: When the target task is medical image classification, the medical image data is input into the medical image classification model to call the medical image classification model to segment the medical image data into multiple image blocks, and the image segmenter converts each image block into a fixed-dimensional image block embedding vector.

[0052] Step S134: Input the image patch embedding vectors corresponding to all image patches into the neural network model with self-attention mechanism to capture the dependencies between different image patches and generate image classification results.

[0053] like Figure 8 As shown, in one embodiment, the medical multimodal model training method provided by the present invention further includes the following steps in step S130: Step S135: When the target task is a medical visual question answering task, medical image data and medical text data associated with the medical image data are input into the medical visual question answering model to call the neural network model to capture the dependency relationship between image tags and text tags in the medical image data and medical text data, and obtain fused visual-text features.

[0054] Step S136: Input the visual-text features into the text decoder to output medical visual question-and-answer text.

[0055] Step S137: When the target task is a medical text classification task, the medical text data is input into the medical text classification model based on the BERT model architecture, so as to call the medical text classification model to output the text classification result according to the text tags corresponding to the medical text data and the preset medical type.

[0056] like Figure 9 As shown, in one embodiment, the medical multimodal model training method provided by the present invention further includes the following steps: Step S910: Obtain the hidden layer results of the medical multimodal model prediction, which include output image data and output text data, and convert the output image data and output text data into corresponding output image label sequences and output text label sequences.

[0057] Step S920: The output image label sequence and output text label sequence are linearly transformed and feature extracted through a fusion network structure with cross-attention mechanism, and the image features and text features corresponding to the hidden layer results are fused.

[0058] The medical multimodal model training device provided by the present invention will be described below. The medical multimodal model training device described below and the medical multimodal model training method described above can be referred to in correspondence.

[0059] like Figure 10 As shown, in one embodiment, a medical multimodal model training device includes a data preprocessing module 1010, a multimodal data association module 1020, and a multimodal model training module 1030.

[0060] The data preprocessing module 1010 is used to acquire multimodal medical data, which includes medical image data and medical text data, and convert the medical image data and medical text data into image tags and text tags, respectively.

[0061] The multimodal data association module 1020 is used to encode image and text tags to extract image and text features, and input the image and text features into the medical multimodal model for information association.

[0062] The multimodal model training module 1030 is used to iteratively train the medical multimodal model based on the association information of image features and text features according to the preset task objectives, so as to obtain the trained medical multimodal model.

[0063] The medical multimodal model consists of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model. The objectives are medical image segmentation, medical image classification, medical visual question answering, and medical text classification.

[0064] In this embodiment, the data preprocessing module 1010 of the medical multimodal model training device provided by the present invention is specifically used for: Medical image data is converted into discrete image tag sequences using an image segmenter. Each image tag sequence has an image start tag and an image end tag, which are used to characterize the start and end of the image tag sequence.

[0065] The medical text data is input into a text segmenter, which then divides the medical text data into a sequence of text tags consisting of multiple text tags. The text tag sequence has a text start tag and a text end tag, which are used to represent the beginning and end of the text tag sequence.

[0066] In this embodiment, the multimodal data association module 1020 of the medical multimodal model training device provided by the present invention is specifically used for: In the autoregressive model training framework, image and text label sequences are encoded and decoded through an autoregressive mechanism to generate prediction results for multimodal medical data and the probability of those predictions occurring.

[0067] The image encoder maps image tags to fixed-dimensional image embedding vectors, so that the image decoder can be invoked during the decoding phase to convert the image embedding vectors into RGB format image pixel values.

[0068] The text encoder maps text tokens to fixed-dimensional text embedding vectors, and the text decoder is invoked during the decoding phase to map the text embedding vectors to the corresponding words and sub-word units in the vocabulary.

[0069] In this embodiment, the medical multimodal model training device provided by the present invention, specifically uses the multimodal model training module 1030 for: When the target task is medical image segmentation, the medical image data is input into the medical image segmentation model, which calls multiple convolutional layers and downsampling layers of the medical image segmentation model to extract image features from the medical image data, and compresses the image features at the bottleneck layer.

[0070] In the decoding stage, image features are converted into the image size of medical image data through multiple upsampling layers and convolutional layers, and the image features of corresponding layers of the decoder and encoder are fused to output the image segmentation result.

[0071] In this embodiment, the modal model training module 1030 of the medical multimodal model training device provided by the present invention is further used for: When the target task is medical image classification, the medical image data is input into the medical image classification model to call the medical image classification model to segment the medical image data into multiple image patches, and the image segmenter converts each image patch into a fixed-dimensional image patch embedding vector.

[0072] The image patch embedding vectors corresponding to all image patches are input into a neural network model with a self-attention mechanism to capture the dependencies between different image patches and generate image classification results.

[0073] In this embodiment, the modal model training module 1030 of the medical multimodal model training device provided by the present invention is further used for: When the target task is medical visual question answering, medical image data and medical text data associated with the medical image data are input into the medical visual question answering model. The neural network model is invoked to capture the dependency relationship between image tags and text tags in the medical image data and medical text data, and the fused visual-text features are obtained.

[0074] Visual-text features are input into a text decoder to output medical visual question-and-answer text.

[0075] When the target task is medical text classification, the medical text data is input into a medical text classification model based on the BERT model architecture. The medical text classification model is then invoked to output the text classification results according to the text tags corresponding to the medical text data and the preset medical type.

[0076] In this embodiment, the medical multimodal model training device provided by the present invention further includes a model output fusion module, used for: Obtain the hidden layer results of the medical multimodal model prediction, which include output image data and output text data, and convert the output image data and output text data into corresponding output image label sequences and output text label sequences.

[0077] A fusion network structure with a cross-attention mechanism is used to perform linear transformation and feature extraction on the output image label sequence and the output text label sequence, and the image features and text features corresponding to the hidden layer results are fused.

[0078] Figure 11 This example illustrates a schematic diagram of the physical structure of an electronic device, which can be a smart terminal. Its internal structure diagram can be as follows: Figure 11As shown. The electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a medical multimodal model training method, which includes: Acquire multimodal medical data, which includes medical image data and medical text data, and convert the medical image data and medical text data into image tags and text tags, respectively; Image and text tags are encoded to extract image and text features, which are then input into a medical multimodal model for information association. Based on the correlation information of image features and text features, the medical multimodal model is iteratively trained according to the preset task objectives to obtain the trained medical multimodal model; The medical multimodal model consists of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model. The objectives are medical image segmentation, medical image classification, medical visual question answering, and medical text classification.

[0079] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device to which the present invention is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0080] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the above-described medical multimodal model training method.

[0081] In another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, it implements the aforementioned medical multimodal model training method.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0083] By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for training a medical multimodal model, characterized in that, The method comprises: Acquire multimodal medical data, which includes medical image data and medical text data, and convert the medical image data and medical text data into image tags and text tags, respectively; The image and text tags are encoded to extract image and text features, and the image and text features are then input into a medical multimodal model for information association. Based on the correlation information of the image features and text features, the medical multimodal model is iteratively trained according to the preset task objectives to obtain the trained medical multimodal model. The medical multimodal model is composed of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model. The task objectives are medical image segmentation, medical image classification, medical visual question answering, and medical text classification.

2. The medical multimodal model training method according to claim 1, characterized in that, The acquisition of multimodal medical data, which includes medical image data and medical text data, and the conversion of the medical image data and medical text data into image tags and text tags, respectively, includes: The medical image data is converted into discrete image tag sequences by an image segmenter. The image tag sequences have image start tags and image end tags to characterize the start and end of the image tag sequence. The medical text data is input into a text segmenter to segment the medical text data into a text tag sequence consisting of multiple text tags. The text tag sequence has a text start tag and a text end tag to characterize the start and end of the text tag sequence.

3. The medical multimodal model training method according to claim 2, characterized in that, The process of encoding the image and text tags to extract image and text features, and then inputting these features into a medical multimodal model for information association, includes: In the autoregressive model training framework, the image label sequence and text label sequence are encoded and decoded through an autoregressive mechanism to generate prediction results of the multimodal medical data and the probability of occurrence of the prediction results; The image markers are mapped to fixed-dimensional image embedding vectors by an image encoder, so that the image decoder is invoked during the decoding stage to convert the image embedding vectors into RGB format image pixel values; The text encoder maps the text tokens to fixed-dimensional text embedding vectors, so that the text decoder can be invoked during the decoding phase to map the text embedding vectors to the corresponding words and sub-word units in the vocabulary.

4. The medical multimodal model training method according to claim 1, characterized in that, The association information based on the image features and text features is used to iteratively train the medical multimodal model according to a preset task objective, resulting in a trained medical multimodal model, including: When the target task is the medical image segmentation task, the medical image data is input into the medical image segmentation model to call multiple convolutional layers and downsampling layers of the medical image segmentation model to extract image features from the medical image data, and the image features are compressed in the bottleneck layer; In the decoding stage, the image features are converted into the image size of the medical image data through multiple upsampling layers and convolutional layers, and the image features of the corresponding layers of the decoder and encoder are fused to output the image segmentation result.

5. The medical multimodal model training method according to claim 4, characterized in that, The association information based on the image features and text features, through iterative training of the medical multimodal model according to a preset task objective, to obtain the trained medical multimodal model, further includes: When the target task is the medical image classification task, the medical image data is input into the medical image classification model to call the medical image classification model to segment the medical image data into multiple image blocks, and each image block is converted into a fixed-dimensional image block embedding vector through an image word segmenter. The image patch embedding vectors corresponding to all image patches are input into a neural network model with a self-attention mechanism to capture the dependencies between different image patches and generate image classification results.

6. The medical multimodal model training method according to claim 5, characterized in that, The association information based on the image features and text features, through iterative training of the medical multimodal model according to a preset task objective, to obtain the trained medical multimodal model, further includes: When the target task is the medical visual question answering task, the medical image data and the medical text data associated with the medical image data are input into the medical visual question answering model, so as to call the neural network model to capture the dependency relationship between image tags and text tags in the medical image data and medical text data, and obtain fused visual-text features; The visual-text features are input into a text decoder to output medical visual question-and-answer text; When the target task is the medical text classification task, the medical text data is input into the medical text classification model based on the BERT model architecture, so that the medical text classification model can be invoked to output the text classification result according to the preset medical type based on the text tags corresponding to the medical text data.

7. The medical multimodal model training method according to claim 1, characterized in that, The method further includes: Obtain the hidden layer results predicted by the medical multimodal model, which include output image data and output text data, and convert the output image data and output text data into corresponding output image label sequences and output text label sequences; The output image tag sequence and output text tag sequence are linearly transformed and feature extracted using a fusion network structure with a cross-attention mechanism, and the image features and text features corresponding to the hidden layer results are fused.

8. A medical multimodal model training device, characterized in that, The apparatus for implementing the medical multimodal model training method according to any one of claims 1 to 7, the apparatus comprising: A data preprocessing module is used to acquire multimodal medical data, which includes medical image data and medical text data, and convert the medical image data and medical text data into image tags and text tags, respectively; The multimodal data association module is used to encode the image tags and text tags to extract image features and text features, and input the image features and text features into the medical multimodal model for information association; The multimodal model training module is used to iteratively train the medical multimodal model according to a preset task objective based on the correlation information between the image features and text features, so as to obtain the trained medical multimodal model. The medical multimodal model is composed of a medical image segmentation model, a medical image classification model, a medical visual question answering model, and a medical text classification model. The task objectives are medical image segmentation, medical image classification, medical visual question answering, and medical text classification.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the medical multimodal model training method according to any one of claims 1 to 7.

10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the medical multimodal model training method according to any one of claims 1 to 7.

Images