Multimodal medical large model construction method and device, system, and storage medium

Abstract

The application discloses a multimodal medical large model construction method and device, system and storage medium, and comprises the following steps: S1, acquiring a training data set; S2, obtaining a cross-modal contrast learning model according to the training data set; and S3, fine-tuning the cross-modal contrast learning model based on transfer learning to obtain a task model adapted to a downstream task, and realizing classification task requirements. According to the technical scheme, more comprehensive and refined features can be extracted from different information sources, so that the accuracy of nasal cavity disease prediction is improved.

Description

Technical Field

[0001] This invention belongs to the field of large model technology, and in particular relates to a method, device, system, and storage medium for constructing a multimodal medical large model. Background Technology

[0002] Large-scale AI models are developing rapidly, demonstrating immense potential and numerous broad applications across various fields. From natural language processing and computer vision to medicine and finance, the capabilities of large-scale AI models are continuously improving. In recent years, with advancements in hardware computing power and the expansion of data scale, the training speed and accuracy of large-scale AI models have significantly increased. They are capable of processing massive amounts of data, rapidly recognizing patterns, and generating high-quality text, images, and other content.

[0003] However, the development of large-scale nasal cavity medical models also faces some key challenges, one of which is the accuracy of training data. If one of the data sources relied upon by the large-scale medical model is not accurate, comprehensive, or of high quality, it will lead to problems when the model is handling complex medical tasks. Nasal cavity medical data is very special, covering a large amount of medical history, medical imaging, experimental results, etc., but this data is usually highly specialized, complex, and private, making it difficult to obtain a sufficiently large scale and high-quality training data. Therefore, improving the quality of nasal cavity medical data and ensuring its diversity and accuracy have become key steps in improving the performance of large-scale nasal cavity medical models.

[0004] Currently, while large models built from modal data may exhibit certain advantages on specific tasks, their overall accuracy and generalization ability remain limited. Modal data images, such as models trained solely on text, single information sources, or biosignals, often struggle to fully capture the multidimensional information involved in complex tasks. This limitation results in insufficient performance when dealing with tasks requiring the integration of multidimensional information, thus affecting the accuracy and reliability of nasal disease diagnosis or prediction. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, device, system and storage medium for constructing a multimodal medical large model, which can extract more comprehensive and refined features from different information sources, thereby improving the accuracy of nasal disease prediction.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for constructing a multimodal medical large model includes:

[0008] Step S1: Obtain the training dataset;

[0009] Step S2: Obtain a cross-modal contrastive learning model based on the training dataset;

[0010] Step S3: Fine-tune the cross-modal contrastive learning model based on transfer learning to obtain a task model that adapts to downstream tasks and meets the requirements of classification tasks.

[0011] Preferably, the training dataset contains nasal endoscopy records, each of which includes an examination report and endoscopic images.

[0012] Preferably, step S2 includes:

[0013] Image reconstruction of endoscopic images was performed using the MAE paradigm;

[0014] Based on the inspection report and the reconstructed images, a multimodal cross-attention module is used to merge the visual representations of multiple images into the text encoder:

[0015] A cross-modal contrastive learning model is obtained by performing hierarchical, multi-granular image report cross-modal alignment through a multi-level semantic alignment module.

[0016] The present invention also provides a multimodal medical large model construction device, comprising:

[0017] The acquisition module is used to acquire the training dataset;

[0018] The training module is used to obtain a cross-modal contrastive learning model based on the training dataset;

[0019] The fine-tuning module is used to fine-tune the cross-modal contrastive learning model based on transfer learning, obtaining a task model adapted to downstream tasks and meeting the requirements of classification tasks.

[0020] Preferably, the training dataset contains nasal endoscopy records, each of which includes an examination report and endoscopic images.

[0021] Preferably, the training module includes:

[0022] The first processing unit is used to reconstruct the endoscopic image using the MAE paradigm;

[0023] The second processing unit is used to merge the visual representations of multiple images into the text encoder based on the inspection report and the reconstructed images, using a multimodal cross-attention module.

[0024] The third processing unit is used to perform hierarchical multi-granularity image report cross-modal alignment through a multi-level semantic alignment module to obtain a cross-modal contrastive learning model.

[0025] The present invention also provides a multimodal medical large model construction system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a multimodal medical large model construction method when run by the processor.

[0026] This invention also provides a storage medium storing a computer program, which executes a multimodal medical large model construction method during runtime.

[0027] Compared with the prior art, the present invention has the following technical effects:

[0028] 1. Cross-modal data fusion can significantly enhance the insight capabilities of large models. In the field of nasal medicine, the application of cross-modal data is extremely important. For example, information from various modalities, such as patient medical records, medical images (e.g., X-rays, CT scans), laboratory test results, and genomic data, typically contain different levels and perspectives of pathological features.

[0029] 2. Cross-modal large models can effectively compensate for the deficiencies of single-modal data. For example, nasal diseases may not be obvious in imaging data, but by combining genetic data or biomarkers, they can be captured more accurately. In addition, cross-modal large models can better cope with the uncertainties in medical data and reduce the risk of misjudgment because information from different modalities can be mutually verified and complemented, improving the model's risk robustness.

[0030] 3. Cross-modal data integration can improve the accuracy of models in diagnosing nasal diseases. By integrating information from different levels, cross-modal models can analyze patients' health status at a more granular level, providing more comprehensive, accurate and personalized health care solutions, thereby greatly promoting the development of precision medicine. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0032] Figure 1 This is a flowchart of the multimodal medical large model construction method according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of multimodal large model pre-training;

[0034] Figure 3 This is a schematic diagram of the Multimodal Cross-Attention Module (MMCA).

[0035] Figure 4 This is a schematic diagram of the Multilevel Semantic Alignment (MLSA) module. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Example 1:

[0039] like Figure 1 As shown, this embodiment of the invention provides a method for constructing a multimodal medical large model, including:

[0040] Step S1: Obtain the training dataset;

[0041] Step S2: Obtain a cross-modal contrastive learning model based on the training dataset;

[0042] Step S3: Fine-tune the cross-modal contrastive learning model based on transfer learning to obtain a task model that adapts to downstream tasks and meets the requirements of classification tasks.

[0043] In one embodiment of the present invention, in step S1, the training dataset includes nasal endoscopy records. Each nasal endoscopy record includes an examination report and endoscopic images. The examination report includes abnormalities and diagnostic suggestions for 17 nasal anatomical regions, and four endoscopic images, including views of the left and right nasal cavities and the left and right nasopharynx. Each nasal endoscopy record is parsed to generate a report in the format [Abnormal Anatomical Region]: [Morphological Description], with a standardized diagnostic sentence appended.

[0044] As one embodiment of the present invention, such as Figure 2 As shown, step S2 includes:

[0045] Step S21: Reconstruct the endoscopic image using the MAE paradigm;

[0046] Step S22: Based on the inspection report and the reconstructed images, the visual representations of multiple images are merged into the text encoder using a multimodal cross-attention module:

[0047] Step S23: Perform hierarchical multi-granularity image report cross-modal alignment through a multi-level semantic alignment module to obtain a cross-modal contrastive learning model.

[0048] Furthermore, in step S21, the image representation is learned by reconstructing a high-resolution image from a low-resolution image. The underlying mechanism is to enhance the potential embedding label by incorporating detailed local information through super-resolution techniques, including:

[0049] Step 211: Each high-resolution image is downsampled by a factor of two to produce a low-resolution input image;

[0050] Step 212: Each input image is randomly masked by 75% and segmented into non-overlapping image patches;

[0051] Step 213: L low-resolution image patches of each input image that are not occluded. This will be used as input to reconstruct the H occluded high-resolution image patches. This is used as the true baseline for comparison and as the target for the final image reconstruction.

[0052] Step 214, Image Encoder Each unmasked image patch Encoding into an L-subregion representation, it can be represented as Where l represents the spatial index of each sub-region, and C represents the feature dimension;

[0053] Step 215: Measure the difference between the predicted reconstructed image patch and the true high-resolution image patch using the mean squared error (MSE), where all pixel values ​​are normalized to the range [0,1]. The objective function for image reconstruction is calculated using the following formula:

[0054]

[0055] Furthermore, in step S22, the performance of image coding is enhanced by participating in the reconstruction of the masked report, maximizing the use of paired report-image visual representation learning, such as... Figure 3 As shown, it includes:

[0056] Step 221: Use WordPiece tags to mark each report Convert to tags, where P represents the number of report tags;

[0057] Step 222, Shielding 50% of the information is replaced with the [MASK] tag to ensure consistent tag count, and is not masked. 50% of the information is used as input to the model to reconstruct...

[0058] Step 223, Uncoded Report Marks and corresponding images The embedded tags are input into the MMCA module to generate cross-modal embedded tags. MMCA is a variant of the multi-head self-attention module (MSA) designed for flexible input of cross-modal embedded tags for text and images, and it can be expressed by the following formula:

[0059]

[0060] Where Q represents the process of embedding Bert. The generated text tags, K and V, represent the concatenation of four endoscopic image embedding tags. This image embedding representation comes from The derivative of the local representation, where C denotes the embedding dimension. It is the head projection matrix. This represents the multi-head linear projection matrix. Finally, the cross-modal embedding representation generated by MMCA is added to the text embedding tag. It can be represented as

[0061] To reconstruct the masked text markup, the multimodal embedded markup is generated by the text encoder. and text tag classifier The text reconstruction training is supervised by the cross-entropy between the predicted and masked text tokens, as follows:

[0062]

[0063] Furthermore, in step S23, to effectively capture the mutual and internal similarities between paired medical image reports, NasVLM combines cross-modal contrastive learning with masked images and reports for modeling, such as... Figure 4 As shown, it includes:

[0064] Step 231: Each nasal endoscopy record consists of four images and one report, constructed into a uniform sentence structure: [Anatomical region]: [Morphological representation of the region]. Using this structured sentence structure facilitates multi-level supervised learning, promotes multi-level semantic alignment (MLSA) between image representations and text embeddings at two different granularities, thereby enhancing the effectiveness of semantic representation alignment.

[0065] Step 232: Global image features and report-level text embeddings are synchronized to achieve global semantic alignment.

[0066] Sentence embedding representation in the report By embedding unmasked text The average is used for aggregation. Therefore, the similarity matrix representing the P sentence embeddings and the 4 image feature representations can be calculated using the following formula:

[0067]

[0068] The weighted representation of the image can be obtained through sentence embedding and the local feature matching function Z, calculated as follows:

[0069]

[0070] Where τ1 and τ2 are scaling factors, and Φ represents the cosine similarity measure between the two vectors.

[0071] The contrastive loss for local semantic alignment can be calculated as follows:

[0072]

[0073] Where τ3 is the scaling factor and N is the training count.

[0074] For global semantic alignment, global image representation and report-level embedding representation The loss function is calculated using paired and unpaired report samples and four corresponding images:

[0075]

[0076] The report embeds representation Represented by text tag embedding The global image representation is obtained by aggregating the average values. Represented by images

[0077] Step 233: Iterate the model parameters according to the NasVLM loss function: The loss function consists of four loss terms, including image reconstruction loss. Report on reconstruction losses Multimodal contrast loss and It can be calculated using the following formula:

[0078]

[0079] Where, λ R , λ ls and λ gs Hyperparameters used to balance the losses of image reconstruction, report reconstruction, and cross-modal contrastive learning tasks during specific model training processes.

[0080] As one embodiment of the present invention, step S3 includes:

[0081] Step S31, Nasopharyngeal carcinoma screening task

[0082] The first step was to collect and organize the training data: The Nas-Endo training dataset was divided into three parts: a training set containing 60,900 records, an internal validation set containing 7,000 records, and an external validation set containing 230 records. Each record contained a medical report and four nasal endoscopic images. Each report used a standardized sentence format: [anatomical region]: [morphological features of the region], and included a diagnostic suggestion.

[0083] The second step involves training the training data to obtain the trained model. Specifically, ViT-B is used as the shared visual encoder, and BioClinicalBERT is used as the training text encoder, with output embedding representations of the same size (C=768). All images are resized to 224×224 with a center crop, the ViT block size is 16×16, and each image has a total of 14×14 blocks. The image occlusion rate is set to 75%, and the reported occlusion rate is set to 50%. The model uses AdamW as the optimizer with a learning rate of 1e-4, a weight decay of 0.02, and runs for 300 epochs on four Tesla A800 GPUs with a batch size of 256. The coefficients τ1, τ2, and τ3 are all set to 0.01, and λ... R , λ ls and λ gs Set them to 1, 0.05, and 0.1 respectively.

[0084] Step 3: Fine-tuning the Nasopharyngeal Carcinoma (NPC) Dataset. The NPC fine-tuning dataset contains 27,362 nasal endoscopy images from 8 renowned NPC centers. This includes 10,693 biopsy-confirmed NPC images (NPC, 39.08%), 7,073 biopsy-confirmed benign hyperplasia (BH, 25.85%), and 9,596 normal images (NOR, 35.07%). The dataset is divided into training, validation, and test sets, with proportions of 70%, 20%, and 10%, respectively. Cross-entropy loss is employed, and the optimal initial learning rate is searched from 3e-2, 3e-3, and 5e-4 to achieve the best performance on the validation set.

[0085] The fourth step was to validate the performance of the fine-tuned nasopharyngeal carcinoma screening model. The 95% confidence intervals for all metrics were calculated using the Clopper-Pearson exact method to assess the model's independent performance. t-SNE was used to visualize the high-level embeddings of the final layer of the image encoder, thereby evaluating the quality of the representations generated by the proposed method.

[0086] Step S32, Post-endoscopic assessment task

[0087] Step 1: Collect training data: The Nas-Endo training dataset was divided into three parts: a training set containing 60,900 records, an internal validation set containing 7,000 records, and an external validation set containing 230 records. Each record contained a medical report and four nasal endoscopic images. Each report used a standardized sentence formatted as [anatomical region]: [morphological features of the region], and included a diagnostic suggestion.

[0088] The second step involves training the training data to obtain the trained model. Specifically, ViT-B is used as the shared visual encoder, and BioClinicalBERT is used as the training text encoder, with output embedding representations of the same size (C=768). All images are resized to 224×224 with a center crop, ViT's block size is 16×16, and each image has a total of 14×14 blocks. The image occlusion rate is set to 75%, and the reported occlusion rate is set to 50%. The model uses AdamW as the optimizer with a learning rate of 1e-4, a weight decay of 0.02, and runs for 300 epochs on four Tesla A800 GPUs with a batch size of 256. The coefficients τ1, τ2, and τ3 are all set to 0.01, and λ... R , λ ls and λ gs Set them to 1, 0.05, and 0.1 respectively.

[0089] The third step involved fine-tuning the dataset using the Endoscopic Sinus Surgery (ESS) dataset. This dataset contains 1,600 images, each annotated by two nasal endoscopy experts with over 15 years of clinical experience. The dataset was then split into training, validation, and test sets, allocated 80%, 10%, and 10% respectively. Cross-entropy loss was employed, and the optimal initial learning rate was searched from 3e-2, 3e-3, and 5e-4 to achieve the best performance on the validation set.

[0090] The fourth step was to validate the performance of the fine-tuned postoperative ESS model. The 95% confidence intervals for all metrics were calculated using the Clopper-Pearson exact method to assess the model's independent performance. t-SNE was used to visualize the high-level embeddings of the last layer of the image encoder, thereby evaluating the quality of the representations generated by the proposed method.

[0091] Step S33, Nasopharyngeal carcinoma segmentation task

[0092] Step 1: Collect and organize training data: The Nas-Endo training dataset was divided into three parts: a training set containing 60,900 records, an internal validation set containing 7,000 records, and an external validation set containing 230 records. Each record contained a medical report and four nasal endoscopic images. Each report used a standardized sentence format: [anatomical region]: [morphological features of the region], and included a diagnostic suggestion.

[0093] The second step involves training the training data to obtain the trained model. Specifically, ViT-B is used as the shared visual encoder, and BioClinicalBERT is used as the training text encoder, with output embedding representations of the same size (C=768). All images are resized to 224×224 with a center crop, the ViT block size is 16×16, and each image has a total of 14×14 blocks. The image occlusion rate is set to 75%, and the reported occlusion rate is set to 50%. The model uses AdamW as the optimizer with a learning rate of 1e-4, a weight decay of 0.02, and runs for 300 epochs on four Tesla A800 GPUs with a batch size of 256. The coefficients τ1, τ2, and τ3 are all set to 0.01, and λ... R , λ ls and λ gs Set them to 1, 0.05, and 0.1 respectively.

[0094] The third step involves fine-tuning using a nasopharyngeal carcinoma segmentation dataset. This dataset contains 100 pixel-level segmented and annotated images of nasopharyngeal carcinoma lesions from the Eye, Ear, Throat and Nose Hospital of Fudan University. Data grouping and annotation were completed using the Tencent AIMIS open platform. This dataset is divided into training, validation, and test sets, allocated 80%, 10%, and 10% respectively. Cross-entropy loss is employed, and the optimal initial learning rate is searched from 3e-2, 3e-3, and 5e-4 to obtain the best performance on the validation set.

[0095] The fourth step is to validate the performance of the fine-tuned nasopharyngeal carcinoma segmentation model. The 95% confidence intervals for all metrics were calculated using the Clopper-Pearson exact method to assess the model's independent performance. t-SNE was used to visualize the high-level embeddings of the last layer of the image encoder, thereby evaluating the quality of the representations generated by the proposed method.

[0096] Example 2:

[0097] This invention also provides a multimodal medical large model construction device, comprising:

[0098] The acquisition module is used to acquire the training dataset;

[0099] The training module is used to obtain a cross-modal contrastive learning model based on the training dataset;

[0100] The fine-tuning module is used to fine-tune the cross-modal contrastive learning model based on transfer learning, obtaining a task model adapted to downstream tasks and meeting the requirements of classification tasks.

[0101] As one embodiment of the present invention, the training dataset contains nasal endoscopy records, and each nasal endoscopy record contains an examination report and an endoscopic image.

[0102] As one embodiment of the present invention, the training module includes:

[0103] The first processing unit is used to reconstruct the endoscopic image using the MAE paradigm;

[0104] The second processing unit is used to merge the visual representations of multiple images into the text encoder based on the inspection report and the reconstructed images, using a multimodal cross-attention module.

[0105] The third processing unit is used to perform hierarchical, multi-granularity image report cross-modal alignment through a multi-level semantic alignment module to obtain a cross-modal contrastive learning model.

[0106] Example 3:

[0107] This invention also provides a multimodal medical large model construction system, including: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a multimodal medical large model construction method when run by the processor.

[0108] Example 4:

[0109] This invention also provides a storage medium storing a computer program, which executes a multimodal medical large model construction method during runtime.

[0110] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for constructing a multimodal medical large model, characterized in that, include: Step S1: Obtain the training dataset; Step S2: Obtain a cross-modal contrastive learning model based on the training dataset; Step S3: Fine-tune the cross-modal contrastive learning model based on transfer learning to obtain a task model that adapts to downstream tasks and meets the requirements of classification tasks.

2. The method for constructing a multimodal medical large model as described in claim 1, characterized in that, The training dataset contains nasal endoscopy records, each of which includes an examination report and endoscopic images.

3. The method for constructing a multimodal medical large model as described in claim 2, characterized in that, Step S2 includes: Image reconstruction of endoscopic images was performed using the MAE paradigm; Based on the inspection report and the reconstructed images, a multimodal cross-attention module is used to merge the visual representations of multiple images into the text encoder: A cross-modal contrastive learning model is obtained by performing hierarchical, multi-granular image report cross-modal alignment through a multi-level semantic alignment module.

4. A multimodal medical large-scale model construction device, characterized in that, include: The acquisition module is used to acquire the training dataset; The training module is used to obtain a cross-modal contrastive learning model based on the training dataset; The fine-tuning module is used to fine-tune the cross-modal contrastive learning model based on transfer learning to obtain a task model that adapts to downstream tasks and meets the requirements of classification tasks.

5. The multimodal medical large-scale model construction device as described in claim 4, characterized in that, The training dataset contains nasal endoscopy records, each of which includes an examination report and endoscopic images.

6. The multimodal medical large-scale model construction device as described in claim 5, characterized in that, The training module includes: The first processing unit is used to reconstruct the endoscopic image using the MAE paradigm; The second processing unit is used to merge the visual representations of multiple images into the text encoder based on the inspection report and the reconstructed images, using a multimodal cross-attention module. The third processing unit is used to perform hierarchical multi-granularity image report cross-modal alignment through a multi-level semantic alignment module to obtain a cross-modal contrastive learning model.

7. A multimodal medical large-scale model construction system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program executing the multimodal medical large model construction method as described in any one of claims 1-3 when run by the processor.

8. A storage medium, characterized in that, The storage medium stores a computer program, which executes the multimodal medical large model construction method as described in any one of claims 1-3 when it runs.

Images