Large model training method and system based on medical multi-modal data
By integrating the encoding and mapping layers of multimodal data and combining it with a multi-task learning framework, the multimodal processing problem of perioperative data is solved, cross-stage data alignment and task collaboration are achieved, and the accuracy and efficiency of medical data processing are improved.
Patent Information
- Application Number
- CN202510754411.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-06
- Publication Date
- 2025-09-19
AI Technical Summary
Existing technologies find it difficult to effectively integrate and process perioperative multimodal medical data, especially in the lack of cross-stage collaboration and unified processing capabilities between data at different stages. Single-modal models are unable to process multimodal data, and multimodal models perform poorly in heterogeneous data fusion and cross-stage task collaboration.
By integrating data modalities such as text, time series, images, and tables, designing specific encoders and using learnable mapping layers to achieve modality alignment, combined with multi-task output structure and joint training strategy, large language models are optimized to process multimodal data.
It realizes the effective fusion and alignment of multiple tasks of multimodal data, improves the accuracy and efficiency of data processing, implements the alignment mechanism, realizes the effective fusion of multiple tasks, and realizes the multi-task learning of multiple tasks.
Smart Images
Figure CN120674032A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a large model training method and system based on medical multimodal data, and belongs to the technical field of medical data processing. Background Art
[0002] With the rapid development of medical informatization, the amount of medical data generated is growing exponentially, encompassing everything from electronic medical records (EMRs), imaging data (such as CT and MRI), laboratory test results, surgical records, time-series monitoring data, and postoperative patient follow-up information. This data is not only massive in size but also diverse in form, encompassing multiple modalities such as text, tables, time-series data, and images, and exhibiting a high degree of heterogeneity. The multimodality and heterogeneity of medical data pose significant challenges to perioperative management and decision-making.
[0003] The perioperative period is usually divided into three stages: preoperative, intraoperative, and postoperative, and each stage involves different data types. The preoperative stage includes medical history records, imaging test results, laboratory data, and drug usage; the intraoperative stage is mainly based on real-time physiological monitoring data (such as electrocardiogram, blood pressure, pulse and other time series data) and event records during the operation; the postoperative stage mainly includes discharge reports, postoperative recovery monitoring, follow-up records, and postoperative imaging examinations. These data not only span different modalities, but also need to be dynamically processed according to the time dimension. However, existing machine learning models find it difficult to fully integrate and process these heterogeneous data, and cannot simultaneously complete complex tasks such as text generation, classification, time series prediction, and image generation.
[0004] In recent years, with the development of large language models (such as LLaMA3 and Qwen2), the use of pre-trained models for medical data processing has brought new hope. These large models have performed well in natural language understanding and generation, and have shown great potential in tasks such as medical text processing and clinical decision support. However, simple large language models are difficult to directly apply to perioperative management because of their limited support for multimodal data. The representation methods of non-text modal data such as time series data and image data are quite different from those of language models. How to effectively fuse and align these heterogeneous data has become a hot topic of current research. In addition, perioperative data involves dynamic and real-time changes, and the data at different stages are highly correlated. It is difficult for existing models to achieve global coordination and unified processing.
[0005] In general, the existing technologies have the following limitations:
[0006] (1) Models based on single modality. Currently, most of the widely used machine learning models in perioperative management rely on single modality data. For example, natural language processing models (such as BERT and BioBERT) used to process medical record text can effectively extract key information from preoperative text records, such as patients' medical history and allergy records. However, the limitation of these models is that they can only process text data and lack the corresponding processing capabilities for intraoperative monitoring data (such as time-series physiological signals) or postoperative imaging examination results (such as CT and MRI images). Similarly, other models used to process single modality (such as images, time series, etc.) are also unable to process data from other modalities. Therefore, in scenarios where multimodal data needs to be combined, single-modality models show great limitations and it is difficult to fully explore the complementary information between data.
[0007] (2) Models specifically for time series data. Common methods for time series modeling of intraoperative physiological monitoring data include long short-term memory networks (LSTM) or time series Transformer models based on attention mechanisms. These models are good at capturing dynamic changes in the time dimension and can better predict the changing trends of key physiological parameters during surgery. However, time series models are weak in processing other modalities (such as preoperative text records or postoperative imaging data), and it is difficult to achieve a global understanding of data at different stages before, during, and after surgery.
[0008] (3) Limitations of multimodal learning models. Although there have been some explorations of multimodal learning in recent years, such as visual-text models (such as CLIP, which can combine image and text data for processing), these models mainly focus on simple matching tasks of images and text. At the same time, most of these models rely on paired data (i.e., images and their corresponding text descriptions). However, in actual perioperative data, images, text, and time series data are often not directly related to each other, and existing multimodal models cannot solve the fusion problem of such unpaired data. In addition, existing multimodal models perform poorly in processing complex tasks across stages of the perioperative period, and it is difficult to simultaneously meet the needs of preoperative risk prediction, intraoperative status monitoring, and postoperative recovery assessment.
[0009] (4) Lack of cross-stage task collaboration capabilities. Perioperative data and tasks have a strong temporal correlation. Preoperative medical history and imaging examination results can provide important clues for intraoperative status monitoring, and changes in intraoperative physiological signals may have a direct impact on postoperative recovery. However, existing models are mostly optimized for a single stage or a single task, and cannot achieve cross-stage collaboration from preoperative to postoperative. For example, although some time series models can predict changes in intraoperative physiological indicators, they cannot use preoperative medical history data as input conditions. Similarly, postoperative recovery assessment cannot fully utilize intraoperative monitoring data.
[0010] Currently, various stages of the perioperative period (preoperative, intraoperative, and postoperative) involve a variety of data types, including text, time series data, images, and tabular data. There are significant modal differences between these data, such as preoperative text medical records, intraoperative dynamic physiological monitoring time series data, and postoperative imaging examination results. Existing technologies have difficulty processing these heterogeneous data simultaneously and lack effective alignment mechanisms for different modalities. At the same time, most existing models are limited to single-modal task processing and lack unified support for multiple perioperative tasks, such as preoperative classification tasks (risk assessment), intraoperative time series prediction tasks (monitoring abnormal events), and postoperative text generation tasks (discharge summaries).
[0011] In summary, existing technologies have obvious deficiencies in processing perioperative multimodal data. Single-modal models are unable to cope with complex multimodal data, and there is a lack of coordination between time series, image, and text processing models. Although multimodal models have initially demonstrated the possibility of cross-modal processing, their application in medical scenarios is not yet mature, especially in terms of the alignment of time series data and image data. In addition, existing technologies lack overall coverage of tasks at all stages of the perioperative period and cross-stage collaborative processing capabilities. Therefore, there is an urgent need for a large model training method that can uniformly process multimodal data and has the ability to process tasks across stages to fully support intelligent management of the perioperative period. Summary of the Invention
[0012] In response to the above-mentioned problems, the present invention provides a large model training method and system based on medical multimodal data. The present invention mainly solves the problems of multimodal medical data integration, cross-modal alignment and task collaboration in different stages of the perioperative period.
[0013] This invention integrates data modalities such as text, time series, images, and tables, designs specific encoders for each modality, and further achieves modality alignment through a learnable mapping layer, enabling large language models to uniformly process this heterogeneous data. Furthermore, by designing a multi-task output structure encompassing classification, regression, and text generation, it comprehensively covers various perioperative tasks.
[0014] The technical solution of the present invention is: a large model training method based on medical multimodal data, the specific steps of the method are as follows:
[0015] Step 1: Select a pre-trained large language model, freeze the attention mechanism and feedforward neural network of the selected pre-trained large language model, and retrain the embedding layer and normalization layer;
[0016] Step 2: Encode the multimodal data using a modal encoder;
[0017] Step 3: Add a projection layer after the output of each modality encoder to map the features of different modalities to the same feature space, align the features of different modalities, and then splice them to form a unified multimodal representation vector;
[0018] Step 4: Design a multi-task output structure to handle different task requirements during the perioperative period;
[0019] Step 5: Use a joint training strategy to optimize the pre-trained large language model under multimodal data and multi-task requirements.
[0020] Furthermore, the step 1 includes:
[0021] Step 1.1. Select an open-source pre-trained large language model that has been fully pre-trained on a large corpus, such as LLaMA3 or Qwen2.
[0022] Step 1.2: Freeze the attention mechanism and feedforward neural network of the selected pre-trained large language model to preserve the general knowledge of the pre-trained large language model.
[0023] Step 1.3: Retrain the embedding layer and normalization layer of the selected pre-trained large language model to adapt the pre-trained large language model to the new input modality.
[0024] Furthermore, in step 2, the multimodal data includes text, images, tables, and time series data, and encoding the multimodal data using a modal encoder specifically includes:
[0025] (1) The text data is remapped to a feature space suitable for the pre-trained large language model using a text data encoder through an embedding layer;
[0026] (2) Time series data uses a time series data encoder to capture short-term and long-term dependencies in time series data;
[0027] (3) The image data is subjected to feature extraction through an image data encoder;
[0028] (4) Tabular data is processed by the TabNet tabular data encoder to retain the structural information of the tabular data.
[0029] Furthermore, the step 3 includes:
[0030] Step 3.1: Add a learnable projection layer after the output of each modality encoder. The projection layer is a linear transformation layer that converts the feature vectors from each modality encoder into feature representations of the same dimension. During training, the projection layer is learnable. By optimizing the projection layer, the features of different modalities are automatically adjusted so that the features of different modalities have consistent semantic representations in the pre-trained large language model, thus implementing an alignment mechanism. This alignment mechanism ensures that the information between different modalities can complement each other and form a rich input.
[0031] Step 3.2: The multimodal features after the projection layer are fused through a splicing operation to form a unified multimodal representation vector, which serves as the final input of the pre-trained large language model. The spliced vector retains the key information of each modality and performs global contextual understanding through the attention mechanism of the pre-trained large language model.
[0032] Furthermore, the step 4 includes:
[0033] (1) Design a classification task output head for task classification in the preoperative stage. The classification task is implemented through a standard classification layer;
[0034] (2) Design a regression task output head for the time series prediction task during the intraoperative stage. The regression task output head uses a fully connected layer to perform linear regression prediction and output continuous value prediction results;
[0035] (3) Design a text generation task output head for the text generation task in the postoperative stage. The text generation task is completed through the generation module of the pre-trained large language model.
[0036] Furthermore, the step 5 includes:
[0037] Step 5.1. Design loss function: Design an independent loss function for each task. By weighted summing the losses of different tasks, we can achieve joint training of multiple tasks. For classification tasks, we use cross entropy loss, for regression tasks, we use mean squared error loss, and for text generation tasks, we use the autoregressive loss of a large language model.
[0038] Step 5.2: Modal alignment through contrastive learning: Introduce a contrastive learning mechanism to enhance the alignment of features from different modalities. By designing positive and negative sample comparisons, the data from different modalities of the same patient are made more similar, ensuring strong semantic associations between different modalities.
[0039] As a further solution of the present invention, a large model training system based on medical multimodal data includes: a module for executing the above-mentioned large model training method based on medical multimodal data.
[0040] As a further solution of the present invention, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the large model training method based on medical multimodal data is implemented.
[0041] As a further solution of the present invention, a non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned large model training method based on medical multimodal data.
[0042] As a further solution of the present invention, a computer program product includes a computer program, which implements the above-mentioned large model training method based on medical multimodal data when executed by a processor.
[0043] The beneficial effects of the present invention are:
[0044] Because the present invention adopts a strategy of partially freezing the pre-trained model, the number of parameters that need to be updated is significantly reduced, while retaining the extensive common sense and semantic knowledge learned by the large language model during pre-training. At the same time, through modality alignment and a multi-task learning framework, the present invention can more effectively process multimodal heterogeneous data and provide more efficient and accurate prediction, generation, and classification results in medical scenarios. Specifically,
[0045] 1. Efficient fusion of multimodal data: This paper uses the self-attention mechanism and graph neural network to intelligently fuse data from different modalities, avoiding the information loss problem caused by simple splicing.
[0046] 2. High feature extraction accuracy: Adopting dedicated feature extraction networks for different modalities ensures that the key information of each data modality can be fully captured and integrated, thereby improving the predictive performance of the overall model.
[0047] 3. High training efficiency for large models: This invention significantly improves the training efficiency of large multimodal models through pre-training and fine-tuning technology, combined with a distributed computing framework, and is suitable for processing large-scale medical data.
[0048] 4. Strong generalization: This method can be flexibly applied to different types of medical data processing tasks, such as disease prediction, personalized diagnosis and treatment, risk assessment, etc., and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0050] Example 1: Figure 1As shown, the present invention provides a large model training method based on medical multimodal data, which is used for full-process data processing during the perioperative period, such as the attached process. Figure 1 This method designs a specialized learnable mapping layer to align preoperative, intraoperative, and postoperative data from different modalities, provides a modality alignment mechanism, and supports multiple perioperative tasks through a multi-task learning framework. The specific steps of the method are as follows:
[0051] Step 1: Select a pre-trained large language model, freeze the attention mechanism and feedforward neural network of the selected pre-trained large language model, and retrain the embedding layer and normalization layer; Step 1 includes:
[0052] Step 1.1. Select an open-source pre-trained large language model that has been fully pre-trained on a large corpus, such as LLaMA3 or Qwen2. These models have been fully pre-trained on large corpora and have a broad commonsense knowledge base, smooth interaction capabilities, and strong contextual understanding capabilities.
[0053] Step 1.2: Freeze the attention mechanism and feed-forward neural network (FFN) of the selected pre-trained large language model to preserve the general knowledge of the pre-trained large language model; the embedding layer is input-specific and naturally needs to be retrained when the input domain is different. In addition, freezing these parts can reduce the number of training parameters, thereby improving training efficiency, especially when medical data is relatively scarce. The present invention adopts a strategy of partially freezing the model to maintain common sense and semantic understanding capabilities during the fine-tuning process;
[0054] Step 1.3: Retrain the Embedding Layer and Normalization Layer of the selected pre-trained large language model to adapt it to new input modalities, such as time series data and image features. The Embedding Layer can be adjusted to better match the input structure of the new modality with the original model. Retraining the Normalization Layer helps ensure stable gradient propagation and effective training when multimodal data flows into the model.
[0055] Through this freeze-fine-tuning strategy, the model can not only retain the common sense base acquired during pre-training, but also effectively adapt to new medical multimodal data and task requirements.
[0056] Step 2: Encode the multimodal data using a modal encoder. In step 2, in order to process multimodal data at different stages of the perioperative period, the multimodal data is encoded using a modal encoder. The main function of each modal encoder is to map the original data into a feature space that is compatible with the input of the large language model. The multimodal data includes text, images, tables, and time series data.
[0057] The multimodal data is encoded using a modality encoder, which includes:
[0058] (1) The text data is remapped to a feature space suitable for the pre-trained large language model using a text data encoder through an embedding layer. Preoperative medical records, intraoperative surgical records, and postoperative follow-up records use the text data encoder provided by the large language model. The large language model already has strong text understanding capabilities during the pre-training process, so these text data do not require further complex processing and only need to be remapped to a feature space suitable for the model through an embedding layer.
[0059] (2) Time series data uses a time series data encoder to capture the short-term and long-term dependencies in the time series data. Intraoperative physiological monitoring data (such as electrocardiogram, blood pressure, etc.) requires a dedicated time series data encoder due to its strong dynamics. The time series data encoder includes a Transformer-based time series model (PatchTST).
[0060] (3) Feature extraction of image data is performed through an image data encoder. Preoperative or postoperative imaging data (such as CT and MRI images) is subjected to feature extraction through an image data encoder. The image data encoder includes a ViT (Vision Transformer) model. The ViT model has powerful image processing capabilities and can effectively extract spatial features.
[0061] (4) Preoperative laboratory examination data and postoperative recovery monitoring data are structured tabular data. The tabular data are processed by TabNet tabular data encoder to retain the structural information of the tabular data.
[0062] Step 3: Add a projection layer after the output of each modality encoder to map features from different modalities to the same feature space, align the features from different modalities, and then concatenate them to form a unified multimodal representation vector. This invention adds a learnable projection layer after the output of each modality encoder to address the data heterogeneity between different modalities. The function of this projection layer is to map features from different modalities to the same feature space so that these modal data can be processed together as input to a large language model.
[0063] The step 3 comprises:
[0064] Step 3.1: Add a learnable projection layer after the output of each modality encoder. The projection layer is a linear transformation layer that converts the feature vectors from each modality encoder into feature representations of the same dimension. During training, the projection layer is learnable. By optimizing the projection layer, the features of different modalities are automatically adjusted so that the features of different modalities have consistent semantic representations in the pre-trained large language model, thus implementing an alignment mechanism. The alignment mechanism ensures that the information between different modalities can complement each other and form a rich input.
[0065] Step 3.2: The multimodal features after the projection layer are fused through concatenation to form a unified multimodal representation vector, which serves as the final input of the pre-trained large language model. The concatenated vector retains the key information of each modality and uses the attention mechanism of the pre-trained large language model to perform global contextual understanding.
[0066] Step 4: Design a multi-task output structure to handle different task requirements during the perioperative period;
[0067] The present invention designs a multi-task output structure to cover different task requirements during the perioperative period. These tasks include preoperative risk assessment, intraoperative time series prediction, and postoperative text generation. The multi-task output structure can handle multiple tasks simultaneously. Step 4 includes:
[0068] (1) Design a classification task output head for preoperative task classification. The classification task is implemented through a standard classification layer (such as Softmax). The preoperative task classification includes classification prediction of surgical risks.
[0069] (2) Design a regression task output head for time series prediction tasks during the intraoperative phase, such as trend prediction of physiological signals or early warning of abnormal events. The regression task output head uses a fully connected layer to perform linear regression prediction and output continuous value prediction results;
[0070] (3) Design a text generation task output head for the postoperative text generation task. The text generation task is completed by the generation module of the pre-trained large language model. The text generation task in the postoperative stage includes generating discharge reports or postoperative follow-up summaries. The generation module of the pre-trained large language model relies on the text generation capability of the large language model to generate fluent and medically logical report text.
[0071] Through this multi-task learning approach, the model can optimize multiple tasks simultaneously, avoiding the complexity of independent training for each task;
[0072] Step 5: Adopt a joint training strategy to optimize the pre-trained large language model under multimodal data and multi-task requirements. Adopt a joint training strategy to enable the model to be efficiently optimized under multimodal data and multi-task requirements. Step 5 includes:
[0073] Step 5.1. Design loss function: Design an independent loss function for each task. By weighted summing the losses of different tasks, we can achieve joint training of multiple tasks. For classification tasks, we use cross entropy loss, for regression tasks, we use mean squared error loss, and for text generation tasks, we use the autoregressive loss of a large language model.
[0074] Step 5.2: Modal alignment through contrastive learning: To enhance the alignment of features from different modalities, a contrastive learning mechanism is introduced. By designing a positive and negative sample comparison, data from different modalities of the same patient have a higher degree of similarity, ensuring strong semantic associations between different modalities.
[0075] As a further solution of the present invention, a large model training system based on medical multimodal data includes: a module for executing the above-mentioned large model training method based on medical multimodal data.
[0076] As a further solution of the present invention, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the large model training method based on medical multimodal data is implemented.
[0077] As a further solution of the present invention, a non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned large model training method based on medical multimodal data.
[0078] As a further solution of the present invention, a computer program product includes a computer program, which implements the above-mentioned large model training method based on medical multimodal data when executed by a processor.
[0079] The present invention can be widely applied to the following scenarios:
[0080] 1. Intelligent diagnosis and prediction: By combining a patient's imaging, time series, text, and tabular data, the model can more accurately predict disease risk and provide intelligent diagnostic support, especially for complex diseases such as cancer and cardiovascular disease.
[0081] 2. Personalized treatment plans: By integrating patients' medical records, laboratory results, imaging examinations, and genomic data, the model can provide doctors with more personalized treatment plan recommendations, improving the scientific nature and accuracy of clinical decision-making.
[0082] 3. Health monitoring and management: Utilizes time series data from wearable devices, combined with tables and text information in electronic medical records, to monitor patients' health status in real time and provide personalized health management and intervention recommendations. It is widely used in chronic disease management and remote health monitoring.
[0083] 4. Scientific research and drug development: In large-scale medical research, the combination of multimodal data analysis can provide richer information support for disease mechanism research, drug development and efficacy evaluation.
[0084] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.
Claims
1. A large model training method based on medical multimodal data, characterized by: The specific steps of the method are as follows: Step 1: Select a pre-trained large language model, freeze the attention mechanism and feedforward neural network of the selected pre-trained large language model, and retrain the embedding layer and normalization layer; Step 2: Encode the multimodal data using a modal encoder; Step 3: Add a projection layer after the output of each modality encoder to map the features of different modalities to the same feature space, align the features of different modalities, and then splice them to form a unified multimodal representation vector; Step 4: Design a multi-task output structure to handle different task requirements during the perioperative period; Step 5: Use a joint training strategy to optimize the pre-trained large language model under multimodal data and multi-task requirements.
2. The large model training method based on medical multimodal data according to claim 1, characterized in that: The step 1 comprises: Step 1.
1. Select an open-source pre-trained large language model that has been fully pre-trained on a large corpus, such as LLaMA3 or Qwen2. Step 1.2: Freeze the attention mechanism and feedforward neural network of the selected pre-trained large language model to preserve the general knowledge of the pre-trained large language model. Step 1.3: Retrain the embedding layer and normalization layer of the selected pre-trained large language model to adapt the pre-trained large language model to the new input modality.
3. The large model training method based on medical multimodal data according to claim 1, characterized in that: In step 2, the multimodal data includes text, images, tables, and time series data. The multimodal data is encoded using a modal encoder, specifically including: (1) The text data is remapped to a feature space suitable for the pre-trained large language model using a text data encoder through an embedding layer; (2) Time series data uses a time series data encoder to capture short-term and long-term dependencies in time series data; (3) The image data is subjected to feature extraction through an image data encoder; (4) Tabular data is processed by the TabNet tabular data encoder to retain the structural information of the tabular data.
4. The large model training method based on medical multimodal data according to claim 1, characterized in that: The step 3 includes: Step 3.1: Add a learnable projection layer after the output of each modality encoder. The projection layer is a linear transformation layer that converts the feature vectors from each modality encoder into feature representations of the same dimension. During training, the projection layer is learnable. By optimizing the projection layer, the features of different modalities are automatically adjusted so that the features of different modalities have consistent semantic representations in the pre-trained large language model, thus implementing an alignment mechanism. This alignment mechanism ensures that the information between different modalities can complement each other and form a rich input. Step 3.2: The multimodal features after the projection layer are fused through a splicing operation to form a unified multimodal representation vector, which serves as the final input of the pre-trained large language model. The spliced vector retains the key information of each modality and performs global contextual understanding through the attention mechanism of the pre-trained large language model.
5. The large model training method based on medical multimodal data according to claim 1, characterized in that: The step 4 comprises: (1) Design a classification task output head for task classification in the preoperative stage. The classification task is implemented through a standard classification layer; (2) Design a regression task output head for the time series prediction task during the intraoperative stage. The regression task output head uses a fully connected layer to perform linear regression prediction and output continuous value prediction results; (3) Design a text generation task output head for the text generation task in the postoperative stage. The text generation task is completed through the generation module of the pre-trained large language model.
6. The large model training method based on medical multimodal data according to claim 1, characterized in that: The step 5 comprises: Step 5.
1. Design loss function: Design an independent loss function for each task. By weighted summing the losses of different tasks, we can achieve joint training of multiple tasks. For classification tasks, we use cross entropy loss, for regression tasks, we use mean squared error loss, and for text generation tasks, we use the autoregressive loss of a large language model. Step 5.2: Modal alignment through contrastive learning: Introduce a contrastive learning mechanism to enhance the alignment of features from different modalities. By designing positive and negative sample comparisons, the data from different modalities of the same patient are made more similar, ensuring strong semantic associations between different modalities.
7. A large model training system based on medical multimodal data, characterized by: include: A module for executing the large model training method based on medical multimodal data as described in any one of claims 1 to 6.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the large model training method based on medical multimodal data as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large model training method based on medical multimodal data as described in any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the large model training method based on medical multimodal data as described in any one of claims 1 to 6 is implemented.