Multi-modal medical data generation method and related device

Abstract

The invention belongs to a data generation method, provides a multi-modal medical data generation method and a related device, and aims to solve the technical problems of multi-modal medical data resource limitation, modal missing and data scarcity of an existing medical data generation method. Obtaining a fine-tuning medical large model; obtaining a text instruction and a multi-modal data stream associated with the task, and obtaining a task type, an existing modal list, a target modal list and a task instruction after analysis of the text instruction according to the text instruction and the multi-modal data stream; according to the existing modalities in the existing modality list and the task instruction, obtaining a comprehensive shared feature representation; encoding and decoding the comprehensive shared feature representation to obtain a converted feature instruction, and generating a modal mark for a target modal in the target modal list; and updating a generation state through dynamic estimation based on the converted feature instruction.

Description

Technical Field

[0001] The present application relates to a data generation method, and specifically to a multimodal medical data generation method and related devices. Background Art

[0002] With the continuous and in-depth application of artificial intelligence technology in the field of medical health, the use of artificial intelligence methods to assist in the diagnosis, prediction and treatment of diseases has become a hot topic in medical research. For some diseases with highly complex pathogenesis, such as Alzheimer's disease and depression, single-modality data is difficult to fully reveal their pathological characteristics. At the same time, the training of large medical models requires massive multimodal data to provide support to ensure the disease reasoning ability of the model. However, in actual research, due to the limitations of collection costs, social psychology, ethics and other factors, the available multimodal medical data resources are limited, and key modalities are often missing, which has become a bottleneck restricting the pre-training and optimization of large medical models for major neuropsychiatric diseases, limiting the in-depth understanding of the complexity of pathology.

[0003] The existing medical data generation methods have the following main problems: (1) Lack of effective use of the deep connections between multimodal medical data, and limited completion effect; (2) It is difficult to generate high-quality and diverse virtual medical samples, and cannot alleviate the problem of data scarcity; (3) It cannot fully adapt to specific medical fields and lacks targeted model optimization strategies; (4) The degree of systematization is insufficient, and lacks practicality and scalability. Summary of the invention

[0004] The present application aims to solve the technical problems of limited multimodal medical data resources, missing modalities and data scarcity in existing medical data generation methods, and provides a multimodal medical data generation method and related devices.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions: In a first aspect, the present application proposes a method for generating multimodal medical data, comprising: Migrate the general medical macromodel to the target specific medical field to obtain the fine-tuned medical macromodel; Obtain text instructions and multimodal data streams associated with the task, and obtain the task type, existing modal list, target modal list, and task instructions after parsing the text instructions based on the text instructions and multimodal data streams; According to the existing modalities and task instructions in the existing modality list, a comprehensive shared feature representation is obtained; After encoding and decoding the comprehensive shared feature representation, a transformed feature instruction is obtained, and a modality tag is generated for the target modality in the target modality list; Based on the converted feature instructions, the generation state is updated through dynamic estimation.

[0006] Furthermore, the migration of the general medical macromodel to a target specific medical field includes: Freeze some parameters of the general medical big model, add an adaptive transfer layer after each layer of the general medical big model, introduce a trainable multimodal low-rank matrix in the feature association layer, and obtain an adjusted model; Perform end-to-end instruction adjustment for a specific target medical field, update the projection layer of the adjustment model and the pre-trained weights of the general medical large model therein, and obtain an updated model; Prompt words are set according to the migration target to guide the updated model to generate outputs that meet the needs of the target specific medical field, and obtain a fine-tuned medical large model.

[0007] Furthermore, the task type, the existing modality list, the target modality list and the task instruction after the text instruction is parsed are obtained according to the text instruction and the multimodal data stream, including: Obtain the corresponding part-of-speech sequence according to the text instruction; Perform file existence and integrity verification on data in multimodal data streams; Extract task types, modality names, and operation instructions from part-of-speech sequences and multimodal data streams through medical language models; According to the extracted modality names, a list of existing modalities is established; Determine the target modality list according to the target modality in the operation instruction or according to the existing modality combination rule; Organize task types, existing modal lists, and target modal lists to form structured task instructions.

[0008] Furthermore, obtaining the comprehensive shared feature representation and the encoded task instruction representation according to the existing modes and task instructions in the existing mode list includes: Encode the data of the existing modality to obtain the corresponding feature representation; Encode the task instruction to obtain the task instruction features; The feature representations corresponding to each existing modality and the task instruction features are respectively mapped to the shared latent space to obtain the corresponding mapping features of each modality and the task instruction mapping features; Concatenate the mapping features of each modality and the mapping features of the task instruction to obtain a comprehensive feature representation; Using multiple parallel attention heads, the comprehensive feature representation is mapped to the query vector matrix, the key vector matrix and the numerical vector matrix to learn the importance of each modal feature. After linear transformation and activation function, the comprehensive shared feature representation is obtained.

[0009] Furthermore, after encoding and decoding the comprehensive shared feature representation, the converted feature instruction is obtained, and a modality tag is generated for the target modality in the target modality list, including: The comprehensive shared feature representation is encoded to obtain the corresponding high-level feature representation h i ; By fine-tuning the medical big model, high-level feature representation h i Perform semantic understanding and capture the correlation information between different modalities; Decode the high-level feature representation and convert it into feature instructions S i ; According to the characteristic instruction S i The association information between each modality is combined with the target modality to obtain the modality label.

[0010] Furthermore, the update generation status includes: Set initial contribution ; Based on the current build status And each mode mapping feature , dynamically calculate contribution : ; Update build status:

[0011] in, Indicates the generated state obtained after updating at time t+1.

[0012] In a second aspect, the present application proposes a multimodal medical data generation system, comprising: A fine-tuning module is used to migrate the general medical macromodel to a target specific medical field to obtain a fine-tuned medical macromodel; The multimodal intelligent task allocation module is used to obtain the text instructions and multimodal data streams associated with the task, and obtain the task type, the existing modal list, the target modal list and the task instructions after the text instructions are parsed according to the text instructions and the multimodal data stream; A multi-modal dynamic adaptive encoding module, used to obtain a comprehensive shared feature representation based on existing modalities and task instructions in the existing modality list; A medical semantic understanding module is used to encode and decode the comprehensive shared feature representation to obtain the converted feature instructions and generate a modality tag for the target modality in the target modality list; The multimodal data dynamic contribution estimation module is used to update the generation state through dynamic estimation based on the converted feature instructions.

[0013] Furthermore, it also includes a multimodal collaborative mechanism module, which is used to control the interaction between the multimodal intelligent task allocation module, the multimodal dynamic adaptive encoding module, the medical semantic understanding module and the multimodal data dynamic contribution estimation module to maintain multimodal consistency.

[0014] In a third aspect, the present application proposes an electronic device, comprising: a memory and one or more processors; the memory is coupled to the processor; wherein computer program code is stored in the memory, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the steps of the above-mentioned multimodal medical data generation method.

[0015] In a fourth aspect, the present application proposes a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multimodal medical data generation method are implemented.

[0016] Compared with the prior art, this application has the following beneficial effects: This application proposes a method for generating multimodal medical data, which achieves smooth migration of a small number of samples by fine-tuning the adaptability of existing general large models to specific fields, migrating them to specific medical fields at a low cost. By utilizing the deep association of multimodal data, high-quality completion of missing modalities is achieved, and the integrity of multimodal data is improved. It can alleviate the problem of scarce data resources, efficiently generate high-quality and diverse virtual medical samples, and efficiently generate a large number of high-quality virtual samples, with high quality and diversity of generated samples. The systematic multimodal data enhancement scheme provided is highly practical and scalable. It provides rich high-quality multimodal medical data support for the training and reasoning of large medical models, thereby improving the understanding, diagnosis and treatment of major neuropsychiatric diseases.

[0017] The present application also proposes a multimodal medical data generation system, an electronic device and a computer storage medium, which have all the advantages of the above-mentioned multimodal medical data generation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 This is a first flow chart of the multimodal medical data generation method of the present application; Figure 2 A schematic diagram of a multimodal medical data generation system of the present application; Figure 3 This is a schematic diagram of the system corresponding to the multimodal medical data enhancement in the embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0022] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0023] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invented product is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0024] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0025] In the description of the embodiments of the present application, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0026] With the continuous and in-depth application of artificial intelligence technology in the field of medical health, the use of artificial intelligence methods to assist in the diagnosis, prediction and treatment of diseases has become a hot topic in medical research. The pathogenesis of major neuropsychiatric diseases represented by Alzheimer's disease and depression is highly complex, and single-modality data is difficult to fully reveal their pathological characteristics. At the same time, the training of large medical models requires massive multimodal data to provide support to ensure the disease reasoning ability of the model. However, in actual research, due to the limitations of collection costs, social psychology, ethics and other factors, the available multimodal medical data resources are limited, and key modalities are often missing, which has become a bottleneck restricting the pre-training and optimization of large medical models of major neuropsychiatric diseases. Multimodal medical data provides a comprehensive perspective for analyzing the complex mechanisms of neuropsychiatric diseases, but most existing studies are limited to the use of some modalities, which limits the in-depth understanding of the complexity of pathology.

[0027] In response to the need for pathogenesis analysis and precise diagnosis and treatment of major neuropsychiatric diseases such as Alzheimer's disease and depression, a large number of scientific research forces are conducting in-depth explorations in multiple dimensions and levels based on multimodal medical data. These interdisciplinary, multi-center collaborative studies have promoted the pathological analysis of major neuropsychiatric diseases and laid the foundation for precise diagnosis and effective treatment of diseases. In addition, the continuous output of high-quality academic papers, the construction of large-scale clinical databases, and the establishment of large-scale international multi-center collaborative research inventions have promoted the exchange and transformation of cutting-edge theories and technologies in the field of diagnosis and treatment of major neuropsychiatric diseases worldwide, and have had an important and far-reaching impact on clinical diagnosis and treatment models.

[0028] The pathogenesis of major neuropsychiatric diseases such as Alzheimer's disease and depression is highly complex and involves abnormalities at multiple biological levels. A single modality cannot fully reflect its pathological process. Therefore, the use of multimodal medical data is crucial to fully understand the complex pathological process of the disease. At present, using existing database resources, many studies on major neuropsychiatric diseases have been carried out worldwide. In the study of Alzheimer's disease, Haque et al. used the ADNI database to improve the diagnosis and prediction of Alzheimer's disease and related diseases by using targeted detection of changes in 48 key proteins. The results showed that the combination of newly detected proteins with existing cerebrospinal fluid biomarkers can significantly improve the diagnostic performance. Guo et al. used artificial intelligence algorithms to analyze and model 1463 plasma proteomics data, thereby mining key biomarkers that can predict the risk of dementia 15 years in advance, providing the possibility for early intervention and treatment of the disease. HSH et al. conducted targeted plasma proteomics analysis based on data from 5676 participants to study the association between organ aging and neurodegenerative diseases such as Alzheimer's disease, and found molecular information about brain aging. Zhang et al. followed up 344,324 non-demented people using UK Biobank data, and mapped the modifiable risk factors for dementia for the first time using exposure group association analysis technology for 7,214 multidimensional candidate risk factors in the participants. In the study of depression, Alagapan et al. used EEG recording and artificial intelligence methods to identify objective biomarkers from the local field potentials of the subcortical cingulate gyrus after deep brain stimulation to accurately capture the individual recovery status of patients with refractory depression and provide new insights into the variability of depression treatment. Hettwer et al. analyzed the common pathological change patterns of six mental illnesses such as depression based on genomic data and medical imaging, and showed that these diseases may have common pathogenesis and synergy of brain structural changes.

[0029] Although relevant research has made some progress using existing data resources, these neuropsychiatric disease databases still have problems such as scarce data resources, such as a relatively insufficient number of case samples and a small number of paired samples with complete modalities. Taking ADNI as an example, although the database contains various modalities, the number of samples collected in all modalities is limited. For the study of depression, due to the lack of a large-scale database of all modalities, most current studies only focus on neuroimaging data and ignore other modal information, such as genomics and clinical phenotype data. The limitations of these studies using partial modality data limit their overall understanding of the complexity of the disease, and only conduct correlation analysis from the perspective of basic research, which cannot meet clinical needs.

[0030] The Chinese invention patent with publication number CN113239022A proposes a method and device, electronic device, and medium for completing missing data in medical diagnosis. The method includes: obtaining a medical diagnosis data set with missing data problems; randomly dividing the original data into initial sample point data and candidate sample point data, and using the initial sample point data to construct and train a generative adversarial network initial completion model; using an influence function to estimate the influence of the sample point on the parameters of the generative adversarial network initial completion model and on the prediction results of the generative adversarial network initial completion model; using a binary search algorithm to sample the most influential sample point in the candidate sample point data, and further iteratively optimize the generative adversarial network initial completion model to complete the missing data in medical diagnosis.

[0031] The Chinese invention patent application with publication number CN116630777A proposes a method for completing missing data of medical indicators with information bottleneck graph compression. The method is characterized in that all attributes of patients of the same type are modeled as an undirected complete graph, and the graph is continuously pruned and compressed according to diagnostic needs and information bottleneck principles to ensure that the total amount of information used for missing interpolation and diagnostic prediction is streamlined under the condition that the subgraph is consistent with the downstream task objectives. Specifically, the method includes: preprocessing of original data, construction of an undirected complete graph, pruning and compression of the initial complete graph, completing subgraphs and node features, minimizing subgraphs and completing subgraphs, etc., which solves the problem of processing missing data in electronic health records.

[0032] The Chinese invention patent application with publication number CN116795826A proposes a multimodal medical missing data completion method and device based on data association mining, which specifically includes: obtaining original multimodal medical data with missing data problems. According to the acquired original multimodal data, a generator model based on data association mining and a mask energy discriminator model are constructed. The constructed generator model based on data association mining and the mask energy discriminator model are combined, and the combined model is game-optimized and trained using the original multimodal medical data. The missing data is completed through the trained generator model to obtain complete data.

[0033] However, the prior art still has the following problems: (1) Existing medical data generation methods lack effective use of the deep connections between multimodal medical data, and the completion effect is limited. These methods mainly target single-modal data, or do not fully utilize the deep connections and complementary information between multimodal medical data, and cannot effectively use data from other modalities to complete the missing modality, resulting in limited completion effect.

[0034] (2) Existing technologies are difficult to generate high-quality and diverse virtual medical samples and are unable to alleviate the problem of data scarcity. Existing methods rely too much on model structure and do not adequately consider the diversity and richness of data. They are unable to generate high-quality and diverse virtual medical samples and are unable to effectively alleviate the problem of data scarcity in the training process of large medical models.

[0035] (3) Existing technologies are not fully adapted to specific medical fields and lack targeted model optimization strategies. Existing methods are not optimized for specific medical fields (such as major neuropsychiatric diseases) and lack domain-specific semantic alignment and instruction fine-tuning strategies, resulting in insufficient model understanding and processing capabilities for data in specific fields, affecting the quality of completion and generation.

[0036] (4) Existing technologies are not sufficiently systematic and lack practicality and scalability. Most of the above existing technologies are complementary methods for specific scenarios or specific data types. They lack a complete, systematic solution for multimodal medical data enhancement and are difficult to adapt to the needs of different medical fields and multimodal data. They lack practicality and scalability.

[0037] Based on the above situation, the present application proposes a multimodal medical data generation method and related devices, and the present application is described in detail below in conjunction with embodiments and drawings.

[0038] like Figure 1 As shown, it is a first flow chart of the multimodal medical data generation method of the present application, which may include: S101, migrate the general medical big model to the target specific medical field to obtain a fine-tuned medical big model.

[0039] Generic medical big models are usually large neural network models pre-trained on a large amount of medical data, which can capture a wide range of knowledge and features in the medical field. The knowledge of the general model can be adapted to specific medical subfields or tasks through transfer learning techniques, which usually include adjusting model parameters, adding or modifying network layers, and further training with data from specific fields. During the migration process, the model can be fine-tuned, for example, a small amount of additional training is performed on the model to make it better adapt to the new task or dataset while retaining its general features learned on the original large dataset.

[0040] S102, obtaining text instructions and multimodal data streams associated with the task, and obtaining the task type, existing modal list, target modal list, and task instructions after the text instructions are parsed according to the text instructions and the multimodal data stream.

[0041] Text instructions are clear instructions given by the user or the system, describing the type of task to be performed, the goal, etc. For example, "generate a brain tumor segmentation map based on MRI images". Multimodal data streams can include various types of data related to the task, such as imaging materials (CT, MRI), clinical records, physiological signals, etc. These data provide specific information required to complete the task. In addition, the system needs to parse the text instructions, identify the task type (such as segmentation, classification, detection), the list of existing modalities (such as MRI images) and the list of target modalities (such as segmentation maps), and convert the text instructions into specific task instructions.

[0042] S103, obtaining a comprehensive shared feature representation according to the existing modalities and task instructions in the existing modality list.

[0043] For each existing modality, appropriate feature extraction techniques (such as convolutional neural networks for images) can be used to extract key features. By fusing the features of different modalities in some way, a comprehensive shared feature representation can be formed, which contains complementary information of all modalities and provides a rich data foundation for subsequent tasks.

[0044] S104, after encoding and decoding the comprehensive shared feature representation, a converted feature instruction is obtained, and a modality tag is generated for the target modality in the target modality list.

[0045] The comprehensive shared feature representation is further processed through an encoder (such as a deep learning network) to extract higher-level features or representations. The decoder can generate data that matches the target modality based on the encoded feature instructions. For example, if the target modality is a segmentation map, the decoder will output a segmentation mask of the same size as the input image. For the generated target modality data, additional tags or annotations may need to be added to facilitate subsequent analysis or application.

[0046] S105, based on the converted feature instructions, update the generation state through dynamic estimation.

[0047] During the generation process, the system may need to dynamically adjust parameters or strategies based on the current generation status and feedback to ensure the accuracy and quality of the generation results. The current generation status can be updated based on the results of dynamic estimation, including adjusting the generation strategy, correcting errors, optimizing the generation results, etc. This step is iterative until a satisfactory generation effect is achieved.

[0048] This application fully explores the deep connections between multimodal medical data, and through a multimodal collaborative mechanism, achieves high-quality completion of missing modalities, making up for the shortcomings of existing technologies in utilizing complementary information of multimodal data. A high-quality virtual sample generation method based on a large medical model is proposed. The adaptively fine-tuned large medical model, combined with a diffusion decoder, can efficiently generate a large number of high-quality and diverse virtual medical samples, enrich the training data, and improve the generalization ability of the model. A strategy for adaptive fine-tuning of the large medical model in specific fields is also proposed. By freezing some parameters and updating the projection matrix, semantic alignment in specific fields is achieved; then, some parameters are unfrozen and end-to-end instruction fine-tuning is performed, so that the model can more effectively understand and process multimodal data in the field of major neuropsychiatric diseases, and improve the quality and efficiency of missing modality completion and virtual sample generation.

[0049] like Figure 2 FIG. 1 is a schematic diagram of a multimodal medical data generation system of the present application, which may include: A fine-tuning module is used to migrate the general medical macromodel to a target specific medical field to obtain a fine-tuned medical macromodel; The multimodal intelligent task allocation module is used to obtain the text instructions and multimodal data streams associated with the task, and obtain the task type, the existing modal list, the target modal list and the task instructions after the text instructions are parsed according to the text instructions and the multimodal data stream; A multi-modal dynamic adaptive encoding module, used to obtain a comprehensive shared feature representation based on existing modalities and task instructions in the existing modality list; A medical semantic understanding module is used to encode and decode the comprehensive shared feature representation to obtain the converted feature instructions and generate a modality tag for the target modality in the target modality list; The multimodal data dynamic contribution estimation module is used to update the generation state through dynamic estimation based on the converted feature instructions.

[0050] The present application is described in detail through the following examples, taking the field of neuropsychiatric diseases as the target specific medical field.

[0051] S201, Domain-specific adaptive fine-tuning of large medical models.

[0052] The goal is to migrate the existing general medical big model to a specific medical field (such as the field of neuropsychiatric diseases) at a low cost, and achieve efficient migration of very few samples. The input of this step is the pre-trained parameters of the general medical big model and a small amount of multimodal medical data in a specific field. The output is the fine-tuned parameters of the specific field medical big model and the projection moment parameters of the adaptive migration layer. Specifically, the following steps can be used to achieve this: (1) Selection of pre-trained general medical large models.

[0053] Choose general medical big models that have been pre-trained on large-scale medical data. These models have been pre-trained on large-scale medical text and image datasets and have strong cross-modal understanding and generation capabilities. For example, the Transformer architecture model has performed well in both natural language processing and computer vision, and is a choice for general medical big models. It supports multi-task learning, can process text and image data simultaneously, and achieve cross-modal understanding and generation, such as Vision Transformer. You can also use BioBERT based on the BERT architecture.

[0054] (2) Collection and preprocessing of target-specific medical field data.

[0055] Multimodal medical data in the field of major neuropsychiatric diseases can be collected from public medical databases, scientific research institutions or cooperative hospitals, including behavioral gait data, MRI (Magnetic Resonance Imaging), PET (Positron Emission Tomography), genomic data, Electronic Medical Record (EMR), Electroencephalogram (EEG), etc. Try to use a small amount of high-quality data to meet the needs of few-sample fine-tuning.

[0056] When performing data preprocessing, for image data, you can use medical image processing tools such as SimpleITK and NiftyReg for denoising, normalization, registration, and cutting; for text data, you can use natural language processing libraries such as NLTK and SpaCy for word segmentation, stop word removal, and named entity recognition; for signal data, you can use signal processing libraries such as SciPy and MNE for filtering, denoising, and feature extraction.

[0057] (3) Adaptive fine-tuning for domain transfer.

[0058] Freeze most of the parameters of the pre-trained model, only add an adaptive transfer layer after each layer, introduce a trainable multimodal low-rank matrix at the key layer, and only allow the adaptive transfer layer and the multimodal low-rank matrix to update parameters. In this way, a large number of multimodal features in the field of neuropsychiatric diseases are embedded in the large medical model and are consistent with the multimodal embedding in pre-training. This stage can extend the aligned multimodal feature embedding to the field of neuropsychiatric diseases. Use a small amount of domain-specific data to train these newly added lightweight modules, significantly reducing computational costs and overfitting risks.

[0059] It should be noted that freezing most of the parameters of the pre-trained model usually refers to the parameters such as weights and biases that have been trained in the pre-trained model. These parameters are optimized through a large amount of data in the pre-training stage, so that the model can capture the key features and patterns in the data. In transfer learning, it is hoped to use the knowledge that has been learned, so these parameters will be frozen (i.e., not updated) to avoid destroying the feature representations that they have learned when training on a small amount of data in a specific field. The key layer refers to the layer in the model that is particularly important for a specific task or field. These layers may be responsible for extracting key features, performing feature fusion, or performing other operations that are critical to the performance of the model. The key layer is selected to introduce a trainable multimodal low-rank matrix, and multimodal features of specific fields (such as the field of neuropsychiatric diseases) can be integrated into these layers. By modifying only the key layers, the performance of the model can be more accurately affected without having to make large-scale adjustments to the entire model. While keeping most of the structure of the model unchanged, the adaptation of specific fields is achieved by introducing a small number of trainable parameters, which helps to maintain the stability and generalization ability of the model. In addition, introducing multimodal low-rank matrices at the key layer can promote the fusion and interaction between different modalities (such as text, images, genomics data, etc.), thereby extracting richer feature representations.

[0060] (4) Through instruction alignment and adjustment, a small number of samples can be transferred to obtain a fine-tuned medical large model.

[0061] Perform end-to-end instruction adjustments for neuropsychiatric diseases and update the pre-trained weights of the projection layer and the general medical big model. Effective prompt words can be designed to guide the model to generate outputs that meet the needs of specific fields, and use the strong generalization ability of the big model to reason on specific field tasks. The fine-tuned medical big model can achieve a better understanding of neuropsychiatric diseases.

[0062] In order to better illustrate the subsequent steps, the subsequent steps will be executed in multiple modules according to the order of steps for clearer explanation.

[0063] Through fine-tuning, a general medical large model can be migrated to a specific medical field under few-sample conditions, achieving low-cost and high-efficiency model adaptation.

[0064] S202, multimodal medical data enhancement based on fine-tuning of the medical big model. Figure 3 As shown, this is a system schematic diagram corresponding to multimodal medical data enhancement.

[0065] 1. Multimodal intelligent task allocation module.

[0066] The purpose of multimodal intelligent task allocation is to parse the input text instructions and data streams, identify the task type and data modality, and guide the subsequent missing modality completion or virtual sample generation process. The input of multimodal intelligent task allocation is text instructions C (natural language description), the provided multimodal data stream X, and the output is the task type T (missing modality completion or virtual sample generation), the existing modality list M existing , target modal list M target , the parsed task instruction I (structured representation). This can be achieved by the following steps: (1) Data reception and preprocessing.

[0067] Convert the input text instruction C into a unified encoding format, remove special characters, blank characters and noise characters, and obtain a clean text string C clean , for the text string C clean Perform word segmentation to obtain the word sequence {w_1, w_2, ..., w_n}. Then perform part-of-speech tagging on the words in the word sequence to obtain the part-of-speech sequence {p_1, p_2, ..., p_n}. Check the data format to ensure data integrity. Traverse the multimodal data stream X and for each data in the multimodal data stream X Perform file existence and integrity checks, preliminarily determine the file type based on the file extension, and ensure that the data format meets expectations.

[0068] (2) Text instruction parsing and data modality recognition Use pre-trained medical language models for named entity recognition, extract task types, modality names, and operation instructions. Build professional dictionaries in the medical field to improve entity recognition accuracy. Use pre-trained medical language models to identify word sequences w i Encode and get the word vector sequence e i , the word vector sequence e i Perform sequence labeling and output the entity category label t of each word i . According to the entity category label, extract information such as task type T, target modality, and specific requirements. File metadata parsing and content analysis, read file header information and parse file extension information. Use the convolutional neural network model to classify the data modally. The data passes through the input layer, the convolution layer extracts features, the pooling layer reduces the dimension, and the fully connected layer maps the features to the modality category space. The output layer uses the Softmax function to output the probability of each modality category.

[0069] (3) Determine the task type and structure the task instructions First, perform logical judgment to determine the task type. According to the extracted task type entity, set the task type variable T, and establish the existing modal list M according to the data modal recognition result.existing , according to the target modality parsed by the task instruction, or according to the common modality combination rules, determine the target modality list M that needs to be completed or generated target .

[0070] Then construct a structured task instruction in JSON format, which contains the task type T, the existing modal list M existing 、Target modal list M target , specific requirements and other information to form structured task instructions I.

[0071] The intelligent task allocation module is used for task identification and data modality analysis. It can automatically parse the user's text instructions and the data modality in the data stream, accurately identify the task type, and guide the subsequent missing modality completion or virtual sample generation process, thereby improving the intelligence of the system.

[0072] 2. Multimodal dynamic adaptive coding module The goal of the multimodal dynamic adaptive encoding module is to encode data and text instructions of any modality into a unified shared feature space, which is convenient for subsequent fine-tuning of the medical large model for semantic understanding. The input of this module is the existing modality data and structured task instructions I, and the output is the shared feature representation Z and the encoded task instruction representation z cmd This can be achieved by the following methods: (1) Construct a multimodal encoder and encode data For each modality m∈M, input the data x into the corresponding encoder consisting of input layer, embedding layer, encoding layer, pooling layer, and fully connected layer to obtain feature representation i . Encode the task instruction I and obtain the task instruction feature e cmd .

[0073] (2) Mapping feature representations to a shared space and fusing them Use the projection matrix W learned during fine-tuning m , mapping each modality feature to a shared latent space:

[0074] in, is the initial feature representation of each mode, is the bias vector of the modal feature map, for the task instruction feature Also map, use the projection matrix to get :

[0075] in, is the bias vector of the task instruction feature map.

[0076] Concatenate the features of each modality to form a comprehensive feature representation z i :

[0077] Use a multi-head attention layer to represent the comprehensive features of the input. Use H parallel attention heads, each with its own projection matrix, to map the input to the query vector Q, key vector K, and value vector V matrix, learn the importance of each modal feature, and obtain the final comprehensive shared feature representation after linear transformation and activation function. For subsequent processing.

[0078] The multimodal dynamic adaptive encoding module realizes the unified feature representation of multimodal data and can encode data and text instructions of any modality into a unified shared feature space, which facilitates subsequent semantic understanding and processing and realizes the effective fusion of multimodal data.

[0079] 3. Medical semantic understanding module The medical semantic understanding module is used to perform deep semantic understanding of the comprehensive shared feature representation with the help of fine-tuning the medical big model, and generate feature instructions and modality labels for missing modality completion and virtual sample generation. The input of the medical semantic understanding module is the comprehensive shared feature representation and the target modal list M target , the output is the characteristic instruction S i and the modal marker M' target .

[0080] (1) Deep semantic understanding The comprehensive shared features are represented Input the medical large model encoder to obtain the high-level feature representation h i ,Use fine-tuned medical big models for semantic understanding and capture the correlation information between modalities.

[0081] (2) Feature instructions and modal mark generation Using a multimodal feature instruction decoder, including self-attention, encoder-decoder attention, and feedforward neural network, h i Converted into feature instruction S i A sequence-to-sequence model is used, and real instructions are used as targets during training. A modality instruction conversion layer is added to output modality tags according to the modality to be generated.

[0082] 4. Multimodal Data Dynamic Contribution Estimation Module The multimodal data dynamic contribution estimation module improves the quality and consistency of generation by dynamically estimating and adjusting the impact of each modality on the generated results. The input of this module is the comprehensive shared feature representation , characteristic instruction Si and the current build status , the output is the dynamic contribution coefficient and updated build status This can be achieved by following the steps below: (1) Initial contribution setting

[0083] According to the quality and relevance of each modal data, set the initial contribution .

[0084] (2) Dynamic adjustment of contribution.

[0085] The dynamic contribution calculation architecture consists of a fusion layer, a hidden layer, and an output layer. And each modal feature , dynamically calculate contribution: .

[0086] (3) Contribution is applied to the generation process According to the dynamic contribution coefficient, each modal feature is weighted and the weighted modal features are input into the generation model:

[0087] in, It is a generative model that outputs dynamic contribution and updates the generation status.

[0088] The multimodal data dynamic contribution estimation module enhances the quality and controllability of the generated results by adjusting the contribution of each mode in real time during the generation process.

[0089] 5. Multimodal Medical Sample Diversity Enhancement Module The multimodal medical sample diversity enhancement module uses the fine-tuned medical model to generate diverse high-quality samples and increase the generalization ability of the fine-tuned medical model. The input of this module is the feature instruction S i , dynamic contribution and random noise n, the output is a diverse generated sample x generated , which can be achieved by the following methods: (1) Diversity generation strategy Random noise n is added in the generation process to increase diversity and adjust the temperature coefficient τ of the generation model to control the randomness of the generation results.

[0090] (2) Diversity Assessment and Feedback Design evaluation indicators (diversity index) to evaluate the diversity of generated samples, calculate the entropy of the generated sample set, and evaluate the mutual information between the generated samples and the real data. According to the evaluation results, dynamically adjust the generation parameters and optimize the generation strategy.

[0091] The multimodal sample diversity enhancement module is used to generate diverse and high-quality samples. It uses large medical models and generative models to generate diverse and high-quality samples, enriching the training data and improving the generalization ability of the model.

[0092] 6. Multimodal medical sample quality control module The multimodal medical sample quality control module is used to evaluate the quality of the generated samples, remove samples of poor quality or not meeting the requirements, and ensure the reliability of the generated data. The input of this module is the generated sample x generated , the output is a sample set X with qualified quality qualified and the eliminated sample set X discarded This can be achieved by the following methods: (1) Sample quality assessment Use a pre-trained sample quality assessment model to automatically detect the quality of the generated samples, and use a pre-trained sample quality assessment model to automatically score the samples. Build a reasoning network based on the knowledge graph, which uses known medical rules and associations (the relationship between diseases and symptoms) to evaluate the medical rationality of the generated samples, reason about the entities and relationships in the knowledge graph, and thus evaluate whether the samples conform to medical logic.

[0093] (2) Pruning of unqualified samples Combine the outputs of multiple networks, use a multi-layer perceptron to further process the scores from different sources, combine the results of automatic quality detection and medical rationality verification, output the final quality and rationality scores, input sample features, output quality and rationality scores, set quality thresholds, and eliminate samples with scores below the threshold.

[0094] The multimodal sample quality control module is used to evaluate and eliminate unqualified samples, perform quality assessment on the generated samples, use the quality evaluation model to automatically determine the eligibility of the samples, and eliminate samples of poor quality or that do not meet the requirements, thus ensuring the reliability of the generated data.

[0095] 7. Multimodal collaboration mechanism module The multimodal collaboration mechanism module is used to control the interaction between modules and ensure the multimodal consistency and high quality of the generated data. The following methods can be used: (1) Inter-module interaction control A multimodal collaboration mechanism is used to achieve data communication and state synchronization between modules. A message queue system based on Kafka (a distributed message queue system) is used for data communication and state synchronization between modules. Each module sends the operation status and data changes in the form of messages to ensure data consistency and real-time updates.

[0096] (2) Multimodal consistency maintenance A multimodal medical data consistency loss function is designed for feedback, which is specifically used to evaluate and optimize the consistency between different modalities. The generation model is optimized through back-propagation to ensure the consistency of the generation results of different modalities.

[0097] (3) Conflict Detection The input is passed through a deep neural network consisting of convolutional layers, transition layers, and fully connected layers to extract topological features of the input, identify potential conflicts between different modal data, and detect conflicts between different modalities.

[0098] The multimodal collaborative mechanism module ensures the consistency and high quality of the generated data. Through interactive control and consistency maintenance between modules, it ensures that the generated data of different modalities are consistent in semantics and features, thereby improving the overall quality of the generated results.

[0099] 8. Resource extension management module The resource expansion management module is used to dynamically adjust the number of instances of the sample diversity enhancement module and the sample quality control module. The input of this module is the task queue length and system resource usage, and the output is the number of module instances and resource allocation plan. The specific implementation method can be: (1) Dynamic expansion of module instances Use containerization technology and orchestration tools to automatically expand or reduce module instances based on system load. Use containers to encapsulate modules, and use Kubernetes (an open source container orchestration and management platform) for cluster management and dynamic expansion. Automatically adjust the number of container instances based on real-time monitoring of system load and task queue length to achieve horizontal expansion or reduction of modules.

[0100] (2) Computing resource allocation Dynamic resource allocation strategy based on priority and resource requirements uses scheduling algorithms to optimize resource allocation, ensure that key tasks are given priority, and improve overall system efficiency and response speed. Implement resource scheduling strategy to dynamically allocate computing resources based on task priority and resource requirements.

[0101] (3) Scaling strategy adjustment Regularly monitor, collect and analyze system performance, adjust expansion strategies based on analysis results, increase or decrease resource quotas of certain modules to adapt to changing workloads and ensure stable and efficient system operation.

[0102] The resource expansion management module realizes the elastic expansion and resource optimization of the system. It can dynamically adjust the number of module instances according to task requirements and system load, reasonably allocate computing resources, and optimize system performance and resource utilization.

[0103] 9. Expert Interaction Supervision Module The expert interactive supervision module is used to provide a supervision interface for experts to manually adjust the quality and direction of generation to ensure the reliability of the generated results. The input of this module is the final generated result X final and expert feedback F, the output is the final result X reviewed by experts expert And updated model parameters. The following methods can be used: (1) Provide human-computer interaction interface Design an expert human-computer interaction interface and provide a visual interface to provide experts with an interface for browsing, annotating, and giving feedback. Allow experts to browse, annotate, and give feedback on the quality of generated samples through a graphical interface, and ensure cross-platform accessibility and a good user experience.

[0104] (2) Manual adjustment by experts Experts can view the detailed information of each sample directly on the interface and make modification suggestions or directly annotate each sample. These feedbacks will be recorded and used in the subsequent sample quality improvement process.

[0105] (3) Feedback fusion and model updating The expert feedback is encoded and used for model retraining. A continuous learning mechanism is used to update and optimize the generated model. The data reviewed by experts is regularly integrated into the training set to continuously improve the model performance and accuracy.

[0106] The expert interactive supervision module provides an expert adjustment and feedback mechanism. A human-computer interactive supervision module is designed to provide an interactive interface for medical experts, allowing experts to manually supervise and adjust the generated results, integrate feedback into the system, and continuously improve the generation effect.

[0107] This application designs multiple functional modules, including a multimodal intelligent task allocation module, a multimodal dynamic adaptive encoding module, a medical semantic understanding module, a multimodal dynamic contribution estimation module, a multimodal medical sample diversity enhancement module, a multimodal medical sample quality control module, a multimodal collaborative mechanism module, a resource extension management module, and an expert interactive supervision module. The modules work together to achieve the completion of missing modal data and the generation of high-quality virtual samples. The high-quality completion of missing modalities is achieved, and the deep association of multimodal data is fully utilized to improve the completion effect. A multimodal medical data enhancement system with a reasonable structure and complete functions is constructed to solve the modality missing and data scarcity problems of multimodal medical data, and provide strong support for the training and application of large medical models. Among them, high-quality and diverse virtual medical samples are generated to alleviate the problem of data scarcity and improve the training effect of large medical models. An adaptive fine-tuning module for specific medical fields is proposed to enable the model to better understand and process multimodal data in specific fields and improve the quality of completion and generation. A multimodal medical data enhancement system is proposed, which is practical and scalable to meet the needs of different medical fields.

[0108] The present application proposes adaptive fine-tuning for specific medical fields so that the medical large model can quickly adapt to specific fields and improve the applicability of the model. The adaptive fine-tuning module for specific fields of the medical large model proposed in the present application enables the model to better understand and process multimodal medical data in specific fields, and improves the quality and efficiency of missing modality completion and virtual sample generation. Compared with the general model that lacks specific field optimization in the prior art, the model of the present application has higher applicability and effectiveness in specific fields. Furthermore, the present application can use the existing multimodal data to generate high-quality missing modality data, make up for the problem of modality missing, and provide complete data support for multimodal data fusion and model training. Make full use of the deep association and complementary information between multimodal data to improve the accuracy and credibility of the completed data. Compared with the methods in the prior art that mainly target single modality or do not fully utilize multimodal associations, the present application can more accurately reconstruct missing modality data and make up for data gaps. Rich virtual samples provide sufficient data support for model training and optimization, enhance the generalization ability and prediction performance of the model, and meet the needs of medical large models for massive data compared to the methods that are difficult to generate high-quality and diverse virtual samples in the prior art. In addition, the present application proposes a complete multimodal medical data enhancement method and system. Compared with the methods in the prior art for specific scenarios or data types, the system of the present application can adapt to the needs of different medical fields, support data enhancement of multiple modalities, have strong practicality and generalizability, and can be applied to multimodal data enhancement in other medical fields. Through the dynamic weight estimation of multimodal data and the multimodal collaborative generation mechanism, the multimodal consistency of the generated data can be guaranteed, which solves the problem that the generation results of different modalities may be inconsistent in the prior art and improves the reliability of the data.

[0109] It should be noted that the method of this application can also be extended to the same problems in other medical backgrounds, such as oncology, cardiovascular disease and other scenarios that require multimodal data enhancement, and has wide adaptability. The difference is that it only needs to replace the corresponding multimodal medical data as training samples during the training phase to use this system for learning.

[0110] It should be noted that in the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of each module is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another device, or some features can be ignored or not executed. The module described as a separate component may or may not be physically separated. The component displayed as a module may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed in multiple different places. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0111] In addition, each module in each embodiment of the present invention may be integrated into a processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0112] An embodiment of the present application also provides an electronic device, which may include one or more processors, a memory, and a communication interface.

[0113] The memory, the communication interface and the processor are coupled, for example, the memory, the communication interface and the processor may be coupled together via a bus.

[0114] The communication interface is used for data transmission with other devices. The memory stores computer program code. The computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the steps of the multimodal medical data generation method.

[0115] Wherein, the processor can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the present disclosure. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, and the like. The processor can be used to support electronic devices to execute the method steps provided in the above embodiments.

[0116] The bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The above bus may be divided into an address bus, a data bus, a control bus, etc.

[0117] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multimodal medical data generation method are implemented.

[0118] The computer-readable storage medium involved in the present application includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the technical field.

[0119] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating multimodal medical data, characterized in that: include: Migrate the general medical macromodel to the target specific medical field to obtain the fine-tuned medical macromodel; Obtain text instructions and multimodal data streams associated with the task, and obtain the task type, existing modal list, target modal list, and task instructions after parsing the text instructions based on the text instructions and multimodal data streams; According to the existing modalities and task instructions in the existing modality list, a comprehensive shared feature representation is obtained; After encoding and decoding the comprehensive shared feature representation, a transformed feature instruction is obtained, and a modality tag is generated for the target modality in the target modality list; Based on the converted feature instructions, the generation state is updated through dynamic estimation.

2. The method for generating multimodal medical data according to claim 1, characterized in that: The migration of the general medical macro model to a target specific medical field includes: Freeze some parameters of the general medical big model, add an adaptive transfer layer after each layer of the general medical big model, introduce a trainable multimodal low-rank matrix in the feature association layer, and obtain an adjusted model; Perform end-to-end instruction adjustment for a specific target medical field, update the projection layer of the adjustment model and the pre-trained weights of the general medical large model therein, and obtain an updated model; Prompt words are set according to the migration target to guide the updated model to generate outputs that meet the needs of the target specific medical field, and obtain a fine-tuned medical large model.

3. The method for generating multimodal medical data according to claim 1, characterized in that: The step of obtaining the task type, the existing modal list, the target modal list and the task instruction after the text instruction is parsed according to the text instruction and the multimodal data stream includes: Obtain the corresponding part-of-speech sequence according to the text instruction; Perform file existence and integrity verification on data in multimodal data streams; Extract task types, modality names, and operation instructions from part-of-speech sequences and multimodal data streams through medical language models; According to the extracted modality names, a list of existing modalities is established; Determine the target modality list according to the target modality in the operation instruction or according to the existing modality combination rule; Organize task types, existing modal lists, and target modal lists to form structured task instructions.

4. The method for generating multimodal medical data according to claim 1, characterized in that: The step of obtaining a comprehensive shared feature representation and an encoded task instruction representation according to the existing modes and task instructions in the existing mode list includes: Encode the data of the existing modality to obtain the corresponding feature representation; Encode the task instruction to obtain the task instruction features; The feature representations corresponding to each existing modality and the task instruction features are respectively mapped to the shared latent space to obtain the corresponding mapping features of each modality and the task instruction mapping features; Concatenate the mapping features of each modality and the mapping features of the task instruction to obtain a comprehensive feature representation; Using multiple parallel attention heads, the comprehensive feature representation is mapped to the query vector matrix, the key vector matrix and the numerical vector matrix to learn the importance of each modal feature. After linear transformation and activation function, the comprehensive shared feature representation is obtained.

5. The multimodal medical data generation method according to claim 1, characterized in that: After encoding and decoding the comprehensive shared feature representation, the converted feature instructions are obtained, and the modality tags are generated for the target modalities in the target modality list, including: The comprehensive shared feature representation is encoded to obtain the corresponding high-level feature representation h i ; By fine-tuning the medical big model, high-level feature representation h i Perform semantic understanding and capture the correlation information between different modalities; Decode the high-level feature representation and convert it into feature instructions S i ; According to the characteristic instruction S i The association information between each modality is combined with the target modality to obtain the modality label.

6. The multimodal medical data generation method according to claim 1, characterized in that: The update generation status includes: Set initial contribution ; Based on the current build status And each mode mapping feature , dynamically calculate contribution : ； Update build status: in, Indicates the generated state obtained after updating at time t+1.

7. A multimodal medical data generation system, characterized in that: include: A fine-tuning module is used to migrate the general medical macromodel to a target specific medical field to obtain a fine-tuned medical macromodel; The multimodal intelligent task allocation module is used to obtain the text instructions and multimodal data streams associated with the task, and obtain the task type, the existing modal list, the target modal list and the task instructions after the text instructions are parsed according to the text instructions and the multimodal data stream; A multi-modal dynamic adaptive encoding module, used to obtain a comprehensive shared feature representation based on existing modalities and task instructions in the existing modality list; A medical semantic understanding module is used to encode and decode the comprehensive shared feature representation to obtain the converted feature instructions and generate a modality tag for the target modality in the target modality list; The multimodal data dynamic contribution estimation module is used to update the generation state through dynamic estimation based on the converted feature instructions.

8. The multimodal medical data generation system according to claim 7, characterized in that: It also includes a multimodal collaborative mechanism module, which is used to control the interaction between the multimodal intelligent task allocation module, the multimodal dynamic adaptive encoding module, the medical semantic understanding module and the multimodal data dynamic contribution estimation module to maintain multimodal consistency.

9. An electronic device, characterized in that: include: A memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the steps of the multimodal medical data generation method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal medical data generating method according to any one of claims 1 to 6 are implemented.

Images