A training method and device of a multi-modal medical pre-training model

Abstract

Embodiments of the present application relate to a kind of multi-modal medical pre-training model training method and device, the method includes: based on text / image pre-training model, neural structure searcher and neural network module set constructs multi-modal medical pre-training model;Text / image pre-training model is configured with a stage text / visual task model set, and text / image pre-training model is trained in a stage based on a stage text / visual task model set;And the model parameter of text / image pre-training model is frozen when a stage training ends;Multi-modal medical pre-training model is configured with one or more downstream task models, and the neural structure searcher and neural network module set of multi-modal medical pre-training model are trained in two stages based on all downstream task models;The model parameter of multi-modal medical pre-training model is frozen after two-stage training ends. Through the present application, the deficiency of traditional single mode analysis framework can be solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a training method and apparatus for a multimodal medical pre-training model. Background Technology

[0002] In the field of medical artificial intelligence (AI), traditional single-modal analysis frameworks are gradually revealing some shortcomings: 1) Insufficient ability to fuse multimodal features of medical text, 2D medical waveforms (such as electrocardiograms, electroencephalograms, electromyograms, electrooculograms, blood pressure waveforms, respiratory waveforms, blood oxygen saturation waveforms, etc.) and 3D medical images (such as computed tomography images, magnetic resonance imaging images, 3D ultrasound images, positron emission tomography images, single-photon emission computed tomography images, optical coherence tomography images, cone-beam computed tomography images, etc.); 2) Poor adaptability of conventional text or image pre-trained models to medical scenarios; 3) Traditional dedicated medical AI models are usually only designed for a single task and cannot build a unified framework covering the entire process of disease prediction, image analysis, report generation, etc., and there is no universal coding sharing mechanism among them. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a training method, apparatus, electronic device, and computer-readable storage medium for a multimodal medical pre-training model. This invention selects a type of pre-trained large language model as the text pre-training model and a type of visual pre-training model capable of simultaneously supporting 2D / 3D image processing as the image pre-training model. It also selects multiple types of basic neural network modules to form a neural network module set and configures a corresponding neural architecture searcher for each neural network module set. Based on the text pre-training model, image pre-training model, neural architecture searcher, and neural network module set, a multimodal medical pre-training model is constructed. A corresponding first-stage text / visual task model set is configured for each text / image pre-training model. The text / image pre-training models are then trained separately using the first-stage text / image task model set. After the first-stage training is completed, multiple downstream task models are configured to form a multi-task training framework for the multimodal medical pre-training model. The second-stage training of the neural architecture searcher and neural network module set is completed by training this multi-task training framework. This invention addresses several shortcomings of traditional single-modal analysis frameworks: 1) It provides a multimodal medical pre-training model capable of fusing multimodal features from medical text, 2D medical waveforms, and 3D medical images; 2) The two-stage training method based on this invention not only reduces overall training difficulty but also improves the model's adaptability to medical scenarios; 3) Based on the multimodal medical pre-training model of this invention, it provides an encoding sharing platform capable of simultaneously addressing multiple downstream tasks, which not only improves the convenience of multi-task fusion but also enhances the overall task prediction accuracy.

[0004] To achieve the above objectives, a first aspect of the present invention provides a training method for a multimodal medical pre-trained model, the method comprising:

[0005] A pre-trained large language model is selected as the text pre-training model, a visual pre-training model that can simultaneously support 2D / 3D image processing is selected as the image pre-training model, and multiple basic neural network modules are selected to form a neural network module set. A corresponding neural architecture searcher is configured for the neural network module set. Based on the text pre-training model, the image pre-training model, the neural architecture searcher, and the neural network module set, a multimodal large model is constructed as the corresponding multimodal medical pre-training model.

[0006] Configure a one-stage text task model set for the text pre-trained model; and configure a one-stage visual task model set for the image pre-trained model; the one-stage text task model set includes a masked language modeling task head, a masked knowledge element prediction task head, an unstructured text entity recognition task head, a structured text entity recognition task head, and a text generation task head; the one-stage visual task model set includes a waveform classification task head, a waveform disease prediction task head, an image object detection task head, and an image semantic segmentation task head;

[0007] The text pre-trained model is trained for one stage based on the one-stage text task model set; and the image pre-trained model is trained for one stage based on the one-stage visual task model set; and the model parameters of the text pre-trained model and the image pre-trained model are frozen at the end of the one-stage text and image model training.

[0008] Configure one or more downstream task models for the multimodal medical pre-training model; and perform two-stage multimodal training on the neural structure searcher and the neural network module set of the multimodal medical pre-training model based on all the downstream task models;

[0009] After the two-stage training is completed, the model parameters of the multimodal medical pre-trained model are frozen and the model training is confirmed to be finished.

[0010] Preferably, the pre-trained large language model is a type of large language model based on the Transformer architecture that has been pre-trained, and the pre-trained large language model includes at least the BERT series models;

[0011] The visual pre-trained model is a large visual model compatible with both 2D and 3D modalities and has been pre-trained. The visual pre-trained model includes at least the MultiResViT model, UniFormer model, Hybrid CNN-Transformer model, nnU-Net model, Uni3D model, Uni-MAE model, and MedFMC model.

[0012] The basic neural network module includes at least a feature splicing module, a weighted feature splicing module, a residual network module, a cross-attention encoding module, a graph neural network module, a 1D convolutional neural network module, a 2D convolutional neural network module, a 3D convolutional neural network module, a residual connection module, and a Hadamard product module.

[0013] Preferably, the multimodal medical pre-trained model is used to perform cross-modal feature encoding processing based on the model input text and the model input image, and output the corresponding multimodal feature tensor;

[0014] The input text for the model is natural language text or structured text in the medical field;

[0015] The model input image is a 2D waveform or 3D image in the medical field; the 2D waveform image type includes at least electrocardiogram, electroencephalogram, electromyography, electrooculogram, blood pressure waveform, respiratory waveform, and blood oxygen saturation waveform; the 3D image image type includes at least computed tomography (CT) scan, magnetic resonance imaging (MRI), three-dimensional ultrasound imaging, positron emission tomography (PET), single-photon emission computed tomography (SPECT), optical coherence tomography (OCT), and cone-beam computed tomography (CBCT).

[0016] The first model input terminal of the multimodal medical pre-trained model is used to receive the model input text, the second model input terminal is used to receive the model input image, and the model output terminal is used to output the corresponding multimodal feature tensor.

[0017] The multimodal medical pre-training model includes the text pre-training model, the image pre-training model, the neural structure searcher, and the neural network module set; the neural network module set includes multiple basic neural network modules.

[0018] The input of the text pre-trained model is connected to the input of the first model, and the output is connected to the first input of the neural architecture searcher; the input of the image pre-trained model is connected to the input of the second model, and the output is connected to the second input of the neural architecture searcher; the output of the neural architecture searcher is connected to the output of the model, and the neural architecture searcher is also connected to the neural network module set.

[0019] The text pre-training model is used to encode the text input to the model to obtain the corresponding text feature tensor, which is then sent to the neural structure searcher.

[0020] The image pre-training model is used to perform 2D / 3D adaptive feature encoding on the input image of the model to obtain the corresponding image feature tensor and send it to the neural structure searcher;

[0021] The neural architecture searcher is used to construct an initial network structure according to the neural network module set according to a preset initial network construction rule before the second-stage training; and to assign a learnable training weight parameter to all candidate modules in each i-th layer candidate module group in the initial network structure; the initial network structure consists of N i-th input layers, N i-th layer candidate module groups, and an output layer, 1 ≤ index i ≤ N, where N is a preset positive integer; each i-th layer candidate module group consists of one or more candidate modules, and each candidate module is a basic neural network module; the sum of all training weight parameters corresponding to each i-th layer candidate module group is 1; the input and output data formats of all candidate modules in each i-th layer candidate module group are consistent;

[0022] In the initial network structure, the input of the first input layer is connected to the input of the neural structure searcher, and its output is connected to the input of each candidate module in the first layer candidate module group. The output of each candidate module in the first layer candidate module group is connected to one input of the second input layer; and so on. The output of each candidate module in the (N-1)th layer candidate module group is connected to one input of the Nth input layer, the output of the Nth input layer is connected to the input of each candidate module in the Nth layer candidate module group, and the output of each candidate module in the Nth layer candidate module group is connected to one input of the output layer; the output of the output layer is connected to the output of the neural structure searcher.

[0023] In the initial network structure, the first input layer is used to receive the text feature tensor and the image feature tensor to form a corresponding first tensor group, and send the first tensor group to each of the candidate modules of the first layer candidate module; each i-th input layer from the second input layer to the Nth input layer is used to perform a weighted summation calculation on the output tensors of all the candidate modules of the (i-1)th layer candidate module group according to the training weight parameters corresponding to all the candidate modules of the corresponding (i-1)th layer candidate module group, and use the calculation result as the corresponding (i-1)th layer output tensor, and send the (i-1)th layer output tensor to each of the candidate modules of the corresponding i-th layer candidate module group; the output layer is used to perform a weighted summation calculation on the output tensors of all the candidate modules of the Nth layer candidate module group according to the training weight parameters corresponding to all the candidate modules of the Nth layer candidate module group to obtain the corresponding Nth layer output tensor, and use the Nth layer output tensor as the corresponding multimodal feature tensor and output it;

[0024] In the initial network structure, the training weight parameters of all candidate modules are updated through training during the two-stage training process;

[0025] The neural architecture searcher is also used to prune the initial network structure at the end of the two-stage training to obtain the corresponding preferred network structure and solidify it.

[0026] The neural architecture searcher is also used to perform feature fusion processing on the input text feature tensor and the image feature tensor based only on the preferred network structure after the preferred network structure is frozen, and output the corresponding multimodal feature tensor.

[0027] Preferably, the task head input of the masked language modeling task head includes medical masked corpus text and the text feature tensor corresponding to the current corpus text, and the task head output is a masked word segmentation sequence; the medical masked corpus text is a sentence or a paragraph of natural language text in the medical field; one or more word segmented texts in the medical masked corpus text are masked by preset mask markers, and the word segmentation index of each mask marker in the medical masked corpus text is recorded as the corresponding masked word segmentation index; the masked word segmentation sequence is formed by sorting one or more masked prediction texts, and the masked prediction text corresponds one-to-one with the mask marker and is the word segmentation prediction text of the corresponding mask marker;

[0028] The mask language modeling task head is used to predict the word segmentation text corresponding to all the mask tags in the medical mask corpus text based on the text feature tensor and output the corresponding mask word segmentation sequence;

[0029] The masked knowledge element prediction task head incorporates a general or specialty knowledge graph in the medical field, denoted as the medical knowledge graph. The task head input includes a medical knowledge triple and the text feature tensor corresponding to the current triple text; the task head output is a triple element. The medical knowledge graph includes a first node set and a first directed edge set. The first node set includes multiple first nodes, and the first directed edge set includes multiple first directed edges. Each first node is a medical subject noun in the current general or specialty medical field. Each first directed edge connects two first nodes, reflecting the association between the two medical subject nouns corresponding to the two first nodes. The medical knowledge triple consists of three text elements: a first medical subject noun, a first subject relation, and a second medical subject noun. The first subject relation is the association between the first and second medical subject nouns. One text element in the medical knowledge triple is masked by the mask marker. The triple element is the predicted text corresponding to the text element masked by the mask marker in the medical knowledge triple.

[0030] The masked knowledge element prediction task head is used to predict the element text of the medical knowledge triplet that is masked by the mask mark based on the text feature tensor and the medical knowledge graph, and output the prediction result as the corresponding triplet element.

[0031] The task head input of the unstructured text entity recognition task head is a text feature tensor, and the task head output is a sequence of medical knowledge triples; the current text feature tensor is a feature tensor of an unstructured medical text; the current unstructured medical text is a piece of natural language text in the medical field; the sequence of medical knowledge triples is formed by sorting multiple medical knowledge triples in the current unstructured medical text.

[0032] The unstructured text entity recognition task head and the masked knowledge element prediction task head share the medical knowledge graph; the unstructured text entity recognition task head is implemented based on an entity recognition model constructed by a deep learning model; the unstructured text entity recognition task head is used to identify knowledge triples based on the text feature tensor and the medical knowledge graph and output the corresponding medical knowledge triple sequence;

[0033] The task head input of the structured text entity recognition task head is a text feature tensor, and the task head output is a sequence of medical knowledge triples; the current text feature tensor is a feature tensor of a structured medical text; the current structured medical text is a structured text in the medical field;

[0034] The structured text entity recognition task head and the masked knowledge element prediction task head share the medical knowledge graph; the structured text entity recognition task head is implemented based on another type of entity recognition model constructed by a deep learning model; the structured text entity recognition task head is used to identify knowledge triples based on the text feature tensor and the medical knowledge graph and output the corresponding medical knowledge triple sequence;

[0035] The text generation task head takes a text feature tensor as input and outputs a medical treatment plan or health management plan text. The current text feature tensor is a feature tensor for generating medical text instructions. The generated medical text instructions are formatted natural language prompts. The configurable parameters for the generated medical text instructions include a state description text parameter and a generated object type parameter. The state description text parameter is either unstructured or structured medical text. The generated object type parameter includes at least treatment plans and health management plans. The generated medical text instructions prompt the model to generate and output a plan text corresponding to the generated object type parameter based on the state description text parameter.

[0036] The text generation task head and the mask knowledge element prediction task head share the medical knowledge graph; the text generation task head is implemented based on another type of text generator model constructed by a deep learning model; the text generation task head is used to perform solution text generation processing based on the text feature tensor and output the corresponding treatment solution text or health management solution text.

[0037] Preferably, the task head input of the waveform classification task head is an image feature tensor corresponding to the 2D waveform, and the task head output is a 2D waveform type; the 2D waveform type includes at least electrocardiogram, electroencephalogram, electromyography, electrooculogram, blood pressure waveform, respiratory waveform, and blood oxygen saturation waveform;

[0038] The waveform classification task head is based on a 2D visual classification model constructed by a deep learning model; the waveform classification task head is used to perform waveform classification and recognition processing based on the image feature tensor input to the task head and output the corresponding 2D waveform type;

[0039] The task head input of the waveform disease prediction task head is an image feature tensor corresponding to the 2D waveform image, and the task head output is a predicted disease type corresponding to the current 2D waveform image. When the image type of the 2D waveform image is an electrocardiogram (ECG), the corresponding predicted disease type includes at least atrial fibrillation, premature beats, tachycardia, bradycardia, myocardial infarction, myocardial ischemia, and cardiac conduction block. When the image type of the 2D waveform image is an electroencephalogram (EEG), the corresponding predicted disease type includes at least epilepsy, cerebral hemorrhage, cerebral thrombosis, cerebral infarction, encephalitis, brain tumor, traumatic brain injury, schizophrenia, bipolar disorder, mental abnormality, and poisoning. When the image type of the 2D waveform image is an electromyography (EMG), the corresponding predicted disease type includes at least myasthenia gravis, muscular dystrophy, polymyositis, dermatomyositis, amyotrophic lateral sclerosis (ALS), and Guillain-Barré syndrome. The predicted disease types include: polyneuropathy, oculomotor nerve injury, trochlear nerve injury, and facial nerve inflammation; when the 2D waveform image type is an electrooculogram, the corresponding predicted disease types include at least toxic retinopathy, vitrectomyelitis, retinal detachment, retinitis pigmentosa, diabetic retinopathy, and recurrent iridocyclitis; when the 2D waveform image type is a blood pressure waveform, the corresponding predicted disease types include at least hypertension, hypotension, white coat hypertension, and circadian rhythm abnormalities; when the 2D waveform image type is a respiratory waveform, the corresponding predicted disease types include at least airway obstruction, acute respiratory distress syndrome, and obstructive respiratory diseases; when the 2D waveform image type is a blood oxygen saturation waveform, the corresponding predicted disease types include at least respiratory failure, cardiovascular diseases, and chronic obstructive pulmonary disease.

[0040] The waveform disease prediction task head is based on another type of 2D visual classification model constructed by a deep learning model; the waveform disease prediction task head is used to identify the waveform type according to the image feature tensor input to the task head, and predict the corresponding disease type according to the identified waveform type and the image feature tensor, and output the corresponding predicted disease type.

[0041] The task head input of the image target detection task head is the image feature tensor corresponding to the 3D image, and the task head output is a set of target detection boxes; the target detection box set includes multiple target detection boxes; each target detection box includes a detection box identifier, detection box 3D size, detection box center coordinates, detection box orientation, and detection box type; the detection box type is a type of anatomical structure in the medical field.

[0042] The image target detection task head is based on a 3D target recognition model constructed by a deep learning model; the image target detection task head is used to perform 3D anatomical structure target recognition processing according to the image feature tensor input to the task head and output the corresponding target detection box set;

[0043] The task head of the image semantic segmentation task is input to an image feature tensor corresponding to a 3D image, and outputs a 3D semantic segmentation map. The three-dimensional dimensions of the image feature tensor are consistent with those of the 3D image. The three-dimensional dimensions of the 3D semantic segmentation map are consistent with those of the image feature tensor. The voxel-level semantics of the 3D semantic segmentation map include background semantics and foreground semantics. The foreground semantics include one or more types of anatomical structure semantics. The anatomical structure semantics correspond one-to-one with the anatomical structure type.

[0044] The image semantic segmentation task head is implemented based on a 3D semantic segmentation model constructed by a deep learning model; the image semantic segmentation task head is used to perform 3D anatomical structure semantic segmentation processing based on the image feature tensor input to the task head and output the corresponding 3D semantic segmentation map.

[0045] Preferably, the step of performing a one-stage text model training on the pre-trained text model based on the one-stage text task model set specifically includes:

[0046] The text pre-trained model is connected to each task head of the first-stage text task model set to form a corresponding first training model; and through big data collection, a corresponding model dataset is constructed for each first training model based on the model function and corresponding input / output data format of the text pre-trained model and the model function and corresponding input / output data format of each task head of the first-stage text task model set, denoted as the corresponding first task head dataset; and a matching model training loss function is designed for each first training model, denoted as the corresponding first task head loss function.

[0047] The first step of text model training is performed on the text pre-training model based on the first task head dataset corresponding to the mask language modeling task head and the first task head loss function;

[0048] After the first step of text model training is completed, the second step of text model training is performed on the text pre-trained model based on the first task head dataset corresponding to the mask knowledge element prediction task head and the first task head loss function.

[0049] After the second step of text model training is completed, the text pre-trained model is trained in the third step based on the first task head dataset corresponding to the unstructured text entity recognition task head and the first task head loss function.

[0050] After the third step of text model training is completed, the text pre-trained model is trained in the fourth step based on the first task head dataset corresponding to the structured text entity recognition task head and the first task head loss function.

[0051] After the fourth step of text model training is completed, the fifth step of text model training is performed on the text pre-trained model based on the first task head dataset and the first task head loss function corresponding to the text generation task head.

[0052] After the fifth step of text model training is completed, the first phase of text model training is confirmed to be finished.

[0053] Preferably, the step of performing a one-stage image model training on the image pre-trained model based on the one-stage visual task model set specifically includes:

[0054] The image pre-trained model is connected to each task head of the first-stage visual task model set to form a corresponding second training model; and through big data collection, a corresponding model dataset is constructed for each second training model based on the model function and corresponding input / output data format of the image pre-trained model and the model function and corresponding input / output data format of each task head of the first-stage visual task model set, denoted as the corresponding second task head dataset; and a matching model training loss function is designed for each second training model, denoted as the corresponding second task head loss function.

[0055] The image pre-training model is trained in the first step based on the second task head dataset corresponding to the waveform classification task head and the second task head loss function.

[0056] After the first step of image model training is completed, the second step of image model training is performed on the image pre-training model based on the second task head dataset corresponding to the waveform disease prediction task head and the second task head loss function.

[0057] After the second step of image model training is completed, the third step of image model training is performed on the image pre-trained model based on the second task head dataset and the second task head loss function corresponding to the image target detection task head.

[0058] After the third step of image model training is completed, the fourth step of image model training is performed on the image pre-trained model based on the second task head dataset corresponding to the image semantic segmentation task head and the second task head loss function.

[0059] After the fourth step of image model training is completed, the first stage of image model training is confirmed to be finished.

[0060] Preferably, the two-stage multimodal training of the neural structure searcher and the neural network module set of the multimodal medical pre-training model based on all the downstream task models specifically includes:

[0061] The output of the multimodal medical pre-trained model is connected to the inputs of all downstream task models to form a corresponding multi-task training framework. Through big data collection, a corresponding model dataset, denoted as the corresponding multi-task dataset, is constructed for the multi-task training framework based on the model functions and corresponding input / output data formats of the multimodal medical pre-trained model and all downstream task models. A matching multi-task loss function is designed for the multi-task training framework. Multi-task training is performed on the multi-task training framework based on the multi-task dataset and the multi-task loss function, and during multi-task training, only the model parameters of all downstream task models and the model parameters of the neural structure searcher and the neural network module set of the multimodal medical pre-trained model are modulated.

[0062] A second aspect of the present invention provides an apparatus for implementing the training method of the multimodal medical pre-trained model described in the first aspect above, the apparatus comprising: a multimodal model construction module, a one-stage model training module, and a two-stage model training module;

[0063] The multimodal model construction module is used to select a type of pre-trained large language model as a text pre-training model, select a type of visual pre-training model that can simultaneously support 2D / 3D image processing as an image pre-training model, select multiple types of basic neural network modules to form a neural network module set, and configure a corresponding neural structure searcher for the neural network module set; and construct a multimodal large model based on the text pre-training model, the image pre-training model, the neural structure searcher, and the neural network module set as the corresponding multimodal medical pre-training model;

[0064] The first-stage model training module is used to configure a first-stage text task model set for the text pre-trained model and a first-stage visual task model set for the image pre-trained model. The first-stage text task model set includes a masked language modeling task head, a masked knowledge element prediction task head, an unstructured text entity recognition task head, a structured text entity recognition task head, and a text generation task head. The first-stage visual task model set includes a waveform classification task head, a waveform disease prediction task head, an image object detection task head, and an image semantic segmentation task head.

[0065] The first-stage model training module is also used to perform a first-stage text model training on the text pre-trained model based on the first-stage text task model set; and to perform a first-stage image model training on the image pre-trained model based on the first-stage visual task model set; and to freeze the model parameters of the text pre-trained model and the image pre-trained model at the end of the first-stage text and image model training.

[0066] The two-stage model training module is used to configure one or more downstream task models for the multimodal medical pre-training model; and to perform two-stage multimodal training on the neural structure searcher and the neural network module set of the multimodal medical pre-training model based on all the downstream task models.

[0067] The two-stage model training module is also used to freeze the model parameters of the multimodal medical pre-trained model and confirm the end of model training after the two-stage training is completed.

[0068] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0069] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0070] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0071] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0072] This invention provides a training method, apparatus, electronic device, and computer-readable storage medium for a multimodal medical pre-training model. As described above, this invention selects a type of pre-trained large language model as the text pre-training model and a type of visual pre-training model capable of simultaneously supporting 2D / 3D image processing as the image pre-training model. It also selects multiple types of basic neural network modules to form a neural network module set and configures a corresponding neural architecture searcher for the neural network module set. A multimodal medical pre-training model is constructed based on the text pre-training model, image pre-training model, neural architecture searcher, and neural network module set. A corresponding first-stage text / visual task model set is configured for the text / image pre-training model. The text / image pre-training model is then trained separately based on the first-stage text / visual task model set. After the first-stage training is completed, multiple downstream task models are configured to form a multi-task training framework for the multimodal medical pre-training model. The second-stage training of the neural architecture searcher and neural network module set is completed by training the multi-task training framework. The embodiments of this invention address several shortcomings of traditional single-modal analysis frameworks: 1) It provides a multimodal medical pre-training model capable of multimodal feature fusion of medical text, 2D medical waveforms, and 3D medical images; 2) The two-stage training method based on the embodiments of this invention not only reduces the overall training difficulty but also improves the model's adaptability to medical scenarios; 3) The multimodal medical pre-training model based on the embodiments of this invention provides an encoding sharing platform that can simultaneously address multiple downstream tasks, improving the convenience of multi-task fusion and the overall task prediction accuracy. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of a training method for a multimodal medical pre-training model provided in Embodiment 1 of the present invention;

[0074] Figure 2 This is a schematic diagram of the modules of the multimodal medical pre-training model provided in Embodiment 1 of the present invention;

[0075] Figure 3 This is a schematic diagram illustrating the process of confirming the initial network structure to the preferred network structure provided in Embodiment 1 of the present invention;

[0076] Figure 4 A schematic diagram of a one-stage text task model set and its corresponding one-stage text model training sequence provided in Embodiment 1 of the present invention;

[0077] Figure 5 A schematic diagram of the training sequence of a one-stage visual task model set and its corresponding one-stage image model provided in Embodiment 1 of the present invention.

[0078] Figure 6This is a schematic diagram of the modules of the multi-task training framework provided in Embodiment 1 of the present invention;

[0079] Figure 7 This is a module structure diagram of a training device for a multimodal medical pre-training model provided in Embodiment 2 of the present invention;

[0080] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0082] Embodiment 1 of the present invention provides a training method for a multimodal medical pre-training model, such as... Figure 1 The diagram illustrates a training method for a multimodal medical pre-training model provided in Embodiment 1 of the present invention. This method mainly includes the following steps:

[0083] Step 1: Select a type of pre-trained large language model as the text pre-training model, select a type of visual pre-training model that can simultaneously support 2D / 3D image processing as the image pre-training model, select multiple types of basic neural network modules to form a neural network module set, and configure a corresponding neural architecture searcher for the neural network module set; and construct a multimodal large model based on the text pre-training model, image pre-training model, neural architecture searcher and neural network module set as the corresponding multimodal medical pre-training model.

[0084] Here, the pre-trained large language models mentioned in this embodiment of the invention refer to a class of large language models (LLMs) implemented based on the Transformer architecture. These models have already undergone pre-training on a series of natural language processing (NLP) tasks using general-domain natural language text corpora. The pre-training implementation process of all LLMs can be obtained from publicly available technical documents and will not be repeated here. These pre-trained LLMs include at least the BERT series models.

[0085] The visual pre-trained models mentioned in this embodiment of the invention are a class of Large Vision Models (LVMs) compatible with both 2D and 3D modalities. They have been pre-trained for a series of image processing tasks based on general-domain 2D / 3D image corpora. The pre-training implementation process of all LVMs can be obtained from publicly available technical documents and will not be repeated here. These pre-trained LVMs include at least MultiResViT, UniFormer, Hybrid CNN-Transformer, nnU-Net, Uni3D, Uni-MAE, and MedFMC models.

[0086] The basic neural network modules mentioned in this embodiment of the invention include at least a feature concatenation module, a weighted feature concatenation module, a residual network module, a cross-attention encoding module, a graph neural network module, a 1D convolutional neural network module, a 2D convolutional neural network module, a 3D convolutional neural network module, a residual connection module, and a Hadamard product module. It should be noted that all basic neural network modules in this embodiment can be considered as unit processing modules with pre-fixed input / output formats, internal model structures, and internal processing flows. The internal model structures of these unit processing modules are conventional small models or standard neural network structures, composed of a series of processing layers connected together. The connection relationships of the processing layers in the internal model structure are fixed, but the model parameters, i.e., the parameters of all processing layers, can be updated through training. For example, the internal processing flow of the feature concatenation module can be considered as a sequential connection of a vector concatenation layer and the processing flow of an MLP network, wherein the nonlinear layers and activation functions of the MLP network are fixed, but the weight matrices and offset vectors of all nonlinear layers can be updated through training.

[0087] The multimodal medical pre-training model of this invention is used to perform cross-modal feature encoding processing based on the model input text and the model input image, and output the corresponding multimodal feature tensor, such as... Figure 2The diagram shows a module schematic of the multimodal medical pre-training model provided in Embodiment 1 of the present invention. The input text for the model is natural language text or structured text in the medical field. The model input image is a 2D waveform or 3D image in the medical field; the 2D waveform image type includes at least Electrocardiogram (ECG), Electroencephalogram (EEG), Electromyogram (EMG), Electrooculogram (ROG), Blood Pressure Waveform (BPW), Respiratory Waveform (RW), and Oxygen Saturation Waveform (SpO2Waveform); the 3D image image type includes at least Computed Tomography (CT), Magnetic Resonance Imaging (MRI), 3D Ultrasound Imaging (3D US), Positron Emission Tomography (PET), and Single-Photon Emission Computed Tomography (SPECT). Optical coherence tomography (SPET), optical coherence tomography (OCT), and cone-beam computed tomography (CBCT).

[0088] like Figure 2 As shown, the first model input terminal of the multimodal medical pre-trained model is used to receive the model input text, the second model input terminal is used to receive the model input image, and the model output terminal is used to output the corresponding multimodal feature tensor.

[0089] like Figure 2 As shown, the model components of the multimodal medical pre-trained model include: a text pre-trained model, an image pre-trained model, a neural structure searcher, and a neural network module set; among which, the neural network module set includes multiple basic neural network modules.

[0090] like Figure 2As shown, the connection relationships of the model components in the multimodal medical pre-training model are as follows: the input end of the text pre-training model is connected to the input end of the first model, and the output end is connected to the first input end of the neural structure searcher; the input end of the image pre-training model is connected to the input end of the second model, and the output end is connected to the second input end of the neural structure searcher; the output end of the neural structure searcher is connected to the output end of the model, and the neural structure searcher is also connected to the neural network module set.

[0091] The functionalities of the model components in the multimodal medical pre-trained model are shown below.

[0092] 1) Text pre-trained model:

[0093] The text pre-training model in this embodiment of the invention is used to encode the text features of the input text to obtain the corresponding text feature tensor and send it to the neural structure searcher.

[0094] 2) Image pre-trained model:

[0095] The image pre-training model in this embodiment of the invention is used to perform 2D / 3D adaptive feature encoding on the input image to obtain the corresponding image feature tensor, which is then sent to the neural structure searcher.

[0096] 3) Neural structure searcher:

[0097] The neural architecture searcher of this invention is used to construct an initial network structure according to a set of neural network modules based on a preset initial network construction rule before two-stage training; and to assign a learnable training weight parameter to all candidate modules of each i-th layer candidate module group in the initial network structure.

[0098] like Figure 3 The schematic diagram illustrating the process of confirming the initial network structure to the preferred network structure provided in Embodiment 1 of the present invention shows that the initial network structure consists of N i-th input layers, N i-th layer candidate module groups, and one output layer, where 1 ≤ index i ≤ N, and N is a preset positive integer. Each i-th layer candidate module group consists of one or more candidate modules, and each candidate module is a basic neural network module. The sum of all training weight parameters corresponding to each i-th layer candidate module group is 1. It should be noted that in this embodiment of the present invention, the input and output data formats of all candidate modules in each i-th layer candidate module group are consistent.

[0099] like Figure 3As shown, the connection relationships of each layer in the initial network structure of this embodiment are as follows: the input end of the first input layer is connected to the input end of the neural structure searcher, and the output end is connected to the input end of each candidate module in the first layer candidate module group. The output end of each candidate module in the first layer candidate module group is connected to one input end of the second input layer. Similarly, the output end of each candidate module in the (N-1)th layer candidate module group is connected to one input end of the Nth input layer, the output end of the Nth input layer is connected to the input end of each candidate module in the Nth layer candidate module group, and the output end of each candidate module in the Nth layer candidate module group is connected to one input end of the output layer. The output end of the output layer is connected to the output end of the neural structure searcher.

[0100] The processing mechanism of each input layer and output layer in the initial network structure of this embodiment is as follows: The first input layer is used to receive text feature tensors and image feature tensors to form a corresponding first tensor group, and send the first tensor group to each candidate module of the first layer candidate module; each i-th input layer from the second input layer to the Nth input layer is used to perform weighted summation calculation on the output tensors of all candidate modules of the i-1th layer candidate module group according to the training weight parameters corresponding to all candidate modules of the corresponding i-1th layer candidate module group, and use the calculation result as the corresponding i-1th layer output tensor, and send the i-1th layer output tensor to each candidate module of the corresponding i-th layer candidate module group; the output layer is used to perform weighted summation calculation on the output tensors of all candidate modules of the Nth layer candidate module group according to the training weight parameters corresponding to all candidate modules of the Nth layer candidate module group to obtain the corresponding Nth layer output tensor, and use the Nth layer output tensor as the corresponding multimodal feature tensor and output it.

[0101] It should be noted that the training weight parameters of all candidate modules in the initial network structure are updated during the second-stage training process.

[0102] The neural architecture searcher in this embodiment of the invention is also used to prune the initial network structure at the end of the two-stage training to obtain the corresponding preferred network structure and fix it. Specifically, in the initial network structure, the candidate module with the highest training weight parameter in each i-th layer candidate module group is taken as the corresponding best i-th layer module; in each i-th layer candidate module group, only the connection between the corresponding best i-th layer module and the previous and next input / output layers is retained, and the other remaining candidate modules in this layer are removed from the initial network structure; the training weight parameter of each best i-th layer module is frozen to 1; the finally obtained network structure is taken as the corresponding preferred network structure; and the network structure and network parameters of the preferred network structure are frozen. Here, the processing mechanism for pruning the initial network structure to obtain the corresponding preferred network structure can be referred to Figure 3 To understand.

[0103] The neural architecture searcher in this embodiment of the invention is also used to perform feature fusion processing on the input text feature tensor and image feature tensor based only on the preferred network structure after the preferred network structure is frozen, and output the corresponding multimodal feature tensor.

[0104] Step 2: Configure a one-stage text task model set for the text pre-trained model; and configure a one-stage visual task model set for the image pre-trained model.

[0105] Here, the task header description information for the one-stage text task model set and the one-stage visual task model set of this embodiment of the invention is as follows.

[0106] 1) One-stage text task model set:

[0107] The one-stage text task model set of this invention includes a masked language modeling task head, a masked knowledge element prediction task head, an unstructured text entity recognition task head, a structured text entity recognition task head, and a text generation task head, such as... Figure 4 The diagram illustrates the one-stage text task model set and its corresponding one-stage text model training sequence provided in Embodiment 1 of the present invention. The description information for each task header is as follows.

[0108] A. Mask Language Modeling (MLM) Task Header:

[0109] The task head input of the masked language modeling task head in this embodiment of the invention includes medical masked corpus text and the text feature tensor corresponding to the current corpus text, and the task head output is a masked word segmentation sequence.

[0110] The task header input consists of a sentence or paragraph of natural language text in the medical field. One or more segmented texts in the medical masked text are masked by preset mask markers, and the segmentation index of each mask marker in the medical masked text is recorded as the corresponding mask segmentation index. The task header outputs a masked segmentation sequence composed of one or more masked prediction texts, with a one-to-one correspondence between the masked prediction texts and the mask markers, and each masked prediction text being the segmented prediction text of the corresponding mask marker.

[0111] The masked language modeling task head is used to predict the segmented text corresponding to all masked markers in the medical masked corpus text based on the text feature tensor and output the corresponding masked word segmentation sequence. Specifically:

[0112] The text feature tensor is mapped to a vocabulary probability distribution vector using the fully connected network built into the task header to obtain the corresponding text probability tensor. The word segmentation probability vectors corresponding to each mask marker in the medical masked corpus text are then used as the corresponding mask probability vectors. The maximum probability among the mask probability vectors is recorded as the corresponding maximum mask probability. The words corresponding to the maximum mask probability in the preset model vocabulary are used as the corresponding word segmentation prediction texts. All the obtained word segmentation prediction texts are sorted according to the corresponding mask word segmentation index to form the corresponding mask word segmentation sequence output.

[0113] The text feature tensor includes multiple word segmentation feature vectors, each corresponding one-to-one with the word segmentation text of the medical mask corpus; the text probability tensor consists of multiple word segmentation probability vectors, each corresponding one-to-one with the word segmentation feature vectors; the length of the word segmentation probability vector matches the total number of words in the model vocabulary; the probabilities in the word segmentation probability vectors correspond one-to-one with the words in the model vocabulary; and the model vocabulary is the vocabulary of the pre-trained text model.

[0114] It should be noted that the embodiments of the present invention only limit the function of the task head model of the MLM task head, and do not limit the specific implementation of the task head model structure. In practical applications, a general model structure with similar functions can be used for implementation, or the model structure can be customized based on the function of the task head model. For example, the general MLM model of LLMs corresponding to the text pre-training model of the embodiments of the present invention can be used directly as the MLM task head, or the developers of the model training party can customize the design based on their own development experience.

[0115] B. Header for Masked Knowledge Element Prediction Task:

[0116] The task head input of the masked knowledge element prediction task head in this embodiment of the invention includes a medical knowledge triplet and a text feature tensor corresponding to the current triplet text. The task head output is a triplet element. Furthermore, the masked knowledge element prediction task head in this embodiment of the invention incorporates a general or specialty knowledge graph in the medical field, denoted as the medical knowledge graph.

[0117] The task header input medical knowledge triples consist of three text elements: a first medical subject noun, a first subject relation, and a second medical subject noun. The first subject relation is the association between the first and second medical subject nouns. One element of the medical knowledge triple is masked. The task header outputs the predicted text corresponding to the masked element in the medical knowledge triple.

[0118] The medical knowledge graph includes a first node set and a first directed edge set; the first node set includes multiple first nodes, and the first directed edge set includes multiple first directed edges; each first node is a medical subject term in the current general practice or specialty medical field; each first directed edge connects two first nodes, which is used to reflect the relationship between the two medical subject terms corresponding to the two first nodes.

[0119] The masked knowledge element prediction task head is used to predict the text elements of medical knowledge triples that are masked by the text feature tensor and the medical knowledge graph, and outputs the prediction results as the corresponding triple elements. Specifically:

[0120] Based on the two unmasked text elements in the medical knowledge triplet, an n-hop domain subgraph is extracted from the medical knowledge graph. Then, using the graph embedding encoding module built into the task header, node embedding and directed edge embedding are performed on the n-hop domain subgraph to obtain corresponding node embedding feature sets and edge embedding feature sets. These node embedding feature sets and edge embedding feature sets form the corresponding subgraph embedding features. Finally, using the graph attention network built into the task header, the subgraph embedding features are feature-encoded to obtain the corresponding subgraph feature tensors. The text feature tensors are mapped to query tensors, and the subgraph feature tensors are mapped to key-value tensors. Finally, the attention operation module built into the task header is used... The block performs attention operations based on the query, key, and value tensors and uses the results as the corresponding text-graph fusion tensor. It then maps the text-graph fusion tensor to node and edge probability distribution vectors using the fully connected network built into the task header, obtaining the corresponding node and edge probability tensors. The block uses the node and edge probability vectors corresponding to the masked text elements in the medical knowledge triplet as the current mask probability vector. The block records the maximum probability in the current mask probability vector as the current maximum probability. Finally, it outputs the medical subject term or the association type of the first directed edge of the first node corresponding to the current maximum probability in the medical knowledge graph as the element of the corresponding triplet.

[0121] The text feature tensor includes three word segmentation feature vectors, each corresponding one-to-one with the text elements of the medical knowledge triplet; the text-graph fusion tensor consists of three text-graph fusion vectors, each corresponding one-to-one with the word segmentation feature vectors; the node and edge probability tensor consists of three node and edge probability vectors, each corresponding one-to-one with the text-graph fusion vector; the length of the node and edge probability vectors matches the total number of nodes and directed edges in the medical knowledge graph; each probability in the node and edge probability vectors corresponds to a first node or a first directed edge in the medical knowledge graph.

[0122] It should be noted that the embodiments of the present invention only limit the function of the task head model of the mask knowledge element prediction task head, and do not limit the specific implementation of the task head model structure; in practical applications, a general model structure with similar functions can be used for implementation, or the model structure can be customized based on the function of the task head model.

[0123] C. Unstructured text entity recognition task head:

[0124] The task head input of the unstructured text entity recognition task head in this embodiment of the invention is a text feature tensor, and the task head output is a sequence of medical knowledge triples.

[0125] Among them, the current text feature tensor is the feature tensor of an unstructured medical text; the current unstructured medical text is a natural language text in the medical field; the medical knowledge triple sequence is formed by sorting multiple medical knowledge triples in the current unstructured medical text.

[0126] The unstructured text entity recognition task head of this invention is used to identify knowledge triples based on text feature tensors and medical knowledge graphs and output the corresponding medical knowledge triple sequence.

[0127] It should be noted that the unstructured text entity recognition task head and the masked knowledge element prediction task head of this embodiment of the invention share a medical knowledge graph. The unstructured text entity recognition task head of this embodiment of the invention is implemented based on an entity recognition model constructed by a deep learning model.

[0128] It should also be noted that the embodiments of the present invention only limit the function of the task head model of the unstructured text entity recognition task head, and do not limit the specific implementation of the task head model structure; in practical applications, a general model structure with similar functions can be used for implementation, or the model structure can be customized based on the function of the task head model.

[0129] D. Structured Text Entity Recognition Task Header:

[0130] The input to the structured text entity recognition task head in this embodiment of the invention is a text feature tensor, and the output is a sequence of medical knowledge triples.

[0131] Here, the current text feature tensor is the feature tensor of a structured medical text; the current structured medical text is a structured text in the medical field.

[0132] The structured text entity recognition task head of this invention is used to identify knowledge triples based on text feature tensors and medical knowledge graphs and output the corresponding medical knowledge triple sequence.

[0133] It should be noted that the structured text entity recognition task head and the masked knowledge element prediction task head of this embodiment of the invention share the medical knowledge graph. The structured text entity recognition task head of this embodiment of the invention is implemented based on another type of entity recognition model constructed from deep learning models.

[0134] It should also be noted that the embodiments of the present invention only limit the function of the task head model of the structured text entity recognition task head, and do not limit the specific implementation of the task head model structure; in practical applications, a general model structure with similar functions can be used for implementation, or the model structure can be customized based on the function of the task head model.

[0135] E. Text Generation Task Header:

[0136] The task head input of the text generation task head in this embodiment of the invention is a text feature tensor, and the task head output is a treatment plan text or health management plan text in the medical field.

[0137] Here, the current text feature tensor is a feature tensor for generating medical text instructions. The generated medical text instruction is a formatted natural language prompt text; its configurable parameters include a state description text parameter and a generated object type parameter. The state description text parameter is an unstructured or structured medical text, and the generated object type parameter includes at least treatment plans and health management plans. This generated medical text instruction is used by the prompting model to generate and output a plan text corresponding to the generated object type parameter based on the state description text parameter.

[0138] The text generation task head in this embodiment of the invention is used to perform solution text generation processing based on text feature tensors and output the corresponding treatment solution text or health management solution text.

[0139] It should be noted that the text generation task head and the masked knowledge element prediction task head in this embodiment of the invention share the medical knowledge graph. The text generation task head in this embodiment of the invention is implemented based on another type of text generator model constructed by a deep learning model.

[0140] It should also be noted that the embodiments of the present invention only limit the function of the task head model for text generation task head, and do not limit the specific implementation of the task head model structure; in practical applications, a general model structure with similar functions can be used for implementation, or the model structure can be customized based on the function of the task head model.

[0141] 2) First-stage visual task model set:

[0142] The one-stage visual task model set of this invention includes a waveform classification task head, a waveform disease prediction task head, an image object detection task head, and an image semantic segmentation task head, such as... Figure 5 The diagram illustrates the one-stage visual task model set and its corresponding one-stage image model training sequence provided in Embodiment 1 of the present invention. The description information for each task head is shown below.

[0143] A. Waveform Classification Task Header:

[0144] The waveform classification task head of this embodiment of the invention takes an image feature tensor corresponding to a 2D waveform as input and outputs a 2D waveform type as output.

[0145] Among them, the 2D waveform types include at least electrocardiogram, electroencephalogram, electromyography, electrooculogram, blood pressure waveform, respiratory waveform, and blood oxygen saturation waveform.

[0146] The waveform classification task head in this embodiment of the invention is used to perform waveform classification and recognition processing based on the image feature tensor input to the task head and output the corresponding 2D waveform type.

[0147] It should be noted that the waveform classification task head in this embodiment of the invention is based on a 2D visual classification model constructed by a deep learning model.

[0148] It should also be noted that the embodiments of the present invention only limit the function of the task head model of the waveform classification task head, and do not limit the specific implementation of the task head model structure; in practical applications, a general model structure with similar functions can be used for implementation, or the model structure can be customized based on the function of the task head model.

[0149] B. Waveform Disease Prediction Task Header:

[0150] The task head input of the waveform disease prediction task head in this embodiment of the invention is an image feature tensor corresponding to a 2D waveform, and the task head output is a predicted disease type corresponding to the current 2D waveform.

[0151] Among them, when the 2D waveform image type is electrocardiogram (ECG), the corresponding predicted disease types include at least atrial fibrillation, premature beats, tachycardia, bradycardia, myocardial infarction, myocardial ischemia, and cardiac conduction block; when the 2D waveform image type is electroencephalogram (EEG), the corresponding predicted disease types include at least epilepsy, cerebral hemorrhage, cerebral thrombosis, cerebral infarction, encephalitis, brain tumor, traumatic brain injury, schizophrenia, bipolar disorder, mental abnormalities, and poisoning; when the 2D waveform image type is electromyography (EMG), the corresponding predicted disease types include at least myasthenia gravis, muscular dystrophy, polymyositis, dermatomyositis, amyotrophic lateral sclerosis (ALS), Guillain-Barré syndrome, polyneuropathy, oculomotor nerve injury, trochlear nerve injury, and facial nerve palsy. When the 2D waveform image type is an electrooculogram, the corresponding predicted disease types include at least toxic retinopathy, yolk sac macular degeneration, retinal detachment, retinitis pigmentosa, diabetic retinopathy, and recurrent iridocyclitis; when the 2D waveform image type is a blood pressure waveform, the corresponding predicted disease types include at least hypertension, hypotension, white coat hypertension, and circadian rhythm abnormalities; when the 2D waveform image type is a respiratory waveform, the corresponding predicted disease types include at least airway obstruction, acute respiratory distress syndrome, and obstructive respiratory disease; when the 2D waveform image type is a blood oxygen saturation waveform, the corresponding predicted disease types include at least respiratory failure, cardiovascular disease, and chronic obstructive pulmonary disease.

[0152] The waveform disease prediction task head of this embodiment of the invention is used to identify the waveform type based on the image feature tensor input to the task head, and to predict the corresponding disease type based on the identified waveform type and image feature tensor, and output the corresponding predicted disease type.

[0153] It should be noted that the waveform disease prediction task head in this embodiment of the invention is based on another type of 2D visual classification model constructed by a deep learning model.

[0154] It should also be noted that the embodiments of the present invention only limit the function of the task head model of the waveform disease prediction task head, and do not limit the specific implementation of the task head model structure; in practical applications, a general model structure with similar functions can be used for implementation, or the model structure can be customized based on the function of the task head model.

[0155] C. Image Target Detection Task Header:

[0156] The task head input of the image target detection task head in this embodiment of the invention is an image feature tensor corresponding to a 3D image map, and the task head output is a set of target detection boxes.

[0157] The target detection box set includes multiple target detection boxes; each target detection box includes a detection box identifier, detection box 3D size, detection box center coordinates, detection box orientation, and detection box type; the detection box type is a type of anatomical structure in the medical field.

[0158] The image target detection task head in this embodiment of the invention is used to perform 3D anatomical structure target recognition processing based on the image feature tensor input by the task head and output the corresponding target detection box set.

[0159] It should be noted that the image target detection task head in this embodiment of the invention is based on a 3D target recognition model constructed by a deep learning model.

[0160] It should also be noted that the embodiments of the present invention only limit the function of the task head model of the image target detection task head, and do not limit the specific implementation of the task head model structure; in practical applications, a general model structure with similar functions can be used for implementation, or the model structure can be customized based on the function of the task head model.

[0161] D. Image semantic segmentation task head:

[0162] The task head of the image semantic segmentation task takes an image feature tensor corresponding to a 3D image as input and outputs a 3D semantic segmentation map.

[0163] Among them, the three-dimensional size of the image feature tensor is consistent with that of the 3D image; the three-dimensional size of the 3D semantic segmentation map is consistent with that of the image feature tensor; the voxel-level semantics of the 3D semantic segmentation map includes background semantics and foreground semantics; the foreground semantics includes one or more types of anatomical structure semantics; and the anatomical structure semantics correspond one-to-one with the anatomical structure type.

[0164] The image semantic segmentation task head of this embodiment of the invention is used to perform 3D anatomical structure semantic segmentation processing based on the image feature tensor input by the task head and output the corresponding 3D semantic segmentation map.

[0165] It should be noted that the image semantic segmentation task head in this embodiment of the invention is based on a 3D semantic segmentation model constructed by a deep learning model.

[0166] It should also be noted that the embodiments of the present invention only limit the function of the task head model of the image semantic segmentation task head, and do not limit the specific implementation of the task head model structure; in practical applications, a general model structure with similar functions can be used for implementation, or the model structure can be customized based on the function of the task head model.

[0167] Step 3: Perform a one-stage text model training on the pre-trained text model based on the one-stage text task model set; and perform a one-stage image model training on the pre-trained image model based on the one-stage visual task model set; and freeze the model parameters of the pre-trained text model and the pre-trained image model at the end of the one-stage text and model training.

[0168] Specifically, it includes: Step 31, performing a one-stage text model training on the pre-trained text model based on the one-stage text task model set;

[0169] Specifically, this includes: Step 311, connecting the pre-trained text model with each task head of the first-stage text task model set to form a corresponding first training model; and through big data collection, constructing a corresponding model dataset for each first training model based on the model function and corresponding input / output data format of the pre-trained text model and the model function and corresponding input / output data format of each task head of the first-stage text task model set, denoted as the corresponding first task head dataset; and designing a matching model training loss function for each first training model, denoted as the corresponding first task head loss function;

[0170] It should be noted that the first task head dataset of each first training model in the embodiments of the present invention consists of multiple first data records; each first data record consists of a first training data and a first label data; wherein, the data formats of the first training data and the first label data are respectively matched with the input and output data formats of the corresponding task head of the current dataset;

[0171] It should also be noted that the first task head loss function of each first training model in the embodiments of the present invention can adopt a series of general loss functions or be customized based on actual training requirements. The embodiments of the present invention do not limit the loss function of each first task head.

[0172] It should also be noted that, in addition to setting the corresponding first task head dataset and first task head loss function for each first training model, this embodiment of the invention also sets the corresponding model parameter optimizer and model evaluation mechanism for each first training model, referred to as the corresponding first model parameter optimizer and first model evaluation mechanism. Each first model parameter optimizer can adopt a general model parameter optimizer (such as Adam optimizer, SGD optimizer, etc.) or can be customized based on actual training requirements. This embodiment of the invention does not specifically limit each first model parameter optimizer. Each first model evaluation mechanism can adopt a general model evaluation function or method (such as MAE function, MSE function, RMSE function, F1 score evaluation method, etc.) or can be customized based on actual training evaluation requirements. This embodiment of the invention does not specifically limit each first model evaluation mechanism.

[0173] It should also be noted that, in the embodiments of the present invention, the overall training process of each first training model is divided into a preprocessing stage, a training stage, and an evaluation stage, specifically as follows:

[0174] 1) Preprocessing stage:

[0175] The first task header dataset is randomly split into two sub-datasets according to a preset dataset splitting ratio (e.g., 8:2): the first training set and the first evaluation set;

[0176] 2) Training phase:

[0177] The first training model is trained based on the first training set, the first task head loss function, and the first model parameter optimizer. Specifically, each first training data in the first training set is input into the corresponding task head to obtain the corresponding task head output data, which is recorded as the corresponding first prediction data. Each first prediction data and its corresponding first label data are combined to form a corresponding first prediction-label pair. All the obtained first prediction-label pairs are input into the first task head loss function to calculate the corresponding first loss value. The first loss value is then checked to see if it meets the expected loss value range. If it does, the process proceeds to the evaluation stage. If it does not meet the requirement, the first model parameter optimizer is used to modulate all the model parameters of the current first training model in the direction of minimizing the current loss function. At the end of this modulation round, the above training steps are repeated to re-input each first training data into the corresponding task head to obtain a new batch of first prediction-label pairs and a new first loss value.

[0178] 3) Evaluation phase:

[0179] The training effect of the first training model is evaluated based on the first evaluation set and the first model evaluation mechanism. Specifically, each first training data in the first evaluation set is input into the corresponding task head to obtain the corresponding task head output data, which is recorded as the corresponding second prediction data. Each second prediction data and its corresponding first label data are combined to form a corresponding second prediction-label pair. The first evaluation data is obtained by evaluating all the obtained second prediction-label pairs based on the first model evaluation mechanism. The first evaluation data is then evaluated to determine whether the current first evaluation data meets the expected evaluation value range. If it does not meet the range, the training phase is returned to continue training. If it does meet the range, the training of the current first training model is confirmed to be completed.

[0180] Step 312, and perform the first step of text model training on the text pre-trained model based on the first task head dataset and the first task head loss function corresponding to the mask language modeling task head;

[0181] Step 313: After the first step of text model training is completed, the second step of text model training is performed on the pre-trained text model based on the first task head dataset corresponding to the mask knowledge element prediction task head and the first task head loss function.

[0182] Step 314, and after the second step of text model training is completed, the third step of text model training is performed on the text pre-trained model based on the first task head dataset and the first task head loss function corresponding to the unstructured text entity recognition task head.

[0183] Step 315, and after the text model training in the third step is completed, the text pre-trained model is trained in the fourth step based on the first task head dataset and the first task head loss function corresponding to the structured text entity recognition task head.

[0184] Step 316, and after the fourth step of text model training is completed, the fifth step of text model training is performed on the text pre-trained model based on the first task head dataset and the first task head loss function corresponding to the text generation task head;

[0185] Step 317, and after the text model training in step 5 is completed, confirm that the first stage of text model training is finished;

[0186] Here, the five-step training sequence of the text pre-training model implemented by steps 312-316 is as follows: Figure 4 As shown;

[0187] Step 32, and perform one-stage image model training on the image pre-trained model based on the one-stage visual task model set;

[0188] Specifically, this includes: Step 321, connecting the image pre-trained model with each task head of the first-stage visual task model set to form a corresponding second training model; and through big data collection, constructing a corresponding model dataset for each second training model based on the model function and corresponding input / output data format of the image pre-trained model and the model function and corresponding input / output data format of each task head of the first-stage visual task model set, denoted as the corresponding second task head dataset; and designing a matching model training loss function for each second training model, denoted as the corresponding second task head loss function;

[0189] It should be noted that the second task head dataset of each second training model in the embodiments of the present invention consists of multiple second data records; each second data record consists of a second training data and a second label data; wherein, the data formats of the second training data and the second label data are respectively matched with the input and output data formats of the corresponding task head of the current dataset;

[0190] It should also be noted that the second task head loss function of each second training model in the embodiments of the present invention can adopt a series of general loss functions or be customized based on actual training requirements. The embodiments of the present invention do not limit the loss function of each second task head.

[0191] It should also be noted that, in addition to setting corresponding second task head datasets and second task head loss functions for each second training model, this embodiment of the invention also sets corresponding model parameter optimizers and model evaluation mechanisms for each second training model, referred to as the corresponding second model parameter optimizer and second model evaluation mechanism. Each second model parameter optimizer can adopt a general model parameter optimizer (such as the Adam optimizer, SGD optimizer, etc.) or can be customized based on actual training requirements. This embodiment of the invention does not impose specific limitations on each second model parameter optimizer. Each second model evaluation mechanism can adopt a general model evaluation function or method (such as the MAE function, MSE function, RMSE function, F1 score evaluation method, etc.) or can be customized based on actual training evaluation requirements. This embodiment of the invention does not impose specific limitations on each second model evaluation mechanism.

[0192] It should also be noted that, in the embodiments of the present invention, the overall training process of each second training model is divided into a preprocessing stage, a training stage, and an evaluation stage, specifically as follows:

[0193] 1) Preprocessing stage:

[0194] The second task head dataset is randomly split into two sub-datasets according to a preset dataset splitting ratio (e.g., 8:2): the second training set and the second evaluation set;

[0195] 2) Training phase:

[0196] The second training model is trained based on the second training set, the second task head loss function, and the second model parameter optimizer. Specifically, each second training data point from the second training set is input into the corresponding task head to obtain the corresponding task head output data, which is recorded as the corresponding third prediction data. Each third prediction data point and its corresponding second label data are combined to form a corresponding third prediction-label pair. All the obtained third prediction-label pairs are then input into the second task head loss function to calculate the corresponding second loss value. The second loss value is then checked to see if it meets the expected loss value range. If it does, the process proceeds to the evaluation stage. If it does not meet the requirement, the second model parameter optimizer is used to modulate all the model parameters of the current second training model in the direction of minimizing the current loss function. At the end of this modulation round, the above training steps are repeated to re-input each second training data point into the corresponding task head to obtain a new batch of second prediction-label pairs and a new second loss value.

[0197] 3) Evaluation phase:

[0198] The training effect of the second training model is evaluated based on the second evaluation set and the second model evaluation mechanism. Specifically, each second training data in the second evaluation set is input into the corresponding task head to obtain the corresponding task head output data, which is recorded as the corresponding fourth prediction data. Each fourth prediction data and its corresponding second label data are combined to form a corresponding fourth prediction-label pair. The second evaluation data is obtained by evaluating all the obtained fourth prediction-label pairs based on the second model evaluation mechanism. The second evaluation data is then identified as meeting the expected evaluation value range. If it does not meet the range, the training phase is returned to continue training. If it does meet the range, the training of the current second training model is confirmed to be completed.

[0199] Step 322, and perform the first step of image model training on the image pre-training model based on the second task head dataset corresponding to the waveform classification task head and the second task head loss function;

[0200] Step 323: After the first step of image model training is completed, the second step of image model training is performed on the image pre-trained model based on the second task head dataset corresponding to the waveform disease prediction task head and the second task head loss function.

[0201] Step 324, and after the second step of image model training is completed, the image pre-trained model is trained in the third step based on the second task head dataset and the second task head loss function corresponding to the image object detection task head;

[0202] Step 325, and after the image model training in the third step is completed, the image pre-trained model is trained in the fourth step based on the second task head dataset and the second task head loss function corresponding to the image semantic segmentation task head.

[0203] Step 326, and after the fourth step of image model training is completed, confirm that the first stage of image model training is finished;

[0204] Here, the one-stage four-step training sequence of the image pre-training model implemented by steps 322-325 is as follows: Figure 5 As shown;

[0205] Step 33, and at the end of the first stage of text and image model training, freeze the model parameters of the text pre-trained model and the image pre-trained model.

[0206] It should be noted that once the model parameters of the text pre-trained model and the image pre-trained model are frozen, the model parameters of the text pre-trained model and the image pre-trained model will not be modulated during the subsequent second-stage training process in step 4.

[0207] Step 4: Configure one or more downstream task models for the multimodal medical pre-training model; and perform two-stage multimodal training on the neural architecture searcher and neural network module set of the multimodal medical pre-training model based on all downstream task models.

[0208] Specifically, this includes: Step 41, configuring one or more downstream task models for the multimodal medical pre-trained model;

[0209] Here, the downstream task models in this embodiment of the invention are specific task models at the application level customized based on application requirements, such as text-image question answering tasks, text-image disease prediction tasks, multimodal image segmentation tasks, etc.; this invention does not specifically limit the various downstream task models.

[0210] Step 42, and perform two-stage multimodal training on the neural architecture searcher and neural network module set of the multimodal medical pre-trained model based on all downstream task models;

[0211] Specifically, this includes: connecting the output of the multimodal medical pre-trained model with the inputs of all downstream task models to form a corresponding multi-task training framework, such as... Figure 6 The diagram shows a module schematic of the multi-task training framework provided in Embodiment 1 of the present invention. Through big data collection, a corresponding model dataset, denoted as the corresponding multi-task dataset, is constructed for the multi-task training framework based on the model functions and corresponding input / output data formats of the multimodal medical pre-trained model and all downstream task models. A matching multi-task loss function is designed for the multi-task training framework. Multi-task training is performed on the multi-task training framework based on the multi-task dataset and the multi-task loss function. During multi-task training, only the model parameters of all downstream task models and the model parameters of the neural structure searcher and neural network module set of the multimodal medical pre-trained model are modulated.

[0212] It should be noted that the multi-task dataset in this embodiment of the invention consists of multiple third data records; each third data record consists of model training data and model label data corresponding to a downstream task; wherein, the data formats of the model training data and model label data are respectively matched with the input / output data formats of the multi-task training framework corresponding to the current downstream task; the model training data specifically consists of the model input text and / or model input image of the multi-task training framework matched with the corresponding downstream task, and the model label data is the output label data of the corresponding downstream task.

[0213] It should also be noted that the multi-task loss function in the embodiments of the present invention can be composed of a series of general loss functions or can be customized based on actual training requirements. The embodiments of the present invention do not limit the multi-task loss function.

[0214] It should also be noted that, in addition to setting the corresponding multi-task dataset and multi-task loss function for the multi-task training framework, this embodiment of the invention also sets the corresponding model parameter optimizer and model evaluation mechanism for the multi-task training framework, referred to as the corresponding multi-task parameter optimizer and multi-task evaluation mechanism. The multi-task parameter optimizer can adopt a general model parameter optimizer (such as the Adam optimizer, SGD optimizer, etc.) or can be customized based on actual training requirements. This embodiment of the invention does not specifically limit the multi-task parameter optimizer. The multi-task evaluation mechanism is composed of the single-task evaluation mechanisms of all downstream tasks. Each single-task evaluation mechanism can adopt a general model evaluation function or method (such as the MAE function, MSE function, RMSE function, F1 score evaluation method, etc.) or can be customized based on actual training evaluation requirements. This embodiment of the invention does not specifically limit each single-task evaluation mechanism. The combination method of all single-task evaluation results can be summation or weighted summation, or can be customized based on actual evaluation requirements. This embodiment of the invention does not specifically limit the combination method of all single-task evaluation results.

[0215] It should also be noted that, in this embodiment of the invention, the multi-task training process is divided into a preprocessing stage, a training stage, and an evaluation stage, specifically as follows:

[0216] 1) Preprocessing stage:

[0217] The multi-task dataset is randomly split into two subsets according to a preset dataset splitting ratio (e.g., 8:2): a third training set and a third evaluation set.

[0218] 2) Training phase:

[0219] The multi-task training framework is trained based on the third training set, the multi-task loss function, and the multi-task parameter optimizer. Specifically, the model training data of each third data record in the third training set is input into the multi-task training framework to obtain the corresponding output data, which is recorded as the corresponding fifth prediction data. Each fifth prediction data and its corresponding model label data are combined to form a corresponding fifth prediction-label pair. All the obtained fifth prediction-label pairs are input into the multi-task loss function to calculate the corresponding third loss value. The current third loss value is then checked to see if it meets the expected loss value range. If it does, the process proceeds to the evaluation stage. If it does not meet the requirement, the multi-task parameter optimizer is used to modulate the model parameters of all downstream task models in the multi-task training framework, as well as the neural structure searcher and neural network module set of the multimodal medical pre-trained model, in the direction of minimizing the current loss function. At the end of this modulation round, the above training steps are repeated to re-input each third training data into the multi-task training framework to obtain a new batch of fifth prediction-label pairs and a new third loss value.

[0220] 3) Evaluation phase:

[0221] The training effect of the multi-task training framework is evaluated based on the third evaluation set and the multi-task evaluation mechanism. Specifically, the model training data of each third data record in the third evaluation set is input into the multi-task training framework to obtain the corresponding output data, which is recorded as the corresponding sixth prediction data. Each sixth prediction data and its corresponding model label data are combined to form a corresponding sixth prediction-label pair. The corresponding third evaluation data is obtained by evaluating all the obtained sixth prediction-label pairs based on the multi-task evaluation mechanism. It is then identified whether the current third evaluation data meets the expected evaluation value range. If it does not meet the range, the training phase is returned to continue training. If it does meet the range, the training of the multi-task training framework is confirmed to be completed, which means the two-stage training is confirmed to be completed.

[0222] Step 5: After the two-stage training is completed, freeze the model parameters of the multimodal medical pre-trained model and confirm that the model training is finished.

[0223] Here, once the model parameters of the multimodal medical pre-trained model are frozen, the model can be used as a multimodal feature encoding sharing platform for all downstream task models.

[0224] Figure 7 This is a module structure diagram of a training device for a multimodal medical pre-training model provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 7 As shown, the device includes: a multimodal model building module 201, a one-stage model training module 202, and a two-stage model training module 203.

[0225] The multimodal model building module 201 is used to select a type of pre-trained large language model as a text pre-training model, select a type of visual pre-training model that can simultaneously support 2D / 3D image processing as an image pre-training model, select multiple types of basic neural network modules to form a neural network module set, and configure a corresponding neural structure searcher for the neural network module set; and build a multimodal large model based on the text pre-training model, image pre-training model, neural structure searcher and neural network module set as the corresponding multimodal medical pre-training model.

[0226] The first-stage model training module 202 is used to configure a first-stage text task model set for the text pre-trained model and a first-stage visual task model set for the image pre-trained model. The first-stage text task model set includes a masked language modeling task head, a masked knowledge element prediction task head, an unstructured text entity recognition task head, a structured text entity recognition task head, and a text generation task head. The first-stage visual task model set includes a waveform classification task head, a waveform disease prediction task head, an image object detection task head, and an image semantic segmentation task head.

[0227] The first-stage model training module 202 is also used to perform a first-stage text model training on the text pre-trained model based on the first-stage text task model set; and to perform a first-stage image model training on the image pre-trained model based on the first-stage visual task model set; and to freeze the model parameters of the text pre-trained model and the image pre-trained model at the end of the first-stage text and image model training.

[0228] The two-stage model training module 203 is used to configure one or more downstream task models for the multimodal medical pre-trained model; and to perform two-stage multimodal training on the neural structure searcher and neural network module set of the multimodal medical pre-trained model based on all downstream task models.

[0229] The two-stage model training module 203 is also used to freeze the model parameters of the multimodal medical pre-trained model and confirm the end of model training after the two-stage training is completed.

[0230] The training device for a multimodal medical pre-training model provided in this embodiment of the invention can execute the method steps in the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0231] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the multimodal model building module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0232] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SoC).

[0233] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0234] Figure 8 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 8 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0235] exist Figure 8The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include Non-Volatile Memory, such as at least one disk storage device.

[0236] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0237] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0238] This invention provides a training method, apparatus, electronic device, and computer-readable storage medium for a multimodal medical pre-training model. As described above, this invention selects a type of pre-trained large language model as the text pre-training model and a type of visual pre-training model capable of simultaneously supporting 2D / 3D image processing as the image pre-training model. It also selects multiple types of basic neural network modules to form a neural network module set and configures a corresponding neural architecture searcher for the neural network module set. A multimodal medical pre-training model is constructed based on the text pre-training model, image pre-training model, neural architecture searcher, and neural network module set. A corresponding first-stage text / visual task model set is configured for the text / image pre-training model. The text / image pre-training model is then trained separately based on the first-stage text / visual task model set. After the first-stage training is completed, multiple downstream task models are configured to form a multi-task training framework for the multimodal medical pre-training model. The second-stage training of the neural architecture searcher and neural network module set is completed by training the multi-task training framework. The embodiments of this invention address several shortcomings of traditional single-modal analysis frameworks: 1) It provides a multimodal medical pre-training model capable of multimodal feature fusion of medical text, 2D medical waveforms, and 3D medical images; 2) The two-stage training method based on the embodiments of this invention not only reduces the overall training difficulty but also improves the model's adaptability to medical scenarios; 3) The multimodal medical pre-training model based on the embodiments of this invention provides an encoding sharing platform that can simultaneously address multiple downstream tasks, improving the convenience of multi-task fusion and the overall task prediction accuracy.

[0239] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0240] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method for a multimodal medical pre-training model, characterized in that, The method includes: A pre-trained large language model is selected as the text pre-training model, a visual pre-training model that can simultaneously support 2D / 3D image processing is selected as the image pre-training model, and multiple basic neural network modules are selected to form a neural network module set. A corresponding neural architecture searcher is configured for the neural network module set. Based on the text pre-training model, the image pre-training model, the neural architecture searcher, and the neural network module set, a multimodal large model is constructed as the corresponding multimodal medical pre-training model. Configure a one-stage text task model set for the text pre-trained model; and configure a one-stage visual task model set for the image pre-trained model; the one-stage text task model set includes a masked language modeling task head, a masked knowledge element prediction task head, an unstructured text entity recognition task head, a structured text entity recognition task head, and a text generation task head; the one-stage visual task model set includes a waveform classification task head, a waveform disease prediction task head, an image object detection task head, and an image semantic segmentation task head; The text pre-trained model is trained for one stage based on the one-stage text task model set; and the image pre-trained model is trained for one stage based on the one-stage visual task model set; and the model parameters of the text pre-trained model and the image pre-trained model are frozen at the end of the one-stage text and image model training. Configure one or more downstream task models for the multimodal medical pre-training model; and perform two-stage multimodal training on the neural structure searcher and the neural network module set of the multimodal medical pre-training model based on all the downstream task models; After the two-stage training is completed, the model parameters of the multimodal medical pre-trained model are frozen and the model training is confirmed to be completed. In the initial stage of the first-stage text model training: the pre-trained text model is connected to each task head of the first-stage text task model set to form a corresponding first training model. Through big data collection, based on the model function and corresponding input / output data format of the pre-trained text model and the model function and corresponding input / output data format of each task head of the first-stage text task model set, a corresponding model dataset is constructed for each first training model, denoted as the corresponding first task head dataset. A matching model training loss function is designed for each first training model, denoted as the corresponding first task head loss function. The two-stage multimodal training of the neural architecture searcher and neural network module set of the multimodal medical pre-training model based on all the downstream task models specifically includes: connecting the output of the multimodal medical pre-training model with the inputs of all the downstream task models to form a corresponding multi-task training framework; constructing a corresponding model dataset for the multi-task training framework, denoted as the corresponding multi-task dataset, based on the model functions and corresponding input / output data formats of the multimodal medical pre-training model and all the downstream task models through big data collection; designing a matching multi-task loss function for the multi-task training framework; and performing multi-task training on the multi-task training framework based on the multi-task dataset and the multi-task loss function, wherein only the model parameters of all the downstream task models and the model parameters of the neural architecture searcher and neural network module set of the multimodal medical pre-training model are modulated during the multi-task training process.

2. The training method for the multimodal medical pre-training model according to claim 1, characterized in that, The pre-trained large language model is a type of large language model that is implemented based on the Transformer architecture and has been pre-trained. The pre-trained large language model includes at least the BERT series models. The visual pre-trained model is a large visual model compatible with both 2D and 3D modalities and has been pre-trained. The visual pre-trained model includes at least the MultiResViT model, UniFormer model, Hybrid CNN-Transformer model, nnU-Net model, Uni3D model, Uni-MAE model, and MedFMC model. The basic neural network module includes at least a feature splicing module, a weighted feature splicing module, a residual network module, a cross-attention encoding module, a graph neural network module, a 1D convolutional neural network module, a 2D convolutional neural network module, a 3D convolutional neural network module, a residual connection module, and a Hadamard product module.

3. The training method for the multimodal medical pre-training model according to claim 1, characterized in that, The multimodal medical pre-trained model is used to perform cross-modal feature encoding processing based on the model input text and model input image, and output the corresponding multimodal feature tensor; The input text for the model is natural language text or structured text in the medical field; The model input image is a 2D waveform or 3D image in the medical field; the 2D waveform image type includes at least electrocardiogram, electroencephalogram, electromyography, electrooculogram, blood pressure waveform, respiratory waveform, and blood oxygen saturation waveform; the 3D image image type includes at least computed tomography (CT) scan, magnetic resonance imaging (MRI), three-dimensional ultrasound imaging, positron emission tomography (PET), single-photon emission computed tomography (SPECT), optical coherence tomography (OCT), and cone-beam computed tomography (CBCT). The first model input terminal of the multimodal medical pre-trained model is used to receive the model input text, the second model input terminal is used to receive the model input image, and the model output terminal is used to output the corresponding multimodal feature tensor. The multimodal medical pre-training model includes the text pre-training model, the image pre-training model, the neural structure searcher, and the neural network module set; the neural network module set includes multiple basic neural network modules. The input of the text pre-trained model is connected to the input of the first model, and the output is connected to the first input of the neural structure searcher. The input of the image pre-training model is connected to the input of the second model, and the output is connected to the second input of the neural structure searcher; the output of the neural structure searcher is connected to the output of the model, and the neural structure searcher is also connected to the neural network module set. The text pre-training model is used to encode the text input to the model to obtain the corresponding text feature tensor, which is then sent to the neural structure searcher. The image pre-training model is used to perform 2D / 3D adaptive feature encoding on the input image of the model to obtain the corresponding image feature tensor and send it to the neural structure searcher; The neural architecture searcher is used to construct an initial network structure according to the neural network module set according to a preset initial network construction rule before the second-stage training; and to assign a learnable training weight parameter to all candidate modules in each i-th layer candidate module group in the initial network structure; the initial network structure consists of N i-th input layers, N i-th layer candidate module groups, and an output layer, 1 ≤ index i ≤ N, where N is a preset positive integer; each i-th layer candidate module group consists of one or more candidate modules, and each candidate module is a basic neural network module; the sum of all training weight parameters corresponding to each i-th layer candidate module group is 1; the input and output data formats of all candidate modules in each i-th layer candidate module group are consistent; In the initial network structure, the input of the first input layer is connected to the input of the neural structure searcher, and its output is connected to the input of each candidate module in the first layer candidate module group. The output of each candidate module in the first layer candidate module group is connected to one input of the second input layer; and so on. The output of each candidate module in the (N-1)th layer candidate module group is connected to one input of the Nth input layer, the output of the Nth input layer is connected to the input of each candidate module in the Nth layer candidate module group, and the output of each candidate module in the Nth layer candidate module group is connected to one input of the output layer; the output of the output layer is connected to the output of the neural structure searcher. In the initial network structure, the first input layer is used to receive the text feature tensor and the image feature tensor to form a corresponding first tensor group, and send the first tensor group to each of the candidate modules of the first layer candidate module; each i-th input layer from the second input layer to the Nth input layer is used to perform a weighted summation calculation on the output tensors of all the candidate modules of the (i-1)th layer candidate module group according to the training weight parameters corresponding to all the candidate modules of the corresponding (i-1)th layer candidate module group, and use the calculation result as the corresponding (i-1)th layer output tensor, and send the (i-1)th layer output tensor to each of the candidate modules of the corresponding i-th layer candidate module group; the output layer is used to perform a weighted summation calculation on the output tensors of all the candidate modules of the Nth layer candidate module group according to the training weight parameters corresponding to all the candidate modules of the Nth layer candidate module group to obtain the corresponding Nth layer output tensor, and use the Nth layer output tensor as the corresponding multimodal feature tensor and output it; In the initial network structure, the training weight parameters of all candidate modules are updated through training during the two-stage training process; The neural architecture searcher is also used to prune the initial network structure at the end of the two-stage training to obtain the corresponding preferred network structure and solidify it. The neural architecture searcher is also used to perform feature fusion processing on the input text feature tensor and the image feature tensor based only on the preferred network structure after the preferred network structure is frozen, and output the corresponding multimodal feature tensor.

4. The training method for the multimodal medical pre-training model according to claim 3, characterized in that, The task head input of the masked language modeling task head includes medical masked corpus text and the text feature tensor corresponding to the current corpus text. The task head output is a masked word segmentation sequence. The medical masked corpus text is a sentence or a paragraph of natural language text in the medical field. One or more word segments in the medical masked corpus text are masked by preset mask markers. The word segmentation index of each mask marker in the medical masked corpus text is recorded as the corresponding masked word segmentation index. The masked word segmentation sequence is formed by sorting one or more masked predicted texts, and the masked predicted texts correspond one-to-one with the masked tags, being the word segmentation predicted texts of the corresponding masked tags; The mask language modeling task head is used to predict the word segmentation text corresponding to all the mask tags in the medical mask corpus text based on the text feature tensor and output the corresponding mask word segmentation sequence; The masked knowledge element prediction task head incorporates a general or specialty knowledge graph in the medical field, denoted as the medical knowledge graph. The task head input includes a medical knowledge triple and the text feature tensor corresponding to the current triple text; the task head output is a triple element. The medical knowledge graph includes a first node set and a first directed edge set. The first node set includes multiple first nodes, and the first directed edge set includes multiple first directed edges. Each first node is a medical subject noun in the current general or specialty medical field. Each first directed edge connects two first nodes, reflecting the association between the two medical subject nouns corresponding to the two first nodes. The medical knowledge triple consists of three text elements: a first medical subject noun, a first subject relation, and a second medical subject noun. The first subject relation is the association between the first and second medical subject nouns. One text element in the medical knowledge triple is masked by the mask marker. The triple element is the predicted text corresponding to the text element masked by the mask marker in the medical knowledge triple. The masked knowledge element prediction task head is used to predict the element text of the medical knowledge triplet that is masked by the mask mark based on the text feature tensor and the medical knowledge graph, and output the prediction result as the corresponding triplet element. The task head input of the unstructured text entity recognition task head is a text feature tensor, and the task head output is a sequence of medical knowledge triples; the current text feature tensor is a feature tensor of an unstructured medical text; the current unstructured medical text is a piece of natural language text in the medical field; the sequence of medical knowledge triples is formed by sorting multiple medical knowledge triples in the current unstructured medical text. The unstructured text entity recognition task head and the masked knowledge element prediction task head share the medical knowledge graph; the unstructured text entity recognition task head is implemented based on an entity recognition model constructed by a deep learning model; the unstructured text entity recognition task head is used to identify knowledge triples based on the text feature tensor and the medical knowledge graph and output the corresponding medical knowledge triple sequence; The task head input of the structured text entity recognition task head is a text feature tensor, and the task head output is a sequence of medical knowledge triples; the current text feature tensor is a feature tensor of a structured medical text; the current structured medical text is a structured text in the medical field; The structured text entity recognition task head and the masked knowledge element prediction task head share the medical knowledge graph; the structured text entity recognition task head is implemented based on another type of entity recognition model constructed by a deep learning model; the structured text entity recognition task head is used to identify knowledge triples based on the text feature tensor and the medical knowledge graph and output the corresponding medical knowledge triple sequence; The text generation task head takes a text feature tensor as input and outputs a medical treatment plan or health management plan text. The current text feature tensor is a feature tensor for generating medical text instructions. The generated medical text instructions are formatted natural language prompts. Configurable parameters for the generated medical text instructions include a status description text parameter and a generated object type parameter. The status description text parameter is either unstructured or structured medical text. The generated object type parameter includes at least treatment plans and health management plans. The medical text generation instruction is used to prompt the model to generate a solution text corresponding to the generated object type parameter based on the state description text parameter and output it. The text generation task head and the mask knowledge element prediction task head share the medical knowledge graph; the text generation task head is implemented based on another type of text generator model constructed by a deep learning model; the text generation task head is used to perform solution text generation processing based on the text feature tensor and output the corresponding treatment solution text or health management solution text.

5. The training method for the multimodal medical pre-training model according to claim 3, characterized in that, The task head input of the waveform classification task head is the image feature tensor corresponding to the 2D waveform, and the task head output is a 2D waveform type; the 2D waveform type includes at least electrocardiogram, electroencephalogram, electromyography, electrooculogram, blood pressure waveform, respiratory waveform, and blood oxygen saturation waveform; The waveform classification task head is based on a 2D visual classification model constructed by a deep learning model; the waveform classification task head is used to perform waveform classification and recognition processing based on the image feature tensor input to the task head and output the corresponding 2D waveform type; The task head input of the waveform disease prediction task head is an image feature tensor corresponding to the 2D waveform image, and the task head output is a predicted disease type corresponding to the current 2D waveform image. When the image type of the 2D waveform image is an electrocardiogram (ECG), the corresponding predicted disease type includes at least atrial fibrillation, premature beats, tachycardia, bradycardia, myocardial infarction, myocardial ischemia, and cardiac conduction block. When the image type of the 2D waveform image is an electroencephalogram (EEG), the corresponding predicted disease type includes at least epilepsy, cerebral hemorrhage, cerebral thrombosis, cerebral infarction, encephalitis, brain tumor, traumatic brain injury, schizophrenia, bipolar disorder, mental abnormality, and poisoning. When the image type of the 2D waveform image is an electromyography (EMG), the corresponding predicted disease type includes at least myasthenia gravis, muscular dystrophy, polymyositis, dermatomyositis, amyotrophic lateral sclerosis (ALS), and Guillain-Barré syndrome. The predicted disease types include: polyneuropathy, oculomotor nerve injury, trochlear nerve injury, and facial nerve inflammation; when the 2D waveform image type is an electrooculogram, the corresponding predicted disease types include at least toxic retinopathy, vitrectomyelitis, retinal detachment, retinitis pigmentosa, diabetic retinopathy, and recurrent iridocyclitis; when the 2D waveform image type is a blood pressure waveform, the corresponding predicted disease types include at least hypertension, hypotension, white coat hypertension, and circadian rhythm abnormalities; when the 2D waveform image type is a respiratory waveform, the corresponding predicted disease types include at least airway obstruction, acute respiratory distress syndrome, and obstructive respiratory diseases; when the 2D waveform image type is a blood oxygen saturation waveform, the corresponding predicted disease types include at least respiratory failure, cardiovascular diseases, and chronic obstructive pulmonary disease. The waveform disease prediction task head is based on another type of 2D visual classification model constructed by a deep learning model; the waveform disease prediction task head is used to identify the waveform type according to the image feature tensor input to the task head, and predict the corresponding disease type according to the identified waveform type and the image feature tensor, and output the corresponding predicted disease type. The task head input of the image target detection task head is the image feature tensor corresponding to the 3D image, and the task head output is a set of target detection boxes; the target detection box set includes multiple target detection boxes; each target detection box includes a detection box identifier, detection box 3D size, detection box center coordinates, detection box orientation, and detection box type; the detection box type is a type of anatomical structure in the medical field. The image target detection task head is based on a 3D target recognition model constructed by a deep learning model; the image target detection task head is used to perform 3D anatomical structure target recognition processing according to the image feature tensor input to the task head and output the corresponding target detection box set; The task head of the image semantic segmentation task is input to an image feature tensor corresponding to a 3D image, and outputs a 3D semantic segmentation map. The three-dimensional dimensions of the image feature tensor are consistent with those of the 3D image. The three-dimensional dimensions of the 3D semantic segmentation map are consistent with those of the image feature tensor. The voxel-level semantics of the 3D semantic segmentation map include background semantics and foreground semantics. The foreground semantics include one or more types of anatomical structure semantics. The anatomical structure semantics correspond one-to-one with the anatomical structure type. The image semantic segmentation task head is implemented based on a 3D semantic segmentation model constructed by a deep learning model; the image semantic segmentation task head is used to perform 3D anatomical structure semantic segmentation processing based on the image feature tensor input to the task head and output the corresponding 3D semantic segmentation map.

6. The training method for the multimodal medical pre-training model according to claim 4, characterized in that, The step of performing a one-stage text model training on the pre-trained text model based on the one-stage text task model set specifically includes: The text pre-trained model is connected to each task head of the first-stage text task model set to form a corresponding first training model; and through big data collection, a corresponding model dataset is constructed for each first training model based on the model function and corresponding input / output data format of the text pre-trained model and the model function and corresponding input / output data format of each task head of the first-stage text task model set, denoted as the corresponding first task head dataset; and a matching model training loss function is designed for each first training model, denoted as the corresponding first task head loss function. The first step of text model training is performed on the text pre-training model based on the first task head dataset corresponding to the mask language modeling task head and the first task head loss function; After the first step of text model training is completed, the second step of text model training is performed on the text pre-trained model based on the first task head dataset corresponding to the mask knowledge element prediction task head and the first task head loss function. After the second step of text model training is completed, the text pre-trained model is trained in the third step based on the first task head dataset corresponding to the unstructured text entity recognition task head and the first task head loss function. After the third step of text model training is completed, the text pre-trained model is trained in the fourth step based on the first task head dataset corresponding to the structured text entity recognition task head and the first task head loss function. After the fourth step of text model training is completed, the fifth step of text model training is performed on the text pre-trained model based on the first task head dataset and the first task head loss function corresponding to the text generation task head. After the fifth step of text model training is completed, the first phase of text model training is confirmed to be finished.

7. The training method for the multimodal medical pre-training model according to claim 5, characterized in that, The step of performing a one-stage image model training on the image pre-trained model based on the one-stage visual task model set specifically includes: The image pre-trained model is connected to each task head of the first-stage visual task model set to form a corresponding second training model; and through big data collection, a corresponding model dataset is constructed for each second training model based on the model function and corresponding input / output data format of the image pre-trained model and the model function and corresponding input / output data format of each task head of the first-stage visual task model set, denoted as the corresponding second task head dataset; and a matching model training loss function is designed for each second training model, denoted as the corresponding second task head loss function. The image pre-training model is trained in the first step based on the second task head dataset corresponding to the waveform classification task head and the second task head loss function. After the first step of image model training is completed, the second step of image model training is performed on the image pre-training model based on the second task head dataset corresponding to the waveform disease prediction task head and the second task head loss function. After the second step of image model training is completed, the third step of image model training is performed on the image pre-trained model based on the second task head dataset and the second task head loss function corresponding to the image target detection task head. After the third step of image model training is completed, the fourth step of image model training is performed on the image pre-trained model based on the second task head dataset corresponding to the image semantic segmentation task head and the second task head loss function. After the fourth step of image model training is completed, the first stage of image model training is confirmed to be finished.

8. An apparatus for performing a training method for a multimodal medical pre-training model according to any one of claims 1-7, characterized in that, The device includes: a multimodal model construction module, a one-stage model training module, and a two-stage model training module; The multimodal model construction module is used to select a type of pre-trained large language model as a text pre-training model, select a type of visual pre-training model that can simultaneously support 2D / 3D image processing as an image pre-training model, select multiple types of basic neural network modules to form a neural network module set, and configure a corresponding neural structure searcher for the neural network module set; and construct a multimodal large model based on the text pre-training model, the image pre-training model, the neural structure searcher, and the neural network module set as the corresponding multimodal medical pre-training model; The first-stage model training module is used to configure a first-stage text task model set for the text pre-trained model and a first-stage visual task model set for the image pre-trained model. The first-stage text task model set includes a masked language modeling task head, a masked knowledge element prediction task head, an unstructured text entity recognition task head, a structured text entity recognition task head, and a text generation task head. The first-stage visual task model set includes a waveform classification task head, a waveform disease prediction task head, an image object detection task head, and an image semantic segmentation task head. The first-stage model training module is also used to perform a first-stage text model training on the text pre-trained model based on the first-stage text task model set; and to perform a first-stage image model training on the image pre-trained model based on the first-stage visual task model set; and to freeze the model parameters of the text pre-trained model and the image pre-trained model at the end of the first-stage text and image model training. The two-stage model training module is used to configure one or more downstream task models for the multimodal medical pre-training model; and to perform two-stage multimodal training on the neural structure searcher and the neural network module set of the multimodal medical pre-training model based on all the downstream task models. The two-stage model training module is also used to freeze the model parameters of the multimodal medical pre-trained model and confirm the end of model training after the two-stage training is completed.

9. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-7; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-7.

Images