Training method, system and equipment of medical multi-mode base model and storage medium

Abstract

The invention discloses a medical multi-mode base model training method, system and device and a storage medium, and relates to the technical field of model training, and the method comprises the steps: constructing a medical multi-mode base model; each data processing branch is trained; and after training of each data processing branch is completed, training the multi-task decoder by using output data of each data processing branch. On one hand, based on the model migration capability, a small amount of multi-modal data can be utilized to better train a multi-task decoder, and the obtained trained medical multi-modal base model can support input of the multi-modal data; and on the other hand, by performing decoupling training on the data processing branch and the multi-task decoder, the interdependence between the data processing branch and the multi-task decoder can be reduced, so that the stability, generalization ability and prediction accuracy of the trained medical multi-modal base model are improved.

Description

Background Art

[0002] At present, trained network models often analyze single-modal data. If multimodal data is to be analyzed, multimodal fusion technology needs to be applied. However, when applying multimodal fusion technology, it is necessary to design a multimodal fusion method for each type of downstream task for each task, and then label, train, and fuse each modality of the collected data to generate a large number of small models for specific tasks. For some specific tasks, the amount of multimodal data is small, the collection is difficult, the model performance is difficult to improve, and it is difficult to generalize and apply. Summary of the invention

[0003] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a training method, system, device and storage medium for a medical multimodal base model, as follows: 1) In the first aspect, the present invention provides a training method for a medical multimodal base model, and the specific technical solution is as follows: Constructing a medical multimodal base model, the medical multimodal base model includes: a multi-task decoder and multiple data processing branches; each data processing branch is used to: process different types of input data and send the processing results to the multi-task decoder; the multi-task decoder is used to: predict the results according to the received processing results; Train each data processing branch; After the training of each data processing branch is completed, the multi-task decoder is trained using the output data of each data processing branch.

[0004] The beneficial effects of the training method of a medical multimodal base model provided by the present invention are as follows: On the one hand, based on the model transfer capability, a small amount of multimodal data can be used to better train the multi-task decoder, and the trained medical multimodal base model can support the input of multimodal data; on the other hand, by decoupling the data processing branch and the multi-task decoder, the mutual dependence between the data processing branch and the multi-task decoder can be reduced, thereby improving the stability, generalization ability and prediction accuracy of the trained medical multimodal base model.

[0005] Based on the above scheme, the training method of a medical multimodal base model of the present invention can also be improved as follows.

[0006] Furthermore, when the type of input data is task instructions and / or text data, the data processing branch corresponding to the input data of this type includes: the encoder used for the input data of this type, and the processing result of the data processing branch is the output of the encoder in the data processing branch; each of the remaining data processing branches includes an encoder and a cross-modal fusion model, the encoder processes the input data of the corresponding type, and the cross-modal fusion model processes the output of the encoder to obtain the processing result.

[0007] Furthermore, it also includes: When training each cross-modal fusion model, the trained encoder within the same data processing branch is optimized.

[0008] The beneficial effects of adopting the above further technical solution are as follows: the prediction accuracy of the trained medical multimodal base model can be further improved.

[0009] Furthermore, it also includes: The target input data is input into the trained medical multimodal base model to obtain the processing result corresponding to the target input data.

[0010] 2) In the second aspect, the present invention also provides a training system for a medical multimodal base model, and the specific technical solution is as follows: Includes model building module and model training module; The model building module is used to: build a medical multimodal base model, the medical multimodal base model includes: a multi-task decoder and multiple data processing branches, each data processing branch is used to: process different types of input data and send the processing results to the multi-task decoder, and the multi-task decoder is used to: predict the results according to the received processing results; The model training module is used to: after the training of each data processing branch is completed, the output data of each data processing branch is used to train the multi-task decoder.

[0011] Based on the above solution, the training system of the medical multimodal base model of the present invention can also be improved as follows.

[0012] Furthermore, when the type of input data is task instructions and / or text data, the data processing branch corresponding to the input data of this type includes: the encoder used for the input data of this type, and the processing result of the data processing branch is the output of the encoder in the data processing branch; each of the remaining data processing branches includes an encoder and a cross-modal fusion model, the encoder processes the input data of the corresponding type, and the cross-modal fusion model processes the output of the encoder to obtain the processing result.

[0013] Furthermore, a model optimization module is included, and the model optimization module is used to: When training each cross-modal fusion model, the trained encoder within the same data processing branch is optimized.

[0014] Furthermore, it also includes a prediction module, which is used to: input the target input data into the trained medical multimodal base model to obtain the processing results corresponding to the target input data.

[0015] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program, and the at least one computer program being loaded and executed by the processor so that the electronic device implements any one of the above-mentioned training methods for the medical multimodal base model.

[0016] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned training methods for the medical multimodal base model.

[0017] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention: Figure 1 One of the flowcharts of a method for training a medical multimodal base model according to an embodiment of the present invention; Figure 2 The second flowchart of a method for training a medical multimodal base model according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the network structure of the medical multimodal base model; Figure 4 A schematic diagram of the structure of a training system for a medical multimodal base model according to an embodiment of the present invention; Figure 5 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0020] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0021] like Figure 1 As shown, a training method for a medical multimodal base model according to an embodiment of the present invention comprises the following steps: S1. Construct a medical multimodal base model, which includes: a multi-task decoder and multiple data processing branches; each data processing branch is used to: process different types of input data and send the processing results to the multi-task decoder; the multi-task decoder is used to: predict the results according to the received processing results; S2, training each data processing branch; S3. After the training of each data processing branch is completed, the multi-task decoder is trained using the output data of each data processing branch.

[0022] On the one hand, based on the model transfer capability, a small amount of multimodal data can be used to better train the multi-task decoder, and the trained medical multimodal base model can support the input of multimodal data; on the other hand, by decoupling the data processing branch and the multi-task decoder, the mutual dependence between the data processing branch and the multi-task decoder can be reduced, thereby improving the stability, generalization ability and prediction accuracy of the trained medical multimodal base model.

[0023] Optionally, in the above technical scheme, when the type of input data is task instructions and / or text data, the data processing branch corresponding to the input data of this type includes: the encoder used for the input data of this type, the processing result of the data processing branch is the output of the encoder in the data processing branch; each of the remaining data processing branches includes an encoder and a cross-modal fusion model, the encoder processes the input data of the corresponding type, and the cross-modal fusion model processes the output of the encoder to obtain a processing result.

[0024] Optionally, in the above technical solution, it also includes: When training each cross-modal fusion model, the trained encoder in the same data processing branch is optimized, which can further improve the prediction accuracy of the trained medical multimodal base model.

[0025] Optionally, in the above technical solution, it also includes: S4. Input the target input data into the trained medical multimodal base model to obtain the processing result corresponding to the target input data.

[0026] The present invention constructs a medical multimodal base model, which can integrate and analyze medical clinical data, one-dimensional, two-dimensional, and three-dimensional multimodal medical data (such as one-dimensional electrocardiogram data, two-dimensional X-ray image data, and three-dimensional CT image data), and can also expand more data modalities. The model decouples training and migration capabilities, uses massive data to train each modality data representation, semantic alignment, and then uses a small amount of multimodal data training to solve the core problems of insufficient data and low generalization, and realizes accurate multimodal diagnosis prediction, report generation, lesion identification and other rich downstream tasks.

[0027] The invention is illustrated by the following examples.

[0028] Different types of input data in this embodiment include: text data, task instructions, one-dimensional signal data, two-dimensional image data and three-dimensional image data, etc., wherein the text data includes medical record data, laboratory test data, medical diagnosis data, etc., the one-dimensional signal data includes: continuously monitored vital signs data (body temperature, blood pressure, respiration, pulse), electrocardiogram signals, electroencephalogram signals and electromyography signals, etc., the two-dimensional image data includes: fundus images, X-ray images, and digital pathology images, etc., and the three-dimensional image data includes ultrasound images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) and CT combination (PET-CT) images, etc.

[0029] like Figure 2 As shown, this embodiment includes the following steps: S101, build a medical multimodal base model, such as Figure 3 As shown, the medical multimodal base model includes a multi-task decoder and n data processing branches, wherein the first data processing branch is used to process text data and task instructions, the first data processing branch includes an encoder, the encoder is specifically a text encoder, and is also the first encoder, the second data processing branch to the nth data processing branch each include an encoder and a cross-modal fusion model, specifically, the i-th data processing branch includes the i-th encoder and the i-1-th cross-modal fusion model, for example, the second data processing branch includes: the second encoder and the first cross-modal fusion model, the third data processing branch includes: the third encoder and the second cross-modal fusion model, the fourth data processing branch includes: the fourth encoder and the third cross-modal fusion model, until the n-th data processing branch includes: the n-th encoder and the n-1-th cross-modal fusion model, 2≤i≤n, i is a positive integer.

[0030] Among them, the output of the first encoder is the output result of the first data processing branch, and the output result can be used as the input of the multi-task decoder. For 2≤i≤n, the i-th encoder in the i-th data processing branch is used to receive the corresponding type of input data and obtain the corresponding output. For example, when the type of input data is one-dimensional signal data, the corresponding encoder is a one-dimensional encoder, when the input data type is two-dimensional image data, the corresponding encoder is a two-dimensional encoder, and when the input data type is three-dimensional image data, the corresponding encoder is a three-dimensional encoder. The output of the i-th encoder is the input of the i-1 cross-modal fusion model, and the output of the i-1 cross-modal fusion model is the input of the multi-task decoder.

[0031] S102, training each data processing branch, specifically including: S1020, training each encoder: A large amount of public data such as electrocardiogram (ECG), X-ray images, CT images, digital pathology images, and non-public data from hospitals (when the user of the present invention is a hospital, the medical data in the hospital can be used) is used as input data to train each encoder through a self-learning mechanism. During training, the input data does not need to be labeled, and the input data is self-learned and mined using technologies such as MAE, MIM, iBOT, and MIL. After training, each encoder learns the characteristics and attributes of the corresponding type of input data. When training the encoder, a large amount of unlabeled data is used to solve the core problem of small multimodal data volume and difficult collection.

[0032] S1021. Train each cross-modal fusion model: When training the i-1th cross-modal fusion model, the corresponding data is obtained in advance (when the user of the present invention is a hospital, the data in the hospital can be used), and combined with the output of the i-th encoder, the i-1th cross-modal fusion model is trained. For example, when training the third cross-modal fusion model, a large amount of X-ray image-report pair / electrocardiogram-report pair data is obtained in advance, and the output of the X-ray image / electrocardiogram data after the fourth encoder operation is used, combined with text data such as reports, to train the third cross-modal fusion model. This step requires training each cross-modal fusion model separately to obtain n-1 trained cross-modal fusion models.

[0033] When training each cross-modal fusion model, we need to mine the high-dimensional semantic features of different modal data such as X-ray, ECG, and CT, align them with clinical semantics, and integrate them for application. This also solves the core problem of small multimodal data volume and difficulty in collection.

[0034] Among them, the cross-modal fusion model can be a Q-Former model, a BLIP model or a CLIP model.

[0035] Among them, when training each cross-modal fusion model, the trained encoder in the same data processing branch is optimized using the data in the training process of each cross-modal fusion model, which can be specifically achieved in the following way: When training the i-1th cross-modal fusion model, the output result of the i-1th cross-modal fusion model at each iteration is converted into a vector form to obtain a first vector, and the output result of the i-1th cross-modal fusion model is manually corrected, or the output result of the i-1th cross-modal fusion model is compared and corrected by the cross-modal corpus pairs of training data (such as image-text pairs), and the corrected output result of the i-1th cross-modal fusion model is converted into a vector form to obtain a second vector, and the similarity between the first vector and the second vector is calculated, which is recorded as the first similarity. The i-th trained encoder is optimized using the first similarity and the first cross entropy loss obtained each time.

[0036] The i-th trained encoder is optimized using the first similarity and the first cross entropy loss obtained each time, including: When training each encoder, a cross entropy loss function can be used, and the product of the first cross entropy loss and the first difference in each iteration is used as the loss of the i-th trained encoder in the next iteration to optimize the i-th trained encoder, where the first difference = 1-first similarity. The higher the similarity between the output result of the i-1 cross-modal fusion model and the adjusted output result, the more accurate the output result of the i-1 cross-modal fusion model. Since the output of the i-th trained encoder is the input of the i-1 cross-modal fusion model, the output of the i-th trained encoder will affect the output result of the i-1 cross-modal fusion model. Therefore, the first difference is used as the weight of the first cross entropy loss, the loss participating in the next iteration is calculated, and the i-th trained encoder is trained again to optimize the i-th trained encoder.

[0037] S103, training the multi-task decoder, specifically: After training each encoder and each cross-modal fusion model, different instruction data sets are constructed for each downstream task application according to the output data of each data processing branch. Multimodal clinical data and corresponding labels are collected, and instructions and data labels are fused to construct a multi-task multimodal VQA corpus. Based on the multi-task multimodal VQA corpus and the multi-task training mechanism, the multi-task decoder is trained.

[0038] When training the multi-task decoder, each encoder and each cross-modal fusion model is optimized in the following ways: When training the multi-task decoder, the output result of the multi-task decoder at each iteration is converted into a vector form to obtain a third vector. Data comparison is performed manually or through a multi-task multimodal VQA corpus. The output result of the multi-task decoder at each iteration is corrected and converted into a vector form to obtain a fourth vector. The similarity between the third vector and the fourth vector is calculated, recorded as the second similarity, and the second similarity obtained each time is used for each encoder and each cross-modal fusion model.

[0039] Among them, each encoder and each cross-modal fusion model are optimized using the second similarity and the second cross entropy loss obtained each time, including: When training each cross-modal fusion model, the cross-entropy loss function can also be used. The product of the second cross-entropy loss and the second difference obtained in each iteration is used as the cross-entropy loss of each encoder in the next iteration and as the loss of each cross-modal fusion model in the next iteration. This can achieve re-optimization of the optimized and trained encoder and optimization of each trained cross-modal fusion model, where the second difference = 1-second similarity.

[0040] The higher the similarity between the output result of the multi-task decoder and the adjusted output result, the more accurate the output result of the multi-task decoder is. Since the output of the i-th trained encoder is the input of the i-1-th cross-modal fusion model of the cross-modal fusion model, and the output of the first encoder and all cross-modal fusion models is the input of the multi-task decoder, the accuracy of the output of each trained encoder and the output of each cross-modal fusion model will affect the accuracy of the output of the multi-task decoder. Using the second difference as the weight of the second cross entropy loss, the loss of each encoder participating in the next iteration and the loss of each cross-modal fusion model participating in the next iteration are calculated, and each trained encoder is re-trained to achieve re-optimization of each trained encoder and optimization of each trained cross-modal fusion model.

[0041] During training, multiple labels are set, for example, labels associated with disease prediction, labels associated with key information extraction, etc., so that the trained medical multimodal base model can perform multimodal diagnosis prediction, data analysis, report generation and other functions.

[0042] S104: Input the target input data into the trained medical multimodal base model to obtain a processing result corresponding to the target input data.

[0043] Among them, the trained medical multimodal base model can be used to perform multimodal diagnosis prediction, the trained medical multimodal base model can also be used to perform data analysis and generate corresponding analysis reports, and the trained medical multimodal base model can be used to extract key indicators from the input data and automatically fill them into the table, specifically: 1) For example, the clinical examination results, CT images, and pathological examination results of patients undergoing liver cancer surgery are input into a trained medical multimodal base model to predict the risk of recurrence of liver cancer patients one year after surgery.

[0044] 2) For example, an X-ray image is input into a trained medical multimodal base model to obtain an image report of the X-ray image, which may include: image findings of the X-ray image and image diagnosis results.

[0045] 3) For example, a CT image is input into a trained medical multimodal base model to obtain an image report, which includes the image findings of the CT image and the image diagnosis results.

[0046] 4) For example, convert medical teaching audio into text data, input the text data into a trained medical multimodal base model, and use the trained medical multimodal base model to analyze the text data. Specifically, identify multiple entities from the text (such as disease symptoms, disease treatment, and disease recovery, etc.), analyze the relationship between entities in the text data, and extract attribute information of each entity from the text data (for example, the attribute information of disease symptoms includes: specific description of the symptoms, such as the presence or absence of cough, etc.; the attribute information of disease treatment includes: the name of the medicine, etc.; the attribute information of disease recovery includes: rehabilitation plan, etc.), and then construct a knowledge graph of the teaching content of the medical teaching audio.

[0047] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0048] like Figure 4 As shown, a medical multimodal base model training system 200 according to an embodiment of the present invention includes a model building module 201 and a model training module 202; The model building module 201 is used to build a medical multimodal base model, which includes a multi-task decoder and multiple data processing branches, each of which is used to process different types of input data and send the processing results to the multi-task decoder, and the multi-task decoder is used to predict the results according to the received processing results. The model training module 202 is used to train the multi-task decoder using the output data of each data processing branch after the training of each data processing branch is completed.

[0049] Optionally, in the above technical scheme, when the type of input data is task instructions and / or text data, the data processing branch corresponding to the input data of this type includes: the encoder used for the input data of this type, the processing result of the data processing branch is the output of the encoder in the data processing branch; each of the remaining data processing branches includes an encoder and a cross-modal fusion model, the encoder processes the input data of the corresponding type, and the cross-modal fusion model processes the output of the encoder to obtain a processing result.

[0050] Optionally, in the above technical solution, a model optimization module is further included, and the model optimization module is used to: When training each cross-modal fusion model, the trained encoder within the same data processing branch is optimized.

[0051] Optionally, the above technical solution also includes a prediction module, which is used to: input target input data into a trained medical multimodal base model to obtain a processing result corresponding to the target input data.

[0052] It should be noted that the beneficial effects of the training system 200 for a medical multimodal base model provided in the above embodiment are the same as the beneficial effects of the training method for a medical multimodal base model, which will not be described in detail here. In addition, when the system provided in the above embodiment realizes its functions, it only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described in detail here.

[0053] Among them, the training system of the medical multimodal base model of the present invention can be a computer program (including program code) running in a computer device. For example, the training system of the medical multimodal base model of the present invention is an application software that can be used to execute the corresponding steps in the training method of the medical multimodal base model of the present invention.

[0054] In some embodiments, the training system of the medical multimodal base model of the present invention can be implemented by a combination of software and hardware. As an example, the training system of the medical multimodal base model of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the medical multimodal base model of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0055] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The name of a module does not limit the module itself in some cases.

[0056] An electronic device according to an embodiment of the present invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned training methods for a medical multimodal base model is implemented. That is to say, an electronic device according to an embodiment of the present invention may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the training method for a medical multimodal base model shown in any embodiment of the present invention by calling the computer program.

[0057] In an alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0058] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0059] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 In the figure, only one thick line is used to represent bus 4002, but this does not mean that there is only one bus or one type of bus.

[0060] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0061] The memory 4003 is used to store application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0062] Among them, the electronic device can also be a terminal device, and the terminal device can be any device that can install applications, including at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0063] It should be noted that Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0064] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, which, when executed by a processor, implements any of the above-mentioned training methods for a medical multimodal base model.

[0065] Optionally, the computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0066] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned training methods for a medical multimodal base model.

[0067] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0068] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0069] The computer-readable storage medium provided by the embodiment of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or component.

[0070] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0071] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

[0072] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0073] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0074] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A training method for a medical multimodal base model, characterized in that: include: Constructing a medical multimodal base model, the medical multimodal base model includes: a multi-task decoder and a plurality of data processing branches; each data processing branch is used to: process different types of input data and send the processing results to the multi-task decoder; the multi-task decoder is used to: perform result prediction according to the received processing results; Train each data processing branch; After the training of each data processing branch is completed, the multi-task decoder is trained using the output data of each data processing branch.

2. A training method for a medical multimodal base model according to claim 1, characterized in that: When the type of input data is task instructions and / or text data, the data processing branch corresponding to this type of input data includes: the encoder used for this type of input data, the processing result of this data processing branch is the output of the encoder in this data processing branch; each of the remaining data processing branches includes an encoder and a cross-modal fusion model, the encoder processes the input data of the corresponding type, and the cross-modal fusion model processes the output of the encoder to obtain the processing result.

3. The training method of a medical multimodal base model according to claim 2, characterized in that: Also includes: When training each cross-modal fusion model, the trained encoder within the same data processing branch is optimized.

4. A training method for a medical multimodal base model according to any one of claims 1 to 3, characterized in that: Also includes: The target input data is input into the trained medical multimodal base model to obtain the processing result corresponding to the target input data.

5. A medical multimodal base model training system, characterized in that: Includes model building module and model training module; The model building module is used to build a medical multimodal base model, the medical multimodal base model includes a multi-task decoder and a plurality of data processing branches, each data processing branch is used to process different types of input data and send the processing results to the multi-task decoder, and the multi-task decoder is used to predict the results according to the received processing results; The model training module is used to: after the training of each data processing branch is completed, use the output data of each data processing branch to train the multi-task decoder.

6. A medical multimodal base model training system according to claim 5, characterized in that: When the type of input data is task instructions and / or text data, the data processing branch corresponding to this type of input data includes: the encoder used for this type of input data, the processing result of this data processing branch is the output of the encoder in this data processing branch; each of the remaining data processing branches includes an encoder and a cross-modal fusion model, the encoder processes the input data of the corresponding type, and the cross-modal fusion model processes the output of the encoder to obtain the processing result.

7. A medical multimodal base model training system according to claim 6, characterized in that: It also includes a model optimization module, wherein the model optimization module is used to: When training each cross-modal fusion model, the trained encoder within the same data processing branch is optimized.

8. A medical multimodal base model training system according to any one of claims 5 to 7, characterized in that: It also includes a prediction module, which is used to: input target input data into a trained medical multimodal base model to obtain a processing result corresponding to the target input data.

9. An electronic device, characterized in that: It comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a training method for a medical multimodal base model as described in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the training method of a medical multimodal base model as described in any one of claims 1 to 4.

Images