Multi-modal large model fine tuning method and device, terminal equipment and storage medium
By fine-tuning the multimodal large model TIS-MOME through single-modal, bimodal, and multimodal training, the problem that the text-image bimodal large model cannot deeply analyze other modal data is solved, the adaptability of the model is improved, and the intelligent and precise diagnosis and treatment of integrated traditional Chinese and Western medicine is promoted.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2026-03-27
AI Technical Summary
Existing text-image bimodal large models cannot deeply analyze and apply data from other modalities, making it difficult to integrate the biomolecular mechanisms in the diagnosis and treatment of integrated traditional Chinese and Western medicine, and thus unable to provide sufficient technical support.
By fine-tuning the multimodal large model TIS-MOME through single-modal, bimodal, and multimodal training, training text, image, and biomolecular expert models using single-modal training data, and combining multimodal transformers for encoding, the adaptability of the model is gradually improved.
It improves the adaptability of multimodal large models and promotes the intelligent and precise development of integrated traditional Chinese and Western medicine diagnosis and treatment.
Smart Images
Figure CN121744173A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a method, apparatus, terminal device and storage medium for fine-tuning a multimodal large model. Background Technology
[0002] The rapid development of artificial intelligence technology and the increasing abundance of multimodal medical big data have made multimodal big data models show broad application prospects in the medical field.
[0003] Currently, most multimodal large-scale models used for diagnosis and treatment are text-image bimodal models based on natural language processing and computer vision technologies. These models cannot deeply analyze and apply data from other modalities and have weak adaptability to some complex medical scenarios. For example, in the field of integrated traditional Chinese and Western medicine, since traditional Chinese medicine and Western medicine belong to two different medical systems, each has its own advantages in disease diagnosis and treatment, but there are huge differences in treatment concepts, diagnostic methods, and clinical measures. It is difficult to integrate and deeply analyze the biomolecular mechanisms of disease diagnosis and treatment through such bimodal large-scale models. Therefore, existing text-image bimodal large-scale models cannot provide more technical support for integrated traditional Chinese and Western medicine clinical diagnosis and treatment. Summary of the Invention
[0004] In view of this, embodiments of this application provide a method, apparatus, terminal device, and storage medium for fine-tuning a multimodal large model, so as to improve the adaptability of the multimodal large model and provide more technical support for clinical diagnosis and treatment combining traditional Chinese and Western medicine.
[0005] The first aspect of this application provides a fine-tuning method for a multimodal large model, applied to TIS-MOME, the fine-tuning method comprising:
[0006] Acquire single-modal training data;
[0007] Based on the single-modal training data, the TIS-MOME is fine-tuned using single-modal training to obtain the first target TIS-MOME;
[0008] Acquire bimodal training data;
[0009] Based on the bimodal training data, the first target TIS-MOME is fine-tuned through bimodal training to obtain the second target TIS-MOME;
[0010] Acquire multimodal training data;
[0011] The second target TIS-MOME is fine-tuned using the multimodal training data to obtain and output the third target TIS-MOME.
[0012] A second aspect of this application provides a fine-tuning device for a multimodal large model, applied to TIS-MOME, the fine-tuning device comprising:
[0013] The first data acquisition module is used to acquire single-modal training data;
[0014] The first training module is used to perform single-modal training fine-tuning on the TIS-MOME based on the single-modal training data to obtain the first target TIS-MOME;
[0015] The second data acquisition module is used to acquire dual-modal training data;
[0016] The second training module is used to perform bimodal training fine-tuning on the first target TIS-MOME based on the bimodal training data to obtain the second target TIS-MOME;
[0017] The third data acquisition module is used to acquire multimodal training data;
[0018] The third training module is used to perform multimodal training fine-tuning on the second target TIS-MOME based on the multimodal training data, to obtain the third target TIS-MOME and output it.
[0019] A third aspect of this application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the fine-tuning method for a multimodal large model as described in the first aspect of this application.
[0020] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the fine-tuning method for a multimodal large model as described in the first aspect of this application.
[0021] The first aspect of this application provides a method for fine-tuning a multimodal large model, applied to TIS-MOME (Text-Image-Sequence Mixture-of-Modality-Experts). This method first acquires unimodal training data; then fine-tunes the TIS-MOME using the unimodal training data to obtain a first target TIS-MOME; next, it acquires bimodal training data; then fine-tunes the first target TIS-MOME using the bimodal training data to obtain a second target TIS-MOME; finally, it acquires multimodal training data; and then fine-tunes the second target TIS-MOME using the multimodal training data to obtain and output a third target TIS-MOME. This application, through fine-tuning the training of the multimodal large model TIS-MOME, can effectively improve the adaptability of the multimodal large model TIS-MOME, which is beneficial for promoting the intelligent and precise development of integrated traditional Chinese and Western medicine diagnosis and treatment.
[0022] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a schematic diagram of the structure of a multimodal large model provided in an embodiment of this application;
[0025] Figure 2 This is a schematic diagram of the first flowchart of the fine-tuning method for a multimodal large model provided in the embodiments of this application;
[0026] Figure 3 This is a second flowchart illustrating the fine-tuning method for a multimodal large model provided in this application embodiment;
[0027] Figure 4 This is a schematic diagram of the structure of the fine-tuning device for a multimodal large model provided in the embodiments of this application;
[0028] Figure 5 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation
[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0030] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0032] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. "A plurality of" means "two" or "more than two."
[0033] Currently, most multimodal large-scale models used for diagnosis and treatment are text-image bimodal models based on natural language processing and computer vision technologies. These models cannot deeply analyze and apply data from other modalities and have weak adaptability to some complex medical scenarios. For example, in the field of integrated traditional Chinese and Western medicine, since traditional Chinese medicine and Western medicine belong to two different medical systems, each has its own advantages in disease diagnosis and treatment, but there are huge differences in treatment concepts, diagnostic methods, and clinical measures. It is difficult to integrate and deeply analyze the biomolecular mechanisms of disease diagnosis and treatment through such bimodal large-scale models. Therefore, existing text-image bimodal large-scale models cannot provide more technical support for integrated traditional Chinese and Western medicine clinical diagnosis and treatment.
[0034] Therefore, this application proposes a multimodal large model TIS-MOME and provides a method for fine-tuning the multimodal large model. The method involves first acquiring single-modal training data; then fine-tuning the TIS-MOME using the single-modal training data to obtain a first target TIS-MOME; next, acquiring dual-modal training data; then fine-tuning the first target TIS-MOME using the dual-modal training data to obtain a second target TIS-MOME; finally, acquiring multimodal training data; and then fine-tuning the second target TIS-MOME using the multimodal training data to obtain a third target TIS-MOME, which is then output. This application, through fine-tuning the training of the multimodal large model TIS-MOME, can effectively improve the adaptability of the multimodal large model TIS-MOME, which is beneficial for promoting the intelligent and precise development of integrated traditional Chinese and Western medicine diagnosis and treatment.
[0035] Example 1
[0036] like Figure 1 As shown, the multimodal large model 1 proposed in this application is TIS-MOME. TIS-MOME is a hybrid model that includes a self-attention submodule (here, a multi-head self-attention module (MHSA), which allows the model to simultaneously pay attention to information from different locations and capture feature information related to its different modal input data through multiple attention heads) and four independent FFNs (Feed-Forward Neural Networks), specifically an image expert model (Vision-FFN), a text expert model (Language-FFN), a biomolecular expert model (Sequence-FFN), and a hybrid expert model (Vision-Language-Sequence–FFN, i.e., an image-text-biomolecular hybrid expert model).
[0037] In one embodiment, the image expert model is used for:
[0038] The input image data is divided into blocks to obtain multiple image blocks;
[0039] Flatten the multiple image blocks to obtain the corresponding multiple image block vectors;
[0040] Linear projection is performed on the plurality of image block vectors, and a first input representation of the image data is obtained through a first position encoding and a first type encoding;
[0041] The text expert model is used for:
[0042] The input text data is segmented to obtain multiple word roots;
[0043] Obtain multiple root vectors corresponding to the multiple root words;
[0044] Linear projection is performed on the plurality of word root vectors, and a second input representation of the text data is obtained through a second positional encoding and a second type encoding;
[0045] The biomolecular expert model is used for:
[0046] The input biomolecular data is segmented to obtain multiple fragments;
[0047] Obtain multiple segment vectors corresponding to the multiple segments;
[0048] Linear projection is performed on the multiple fragment vectors, and a third input representation of the biomolecular data is obtained through third position encoding and third type encoding;
[0049] The hybrid expert model is used to concatenate the first input representation obtained by the image expert model, the second input representation obtained by the text expert model, and the third input representation obtained by the biomolecular expert model to obtain the fourth input representation of the hybrid expert model.
[0050] In applications, if the input data is image data, the image expert model in TIS-MOME will first process the input image data v∈R. H×W×C Perform block processing, dividing into N∈HW / P 2 Image patches of equal size Where C represents the number of channels, N represents the length of a biomolecule, P represents the size of an image patch, and H and W represent the size of the input image; then each image patch is flattened into a corresponding vector (image patch vector), and linearly projected through a linear layer to map each image patch vector to a fixed-dimensional vector space, obtaining a fixed-dimensional vector PatchEmbedding; finally, by adding an [I_CLS] token (used to represent the entire image sequence) and adding a first positional encoding (1-dimensional positional encoding) V pos ∈R (N+1)×D (Used to obtain relative position information between image patches), by adding the first type of encoding V type ∈R D (Data used to distinguish the image from other modalities) ultimately yields the first input representation of the input image data, namely: in Projection as
[0051] In applications, if the input data is text data, the text expert model in TIS-MOME first performs word segmentation on the input text data, converting the input text into a series of tokenized words and obtaining corresponding word vectors. Then, a linear layer performs linear projection, mapping each word vector to a fixed-dimensional vector space to obtain a fixed-dimensional Word Embedding. Finally, by adding a [T_CLS] token (representing the start of the entire text sequence) and an end marker [T_SEP] (representing the end of the entire text sequence), and by adding a second positional encoding (1-dimensional positional encoding) T... pos ∈R (M+2)×D (Used to represent positional information between word roots), by adding a second type of encoding T type ∈R D (Data used to distinguish text from other modalities) ultimately yields the second input representation of the input text data, namely: in, Projection as M represents the length of the word root.
[0052] In applications, if the input data is biomolecular data, the biomolecular expert model in TIS-MOME first segments the input biomolecular data, transforming it into a series of tokenized segments and obtaining corresponding segment vectors. Then, a linear layer performs linear projection, mapping each segment vector to a fixed-dimensional vector space to obtain a fixed-dimensional vector sequence embedding. Finally, by adding a [S_CLS] token (representing the start of the entire biomolecular sequence) and an end marker [S_SEP] (representing the end of the entire biomolecular sequence), and by adding a third positional encoding (1-dimensional positional encoding) S... pos ∈R (M+2)×D (used to represent positional information between segments), by adding a third type of encoding S type ∈R D (Data used to distinguish biomolecules from other modalities) ultimately yields the third input representation of the input biomolecule data, namely: in, Projection as M represents the segment length.
[0053] In applications, if the input data is multi-modal (at least two modalities), the input representation of the corresponding modal data is obtained through the appropriate expert model. Then, the input representations of each modal expert model are concatenated through a hybrid expert model to obtain a fourth input representation of the multi-modal input data. For example, if the input data is image-text-biomolecule data, the first input representation of the input image will be obtained through an image expert model. A second input representation of the input text is obtained through a text expert model. Obtaining a third input representation of the input biomolecule through a biomolecular expert model. By concatenating the components, the fourth input representation is finally obtained, i.e.
[0054] In application, this application also designs a general multimodal transformer for TIS-MOME, namely the TIS-MOME transformer (Text-Image-Sequence Mixture-of-Modality-Experts), for encoding different modalities. The TIS-MOME transformer is based on the input vector H′ l The modalities are analyzed, and appropriate expert models are selected for processing to obtain the corresponding output vector H. l , where H′ l =MSA(LN(H l-1 ))+H l-1 H l =MOME-FFN(LN(H′) l ))+H′ l LN (Layer Normalization) represents layer normalization. For example, if the input contains vectors with multiple modalities, such as vectors of text, images, and biomolecules, TIS-MOME encodes the respective modal vectors using text expert models, image expert models, and biomolecule expert models on the bottom transformer layer, and then uses a hybrid expert model of text, images, and biomolecules on the top layer to capture more modal interactions.
[0055] like Figure 2 As shown, the fine-tuning method for a multimodal large model provided in this application includes the following steps S1 to S6:
[0056] Step S1: Obtain single-modal training data, then proceed to step S2.
[0057] In applications, multimodal data can be categorized into text, image, and biomolecular types. Image data includes various types of image data, including but not limited to X-rays, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound images, and traditional Chinese medicine tongue images. Biomolecular data includes but is not limited to genomic biomolecules, protein biomolecules, and ribonucleic acid (RNA) biomolecules. Text data can be some attributes, features, or interpretations of medical images or biomolecules, and is not limited here.
[0058] In applications, when acquiring single-modal training data, it can be obtained from some published multimodal datasets. For example, the Online Mendelian Inheritance in Man (OMIM) dataset, which contains medical image-biomolecule-text data pairs; open-source datasets released by conferences such as the International Symposium on Biomedical Imaging (ISBI), which contains medical image-text data pairs; the UniProtKB / Swiss-Prot dataset, which contains biomolecule-text data pairs; traditional Chinese medicine datasets and integrated traditional Chinese and Western medicine datasets, which contain information on traditional Chinese medicine prescriptions, medicinal materials, active ingredients, targets, and diseases; and other large numbers of single-modal medical datasets. These are just examples and are not limited to any particular dataset.
[0059] In one embodiment, the single-modal training data includes plain text data, plain image data, or plain biomolecular data.
[0060] In application, the TIS-MOME fine-tuning method provided in this application mainly includes three stages of training fine-tuning. In the first stage, the corresponding unimodal expert model is trained and fine-tuned based on unimodal training data. In the second stage, the corresponding modal expert model is trained and fine-tuned based on bimodal training data. In the third stage, all modal expert models are trained and fine-tuned based on trimodal training data. The unimodal training data can be pure text data, pure image data, or pure biomolecular data; no limitation is made here.
[0061] Step S2: Perform single-modal training fine-tuning on the TIS-MOME based on the single-modal training data to obtain the first target TIS-MOME, and proceed to step S3.
[0062] In applications, when performing unimodal training and fine-tuning of TIS-MOME, it can be done by training and fine-tuning the text expert model using only plain text data, the image expert model using only image data, or the biomolecular expert model using only biomolecular data. It is understood that, depending on actual needs, in step S2, the corresponding expert models can also be trained and fine-tuned sequentially based on the input unimodal training data. That is, this step can train only one, two, or three of the expert models. For example, first input plain text data to train and fine-tune the text expert model, then input plain image data to train and fine-tune the image expert model, and finally input plain biomolecular data to train and fine-tune the biomolecular expert model. This is just an example and not a limitation.
[0063] In one embodiment, step S2 specifically includes:
[0064] If the single-modal training data is the plain text data, then the text expert model in the TIS-MOME is fine-tuned through single-modal training, and the Self-Attention parameters shared by each expert model are updated.
[0065] If the single-modal training data is the pure image data, then the image expert model in the TIS-MOME is fine-tuned through single-modal training, and the Self-Attention parameters shared by each expert model are updated.
[0066] If the single-modal training data is the pure biomolecular data, then the biomolecular expert model in the TIS-MOME is fine-tuned through single-modal training, and the Self-Attention parameters shared by each expert model are updated.
[0067] After the single-modal training and fine-tuning is completed, the first target TIS-MOME is obtained.
[0068] In applications, if the single-modal training data is plain text data, the Masked Language Modeling method is used to pre-train the text expert model on the plain text data. At this time, the FFN weights of the image expert model, the biomolecular expert model and the hybrid expert model are frozen, and the shared Self-Attention parameters are updated.
[0069] If the single-modal training data is pure image data, the Masked Image Modeling method is used to pre-train the image expert model on the pure image data. At this time, the FFN weights of the text expert model, the biomolecular expert model and the hybrid expert model are frozen, and the shared Self-Attention parameters are updated.
[0070] If the single-modal training data is pure biomolecular data, the Masked Sequence Modeling method is used to pre-train a biomolecular expert model on the pure biomolecular data. At this time, the FFN weights of the text expert model, image expert model and hybrid expert model are frozen, and the shared Self-Attention parameters are updated.
[0071] After the first phase of training fine-tuning, the first target TIS-MOME is obtained and saved for subsequent phases of training fine-tuning.
[0072] In applications, if the unimodal training data is pure text data, a token can be randomly selected from the text according to the prediction task method, and the randomly selected token can be replaced with [MASK]. The TIS-MOME model is then trained to predict the replaced token from the "unreplaced text". A classifier is then used to obtain the output distribution of the masked token. Finally, the corresponding cross-entropy loss is calculated based on the cross-entropy loss function. The text expert model is then trained based on this cross-entropy loss, and the shared Self-Attention parameters are updated. It is understandable that if the unimodal training data is pure image data or pure biomolecular data, the image expert model and biomolecular expert model can also be trained according to the prediction task method; this will not be elaborated upon here.
[0073] Step S3: Obtain bimodal training data and proceed to step S4.
[0074] In applications, bimodal training data can also be obtained from some published multimodal datasets, similar to obtaining unimodal training data, which will not be elaborated here.
[0075] In one embodiment, the bimodal training data consists of N data pairs, each including a first data pair and a second data pair, where the first data pair and the second data pair are data from different modalities, and N is a positive integer.
[0076] In applications, bimodal training data can be text-image data pairs, text-biomolecule data pairs, or image-biomolecule data pairs; there is no limitation here. It is understood that the amount of bimodal training data is determined based on the actual situation and is not limited here.
[0077] Step S4: Perform bimodal training fine-tuning on the first target TIS-MOME based on the bimodal training data to obtain the second target TIS-MOME, and proceed to step S5.
[0078] In application, when performing bimodal training and fine-tuning on the first target TIS-MOME, it can be done using only text-image data pairs to train and fine-tune the text expert model, image expert model, and hybrid expert model; it can also be done using only text-biomolecule data pairs to train and fine-tune the text expert model, biomolecule expert model, and hybrid expert model; or it can be done using only image-biomolecule data pairs to train and fine-tune the image expert model, biomolecule expert model, and hybrid expert model.
[0079] Understandably, depending on actual needs, in step S4, the corresponding target expert models can also be trained and fine-tuned sequentially based on the input bimodal training data. For example, first input text-image data pairs and perform bimodal training and fine-tuning on the text expert model, image expert model, and hybrid expert model; then input text-biomolecule data pairs and perform bimodal training and fine-tuning on the text expert model, biomolecule expert model, and hybrid expert model; finally input image-biomolecule data pairs and perform bimodal training and fine-tuning on the image expert model, biomolecule expert model, and hybrid expert model. This is just an example and is not a limitation.
[0080] In one embodiment, such as Figure 3 As shown, step S4 includes the following steps S41 to S43:
[0081] Step S41: Determine the target expert model, proceed to step S42;
[0082] Step S42: Based on the first data and the second data in the data pair, perform dual-modal training and fine-tuning on the target expert model in the first target TIS-MOME, and update the Self-Attention parameters shared by each expert model; proceed to step S43;
[0083] Step S43: After the dual-modal training fine-tuning is completed, the second target TIS-MOME is obtained.
[0084] In applications, the target expert is determined based on the input bimodal training data. The target expert is either a bimodal expert model or a hybrid model. For example, when the input is a text-image data pair, the target expert model is a text expert model, an image expert model, and a hybrid expert model. The same applies to other data pairs, which will not be elaborated here.
[0085] In the application, when performing bimodal training and fine-tuning on the defined target expert model, the non-target expert models are frozen. This stage only updates the target expert model and the shared Self-Attention parameters. Specifically, on the input text-image data pairs, bimodal training and fine-tuning is performed on the text expert model, image expert model, and hybrid expert model. At this time, the FFN weights of the biomolecular expert model are frozen. Similarly, on the input text-biomolecular data pairs, bimodal training and fine-tuning is performed on the text expert model, biomolecular expert model, and hybrid expert model. At this time, the FFN weights of the image expert model are frozen. And on the input image-biomolecular data pairs, bimodal training and fine-tuning is performed on the image expert model, biomolecular expert model, and hybrid expert model. At this time, the FFN weights of the text expert model are frozen. After training and fine-tuning, the obtained second target TIS-MOME is saved for subsequent training and fine-tuning stages.
[0086] In one embodiment, step S42 includes the following steps S421 to S424:
[0087] Step S421: Use the corresponding target expert model to obtain the first feature representation of the first data and the second feature representation of the second data in each data pair;
[0088] Step S422: Perform linear mapping and normalization on the first feature representation and the second feature representation in each data pair to obtain the corresponding first target feature representation and second target feature representation;
[0089] Step S423: Based on the first target feature representation and the second target feature representation, obtain the similarity between each first data and each second data;
[0090] Step S424: Obtain the cross-entropy loss of the similarity between each first data point and each second data point according to the cross-entropy loss function, and perform bimodal training fine-tuning on the target expert model in the first target TIS-MOME according to the cross-entropy loss, and update the Self-Attention parameters shared by each expert model.
[0091] In the application, given N data pairs, the training aims to predict matching pairs from N×N possible data pairs, with N positive samples and N... 2 -N negative samples. Given N text-image data pairs, first output the first feature representation using the text expert model [T_CLS] token, and then output the second feature representation using the image expert model [I_CLS] token;
[0092] After linear mapping and normalization, the corresponding first target feature representation is obtained. Second target feature representation
[0093] According to the formula To obtain the similarity between an image and text, according to the formula... Obtain the text-to-image similarity, where, This represents the image-to-text similarity between the i-th image and the j-th text. This represents the text-to-image similarity between the i-th text and the j-th image;
[0094] The similarity between the acquired images and text, as well as the similarity between text and images, is normalized using the Softmax function. The specific formula is as follows: and Where σ represents the temperature parameter being learned, This represents the similarity between the normalized image and the text. This represents the similarity between text and images after normalization.
[0095] Based on the cross-entropy loss function (where M) i2t =2 indicates two categories, positive and negative, for the image-to-text conversion; M t2i =2 indicates two categories, positive and negative, for text to image; y c It is a one-hot vector where each element has only two values: 0 and 1. A value of 1 is assigned if the predicted image and text are similar, and a value of 0 is assigned if the predicted image and text are dissimilar. , representing the probabilities of predicting image-to-text and text-to-image similarities belonging to category C, respectively. The cross-entropy loss of text-to-image and image-to-text similarities is obtained, and the model is trained using this cross-entropy loss.
[0096] In the application, if N text-biomolecule data pairs are given, the first feature representation is first output through the text expert model [T_CLS] token, and the second feature representation is output through the biomolecule expert model [S_CLS] token;
[0097] After linear mapping and normalization, the corresponding first target feature representation is obtained. and second target feature representation
[0098] According to the formula To obtain the similarity between biomolecules and text, according to the formula... Obtain the similarity between text and biomolecules, where, This represents the sequence-to-text similarity between the i-th biomolecule and the j-th text. This represents the text-to-sequence similarity between the i-th text and the j-th biomolecule.
[0099] The similarity scores from biomolecules to text and from text to biomolecules were normalized using the Softmax function. The specific formula is as follows: and Where σ represents the temperature parameter being learned, This represents the normalized similarity between biomolecules and text. This indicates the similarity between normalized text and biological molecules.
[0100] Based on the cross-entropy loss function (where M) s2t =2 represents the positive and negative categories from biomolecules to text, M t2s =2 indicates two categories, positive and negative, from text to biomolecules, y c It is a one-hot vector where each element has only two values: 0 and 1. A value of 1 is assigned if the predicted biomolecule is similar to the text, and a value of 0 is assigned if the predicted biomolecule is dissimilar to the text. , representing the probabilities of predicting biomolecule-to-text and text-to-biomolecule similarities belonging to category C, respectively. The cross-entropy loss of text-to-biomolecule and biomolecule-to-text similarities is obtained, and the model is trained using this cross-entropy loss.
[0101] In applications, if N image-biomolecule data pairs are given, the corresponding processing flow is similar to that described above for N text-biomolecule data pairs or N text-image data pairs, and will not be repeated here.
[0102] In one embodiment, step S421 is followed by the following steps S425 to S427:
[0103] Step S425: Fuse the first feature representation and the second feature representation to obtain a target feature representation representing the first data-second data pair;
[0104] Step S426: Input the target feature representation into the target classifier for binary classification and obtain the corresponding cross-entropy loss; wherein, the target classifier is a classifier with cross-entropy loss calculation function;
[0105] Step S427: Perform bimodal training and fine-tuning on the target expert model in the first target TIS-MOME according to the cross-entropy loss, and update the Self-Attention parameters shared by each expert model.
[0106] In the application, given N text-image data pairs, the first feature representation output by the text expert model [T_CLS] token and the second feature representation output by the image expert model [I_CLS] token can be obtained through step S421. Then, the first feature representation and the second feature representation are fused to obtain a target feature representation representing the image-text pair. The target feature representation is then input into a classifier with cross-entropy loss calculation function for binary classification, and finally the image-to-text cross-entropy loss is obtained. The model is then trained using this cross-entropy loss.
[0107] In the application, given N text-biomolecule data pairs, step S421 can obtain the first feature representation output by the text expert model [T_CLS] token and the second feature representation output by the biomolecule expert model [S_CLS] token. Then, the first feature representation and the second feature representation are fused to obtain a target feature representation representing the biomolecule-text pair. This target feature representation is then input into a classifier with cross-entropy loss calculation function for binary classification, and finally the image-to-text cross-entropy loss is obtained. The model is then trained using this cross-entropy loss.
[0108] In applications, if N image-biomolecule data pairs are given, the corresponding processing flow is similar to that described above for N text-biomolecule data pairs or N text-image data pairs, and will not be repeated here.
[0109] After the above-mentioned pre-training stages, the parameters of each expert model of TIS-MOME are well initialized. After the following multimodal training fine-tuning stages, the generalization performance of the model can be greatly improved. The specific multimodal training fine-tuning process is described in steps S5 and S6.
[0110] Step S5: Obtain multimodal training data, then proceed to step S6.
[0111] In applications, when obtaining multimodal training data, it can also be obtained from some published multimodal datasets, similar to obtaining single / dual modal training data, which will not be elaborated here.
[0112] In one embodiment, the multimodal training data includes N data pairs, each data pair including a first data pair, a second data pair, and a third data pair. The first data pair, the second data pair, and the third data pair are data from different modalities, and the correspondence between the first data pair and the second data pair is fixed. N is a positive integer.
[0113] In applications, multimodal training data can be (text + image) - biomolecule data pairs, where the correspondence between text and image data is fixed; it can also be (text + biomolecule) - image data pairs, where the correspondence between text and biomolecule data is fixed; or it can be (image + biomolecule) - text data pairs, where the correspondence between image and biomolecule data is fixed. Understandably, the amount of multimodal training data is determined based on the actual situation and is not limited here.
[0114] Step S6: Perform multimodal training and fine-tuning on the second target TIS-MOME based on the multimodal training data to obtain the third target TIS-MOME and output it.
[0115] In applications, when performing multimodal training and fine-tuning of the second objective TIS-MOME, it is possible to perform multimodal training and fine-tuning of the text expert model, image expert model, biomolecular expert model, and hybrid expert model using only (text + image) - biomolecular data pairs; it is also possible to perform bimodal training and fine-tuning of the text expert model, image expert model, biomolecular expert model, and hybrid expert model using only (text + biomolecular) - image data pairs; and it is also possible to perform bimodal training and fine-tuning of the text expert model, image expert model, biomolecular expert model, and hybrid expert model using only (image + biomolecular) - text data pairs.
[0116] Understandably, depending on actual needs, in step S6, each expert model can also be trained and fine-tuned sequentially based on the corresponding multimodal training data input in sequence. For example, first input (text + image) - biomolecule data pairs to perform multimodal training and fine-tuning on each expert model, then input (text + biomolecule) - image data pairs to perform multimodal training and fine-tuning on each expert model, and finally input (image + biomolecule) - text data pairs to perform multimodal training and fine-tuning on each expert model. This is just an example and is not a limitation.
[0117] In one embodiment, step S6 specifically includes the following steps S61 to S63:
[0118] Step S61: Based on the corresponding expert model, obtain the first target feature representation of the first data, the second target feature representation of the second data, and the third target feature representation of the third data in the multimodal training data;
[0119] Step S62: Based on the first target feature representation, the second target feature representation, and the third target feature representation, obtain the first similarity between the first data and the third data, and the second similarity between the second data and the third data;
[0120] Step S63: Obtain the cross-entropy loss of the first similarity and the second similarity according to the cross-entropy loss function, and perform multimodal training and fine-tuning of the expert model in the second target TIS-MOME according to the cross-entropy loss, and update the Self-Attention parameters shared by each expert model.
[0121] In application, the first similarity includes the similarity from the first data to the third data and the similarity from the third data to the first data; the second similarity includes the similarity from the second data to the third data and the similarity from the third data to the second data.
[0122] In this application, taking a (image + biomolecule) - text data pair with a fixed correspondence between image data and biomolecule data as an example, steps S61 to S63 will be explained as follows:
[0123] Given N (image + biomolecule)-text data pairs, output the first feature representation using the text expert model [T_CLS] token, the second feature representation using the image expert model [I_CLS] token, and the third feature representation using the biomolecule expert model [S_CLS] token;
[0124] After linear mapping and normalization, the corresponding first target feature representation is obtained. Second target feature representation and the third target feature representation
[0125] According to the formula To obtain the similarity between an image and text, according to the formula... To obtain the text-to-image similarity, use the formula... To obtain the similarity between biomolecules and text, according to the formula... Obtain the similarity between text and biomolecules; where, This represents the image-to-text similarity between the i-th image and the j-th text. This represents the text-to-image similarity between the i-th text and the j-th image. This represents the sequence-to-text similarity between the i-th biomolecule and the j-th text. This represents the text-to-sequence similarity between the i-th text and the j-th biomolecule.
[0126] The obtained image-to-text, text-to-image, biomolecule-to-text, and text-to-biomolecule similarities are normalized using the Softmax function. The specific formula is as follows: and Where σ represents the temperature parameter being learned, This represents the similarity between the normalized image and the text. This represents the similarity between normalized text and images. This represents the normalized similarity between biomolecules and text. This indicates the similarity between normalized text and biological molecules.
[0127] Based on the cross-entropy loss function (where M) i2t =2 indicates two categories, positive and negative, for the image-to-text conversion; M t2i =2 indicates two categories, positive and negative, for text-to-image conversion; M s2t =2 represents the positive and negative categories from biomolecules to text, M t2s =2 indicates two categories, positive and negative, from text to biomolecules, y c It is a one-hot vector where each element has only two values: 0 and 1. A value of 1 indicates that the predicted image and text are similar, while a value of 0 indicates that they are dissimilar. Similarly, a value of 1 indicates that the predicted biomolecules are similar to the text, while a value of 0 indicates that they are dissimilar. Let represent the probabilities that the predicted image-to-text and text-to-image similarities belong to category C, respectively. , representing the probabilities that the predicted biomolecule-to-text and text-to-biomolecule similarities belong to category C, respectively. The cross-entropy loss of text-to-image, image-to-text, text-to-biomolecule, and biomolecule-to-text similarities is obtained, and the model is trained using this cross-entropy loss.
[0128] It is understandable that if the given data consists of N (text + image) - biomolecule data pairs or N (text + biomolecule) - image data pairs, the corresponding processing flow is similar to that of the given N (image + biomolecule) - text data pairs, and will not be repeated here.
[0129] After training and fine-tuning, TIS-MOME can handle downstream tasks including classification and retrieval. Although current medical-related multimodal datasets primarily contain textual descriptions related to images and biomolecules within the scope of Western medicine, inputting image data into the trained and fine-tuned third-target TIS-MOME allows it to output textual content related to Traditional Chinese Medicine (TCM) using its visual response and reasoning capabilities, thus predicting the connection between medical images and TCM. Similarly, inputting biomolecule data into the third-target TIS-MOME also allows it to predict the connection between biomolecules and TCM. This novel multimodal large-scale model, TIS-MOME, proposed in this application, effectively integrates and correlates Western and Chinese medicine textual and image modal data using biomolecule modal data as a bridge. This is beneficial for revealing the occurrence and development patterns of Western medicine "diseases" and TCM "syndromes," elucidating the biological basis of the integration of Western and Chinese medicine diseases and syndromes, and optimizing precision diagnosis and treatment strategies.
[0130] The fine-tuning method of the multimodal large model TIS-MOME proposed in this application includes a three-stage fine-tuning strategy of single-modality, dual-modality and three (multi)modality. After completing the training and fine-tuning, the third objective of TIS-MOME can establish the correlation between multimodal data of traditional Chinese and Western medicine, break down the barriers between different theoretical systems of traditional Chinese and Western medicine, and promote the intelligent and precise development of integrated diagnosis and treatment of traditional Chinese and Western medicine.
[0131] By using the multimodal large model TIS-MOME proposed in this application to analyze the biomolecular mechanisms of integrated traditional Chinese and Western medicine diagnosis and treatment, more integrated traditional Chinese and Western medicine diagnosis and treatment plans inferred by large models can be generalized and extended to the relationship between traditional Chinese medicine, Chinese herbal medicine and Western medicine, promoting the modernization and internationalization of traditional Chinese medicine.
[0132] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0133] Example 2
[0134] This application also provides a fine-tuning device 100 for a multimodal large model, used to execute the method steps in the above-described fine-tuning method embodiments for a multimodal large model. This device can be a virtual appliance in a terminal device, run by the terminal device's processor, or it can be the terminal device itself.
[0135] like Figure 4 As shown, the fine-tuning device 100 for a multimodal large model provided in this application embodiment includes:
[0136] The first data acquisition module 101 is used to acquire single-modal training data;
[0137] The first training module 102 is used to perform single-modal training fine-tuning on the TIS-MOME based on the single-modal training data to obtain the first target TIS-MOME;
[0138] The second data acquisition module 103 is used to acquire dual-modal training data;
[0139] The second training module 104 is used to perform bimodal training fine-tuning on the first target TIS-MOME based on the bimodal training data to obtain the second target TIS-MOME;
[0140] The third data acquisition module 105 is used to acquire multimodal training data;
[0141] The third training module 106 is used to perform multimodal training fine-tuning on the second target TIS-MOME based on the multimodal training data, to obtain the third target TIS-MOME and output it.
[0142] In one embodiment, the TIS-MOME includes independent image expert models, text expert models, biomolecular expert models, and hybrid expert models;
[0143] The image expert model is used for:
[0144] The input image data is divided into blocks to obtain multiple image blocks;
[0145] Flatten the multiple image blocks to obtain the corresponding multiple image block vectors;
[0146] Linear projection is performed on the plurality of image block vectors, and a first input representation of the image data is obtained through a first position encoding and a first type encoding;
[0147] The text expert model is used for:
[0148] The input text data is segmented to obtain multiple word roots;
[0149] Obtain multiple root vectors corresponding to the multiple root words;
[0150] Linear projection is performed on the plurality of word root vectors, and a second input representation of the text data is obtained through a second positional encoding and a second type encoding;
[0151] The biomolecular expert model is used for:
[0152] The input biomolecular data is segmented to obtain multiple fragments;
[0153] Obtain multiple segment vectors corresponding to the multiple segments;
[0154] Linear projection is performed on the multiple fragment vectors, and a third input representation of the biomolecular data is obtained through third position encoding and third type encoding;
[0155] The hybrid expert model is used to concatenate the first input representation obtained by the image expert model, the second input representation obtained by the text expert model, and the third input representation obtained by the biomolecular expert model to obtain the fourth input representation of the hybrid expert model.
[0156] In one embodiment, the single-modal training data includes plain text data, plain image data, or plain biomolecular data, and the first training module 102 is further used for:
[0157] If the single-modal training data is the plain text data, then the text expert model in the TIS-MOME is fine-tuned through single-modal training, and the Self-Attention parameters shared by each expert model are updated.
[0158] If the single-modal training data is the pure image data, then the image expert model in the TIS-MOME is fine-tuned through single-modal training, and the Self-Attention parameters shared by each expert model are updated.
[0159] If the single-modal training data is the pure biomolecular data, then the biomolecular expert model in the TIS-MOME is fine-tuned through single-modal training, and the Self-Attention parameters shared by each expert model are updated.
[0160] After the single-modal training and fine-tuning is completed, the first target TIS-MOME is obtained.
[0161] In one embodiment, the bimodal training data consists of N data pairs including first data and second data, where the first data and the second data are data from different modalities, and N is a positive integer. The second training module 104 is further configured to:
[0162] Determine the target expert model;
[0163] Based on the first data and the second data in the data pair, the target expert model in the first target TIS-MOME is trained and fine-tuned in a dual-modal manner, and the Self-Attention parameters shared by each expert model are updated.
[0164] After the dual-modal training and fine-tuning is completed, the second target TIS-MOME is obtained.
[0165] In one embodiment, the second training module 104 is further configured to:
[0166] Use the corresponding target expert model to obtain the first feature representation of the first data and the second feature representation of the second data in each data pair;
[0167] For each data pair, the first feature representation and the second feature representation are respectively subjected to linear mapping and normalization processing to obtain the corresponding first target feature representation and second target feature representation;
[0168] Based on the first target feature representation and the second target feature representation, the similarity between each first data point and each second data point is obtained;
[0169] Based on the cross-entropy loss function, the cross-entropy loss of the similarity between each first data point and each second data point is obtained, and the target expert model in the first target TIS-MOME is fine-tuned by bimodal training based on the cross-entropy loss, and the Self-Attention parameters shared by each expert model are updated.
[0170] In one embodiment, the second training module 104 is further configured to:
[0171] The first feature representation and the second feature representation are fused to obtain a target feature representation representing the first data-second data pair;
[0172] The target feature representation is input into a target classifier for binary classification, and the corresponding cross-entropy loss is obtained; wherein, the target classifier is a classifier with cross-entropy loss calculation function;
[0173] The target expert model in the first target TIS-MOME is fine-tuned using bimodal training based on the cross-entropy loss, and the Self-Attention parameters shared by each expert model are updated.
[0174] In one embodiment, the multimodal training data includes N data pairs, each data pair including a first data-second data and a third data, wherein the first data, the second data, and the third data are data of different modalities, and the correspondence between the first data and the second data is fixed, and N is a positive integer; the third training module 106 is further used for:
[0175] Based on the corresponding expert model, the first target feature representation of the first data, the second target feature representation of the second data, and the third target feature representation of the third data in the multimodal training data are obtained;
[0176] Based on the first target feature representation, the second target feature representation, and the third target feature representation, a first similarity between the first data and the third data, and a second similarity between the second data and the third data are obtained;
[0177] Based on the cross-entropy loss function, the cross-entropy loss of the first similarity and the second similarity is obtained, and the expert model in the second target TIS-MOME is fine-tuned through multimodal training based on the cross-entropy loss, and the Self-Attention parameters shared by each expert model are updated.
[0178] In applications, each unit in the above-mentioned device can be a software program module, or it can be implemented by different logic circuits integrated in the processor or by independent physical components connected to the processor, or it can be implemented by multiple distributed processors.
[0179] Example 3
[0180] like Figure 5 As shown, this application embodiment also provides a terminal device 200, including: at least one processor 201 ( Figure 5 The diagram shows only one processor), memory 202, and a computer program 203 stored in memory 202 that can run on at least one processor 201. When processor 201 executes computer program 203, it implements the steps in the various method embodiments described above.
[0181] In applications, terminal devices may include, but are not limited to, processors and memory. Figure 5 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, human-computer interaction devices, input / output devices, network access devices, etc. The network access device may include a communication module for communication between the terminal device and the user terminal.
[0182] In applications, the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. For example, the processor can be a timing controller (TCON). A general-purpose processor can be a microprocessor or any conventional processor.
[0183] In applications, the memory may be an internal storage unit of the terminal device in some embodiments, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. The memory may also include both internal and external storage units of the terminal device. The memory is used to store the operating system, applications, boot loader, data, and other programs, such as program code for computer programs. The memory can also be used to temporarily store data that has been output or will be output.
[0184] In applications, the communication module can be configured as any device capable of long-distance wired or wireless communication directly or indirectly with user terminals, depending on actual needs. For example, the communication module can provide solutions for communication applications on network devices, including Wireless Local Area Networks (WLANs) (such as Wi-Fi networks), Bluetooth, Zigbee, mobile communication networks, Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The communication module can include an antenna, which can have a single element or be an antenna array with multiple elements. The communication module can receive electromagnetic waves through the antenna, frequency modulate and filter the electromagnetic wave signal, and send the processed signal to the processor. The communication module can also receive signals to be transmitted from the processor, frequency modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.
[0185] It should be noted that the information interaction and execution process between the above-mentioned devices / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0186] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The functional modules in the embodiments can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules can be implemented in hardware or as software functional modules. Furthermore, the specific names of the functional modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0187] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.
[0188] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.
[0189] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.
[0190] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0191] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0192] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0193] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0194] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for fine-tuning a multimodal large model, characterized in that, Applied to TIS-MOME, the fine-tuning method includes: Acquire single-modal training data; Based on the single-modal training data, the TIS-MOME is fine-tuned using single-modal training to obtain the first target TIS-MOME; Acquire bimodal training data; Based on the bimodal training data, the first target TIS-MOME is fine-tuned through bimodal training to obtain the second target TIS-MOME; Acquire multimodal training data; Based on the multimodal training data, the second target TIS-MOME is fine-tuned through multimodal training to obtain the third target TIS-MOME and output it.
2. The fine-tuning method for a multimodal large model as described in claim 1, characterized in that, The TIS-MOME includes independent image expert models, text expert models, biomolecular expert models, and hybrid expert models; The image expert model is used for: The input image data is divided into blocks to obtain multiple image blocks; Flatten the multiple image blocks to obtain the corresponding multiple image block vectors; Linear projection is performed on the plurality of image block vectors, and a first input representation of the image data is obtained through a first position encoding and a first type encoding; The text expert model is used for: The input text data is segmented to obtain multiple word roots; Obtain multiple root vectors corresponding to the multiple root words; Linear projection is performed on the plurality of word root vectors, and a second input representation of the text data is obtained through a second positional encoding and a second type encoding; The biomolecular expert model is used for: The input biomolecular data is segmented to obtain multiple fragments; Obtain multiple segment vectors corresponding to the multiple segments; Linear projection is performed on the multiple fragment vectors, and a third input representation of the biomolecular data is obtained through third position encoding and third type encoding; The hybrid expert model is used to concatenate the first input representation obtained by the image expert model, the second input representation obtained by the text expert model, and the third input representation obtained by the biomolecular expert model to obtain the fourth input representation of the hybrid expert model.
3. The fine-tuning method for a multimodal large model as described in claim 2, characterized in that, The single-modal training data includes plain text data, plain image data, or plain biomolecule data. The step of performing single-modal training fine-tuning on the TIS-MOME based on the single-modal training data to obtain the first target TIS-MOME includes: If the single-modal training data is the plain text data, then the text expert model in the TIS-MOME is fine-tuned through single-modal training, and the Self-Attention parameters shared by each expert model are updated. If the single-modal training data is the pure image data, then the image expert model in the TIS-MOME is fine-tuned through single-modal training, and the Self-Attention parameters shared by each expert model are updated. If the single-modal training data is the pure biomolecular data, then the biomolecular expert model in the TIS-MOME is fine-tuned through single-modal training, and the Self-Attention parameters shared by each expert model are updated. After the single-modal training and fine-tuning is completed, the first target TIS-MOME is obtained.
4. The fine-tuning method for a multimodal large model as described in claim 2, characterized in that, The bimodal training data consists of N data pairs, each including a first data pair and a second data pair. The first data pair and the second data pair are data from different modalities, and N is a positive integer. The step of performing bimodal training fine-tuning on the first target TIS-MOME based on the bimodal training data to obtain the second target TIS-MOME includes: Determine the target expert model; Based on the first data and the second data in the data pair, the target expert model in the first target TIS-MOME is trained and fine-tuned in a dual-modal manner, and the Self-Attention parameters shared by each expert model are updated. After the dual-modal training and fine-tuning is completed, the second target TIS-MOME is obtained.
5. The fine-tuning method for a multimodal large model as described in claim 4, characterized in that, The step of performing bimodal training and fine-tuning of the target expert model in the first target TIS-MOME based on the first data and the second data in the data pair, and updating the Self-Attention parameters shared by each expert model, includes: Use the corresponding target expert model to obtain the first feature representation of the first data and the second feature representation of the second data in each data pair; For each data pair, the first feature representation and the second feature representation are respectively subjected to linear mapping and normalization processing to obtain the corresponding first target feature representation and second target feature representation; Based on the first target feature representation and the second target feature representation, the similarity between each first data point and each second data point is obtained; Based on the cross-entropy loss function, the cross-entropy loss of the similarity between each first data point and each second data point is obtained, and the target expert model in the first target TIS-MOME is fine-tuned by bimodal training based on the cross-entropy loss, and the Self-Attention parameters shared by each expert model are updated.
6. The fine-tuning method for a multimodal large model as described in claim 5, characterized in that, After obtaining the first feature representation of the first data and the second feature representation of the second data in each data pair using the corresponding target expert model, the method further includes: The first feature representation and the second feature representation are fused to obtain a target feature representation representing the first data-second data pair; The target feature representation is input into a target classifier for binary classification, and the corresponding cross-entropy loss is obtained; wherein, the target classifier is a classifier with cross-entropy loss calculation function; The target expert model in the first target TIS-MOME is fine-tuned using bimodal training based on the cross-entropy loss, and the Self-Attention parameters shared by each expert model are updated.
7. The fine-tuning method for a multimodal large model as described in claim 2, characterized in that, The multimodal training data includes N data pairs, each data pair including a first data pair, a second data pair, and a third data pair. The first data pair, the second data pair, and the third data pair are data from different modalities, and the correspondence between the first data pair and the second data pair is fixed. N is a positive integer. The step of performing multimodal training and fine-tuning on the second target TIS-MOME based on the multimodal training data to obtain and output the third target TIS-MOME includes: Based on the corresponding expert model, the first target feature representation of the first data, the second target feature representation of the second data, and the third target feature representation of the third data in the multimodal training data are obtained; Based on the first target feature representation, the second target feature representation, and the third target feature representation, a first similarity between the first data and the third data, and a second similarity between the second data and the third data are obtained; Based on the cross-entropy loss function, the cross-entropy loss of the first similarity and the second similarity is obtained, and the expert model in the second target TIS-MOME is fine-tuned through multimodal training based on the cross-entropy loss, and the Self-Attention parameters shared by each expert model are updated.
8. A fine-tuning device for a multimodal large model, characterized in that, Applied to TIS-MOME, the fine-tuning device includes: The first data acquisition module is used to acquire single-modal training data; The first training module is used to perform single-modal training fine-tuning on the TIS-MOME based on the single-modal training data to obtain the first target TIS-MOME; The second data acquisition module is used to acquire dual-modal training data; The second training module is used to perform bimodal training fine-tuning on the first target TIS-MOME based on the bimodal training data to obtain the second target TIS-MOME; The third data acquisition module is used to acquire multimodal training data; The third training module is used to perform multimodal training fine-tuning on the second target TIS-MOME based on the multimodal training data, to obtain the third target TIS-MOME and output it.
9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the fine-tuning method for the multimodal large model as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the fine-tuning method for the multimodal large model as described in any one of claims 1 to 7.