Multi-modal medical large model application and fine adjustment method and device and computer equipment

Abstract

The invention relates to the technical field of natural language processing, in particular to a multi-modal medical large model application and fine adjustment method and device and computer equipment. The multi-modal feature data is input into a pre-trained multi-modal medical large model, an output result is obtained, the multi-modal medical large model comprises a multi-modal semantic alignment model and an integrated enhanced medical large model, and the multi-modal semantic alignment model comprises a multi-modal semantic alignment model and an integrated enhanced medical large model; the multi-modal semantic alignment model is obtained by training an initial multi-modal semantic alignment model through a multi-modal feature sample vector, a representation extraction sample vector and a semantic tag corresponding to the multi-modal feature sample vector. Knowledge and capability of a plurality of source medical models are integrated, dependence on data resources is reduced, generalization capability of the models is improved, and accuracy and reliability of diagnosis are enhanced; multi-modal features are extracted, the semantic alignment effect is optimized, and information loss and semantic offset are avoided.

Description

Technical Field

[0001] This specification relates to the field of natural language processing technology, especially to multimodal medical large model application and fine-tuning methods, devices and computer equipment. Background Art

[0002] Artificial intelligence technology has shown broad application prospects in the field of medical health. As a representative of artificial intelligence in the medical field, multimodal medical big models have the potential to process complex data and provide accurate diagnosis and treatment plans. It can integrate a variety of medical data, including imaging, genes, electronic medical records, etc., to support disease diagnosis, risk prediction and personalized treatment. With the rapid development of medical big models, its parameter scale has expanded from hundreds of millions to tens of billions or even hundreds of billions, enabling it to have a deeper understanding of the mechanisms and individual differences of neuropsychiatric diseases, discover new disease markers and predict the risk and prognosis of the disease. This technological advancement provides strong support for precision medicine and personalized intervention for neuropsychiatric diseases, greatly promotes the progress of medical research and clinical applications, and also brings new possibilities for the early detection, diagnosis and treatment of neuropsychiatric diseases.

[0003] However, the training and fine-tuning process of large medical models also faces unprecedented challenges. As the scale of the model increases, the required computing resources and data resources grow exponentially. High-dimensional medical data usually has complex structures and diversity, requiring a lot of computing power to process. Especially in resource-limited environments, it becomes increasingly difficult to obtain enough high-performance computing devices, such as GPUs. In addition, the scarcity and high annotation costs of multimodal medical data further increase the difficulty of building large medical models. Therefore, how to optimize the use of computing resources and effectively fine-tune large medical models without sacrificing model performance has become an important direction of current research. The continued development of multimodal medical large models will bring more possibilities for precision medicine, help to deeply understand disease mechanisms, discover new diagnostic markers, and promote the application of personalized medicine and intervention methods. Summary of the invention

[0004] To solve the problems in the prior art, this specification provides a multimodal medical large model application and fine-tuning method, device and computer equipment.

[0005] An embodiment of the present specification provides a method for applying a multimodal medical big model, the method comprising: acquiring multimodal feature data; inputting the multimodal feature data into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model comprises a multimodal semantic alignment model and an integrated enhanced medical big model, the multimodal semantic alignment model is obtained by training an initial multimodal semantic alignment model with multimodal feature sample vectors, representation extraction sample vectors and semantic labels corresponding to the multimodal feature sample vectors, and the integrated enhanced medical big model is obtained by training a predicted distribution matrix obtained by multiple source medical models based on the prediction of single-modal medical sample data and the real distribution matrix corresponding to each source medical model.

[0006] The embodiment of the present specification provides a method for fine-tuning a multimodal medical large model, the method comprising: inputting a multimodal feature sample vector and a representation extraction sample vector into an initial multimodal semantic alignment model to obtain a prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features in the multimodal feature sample vector; calculating a multimodal matching loss function according to the prediction result and the semantic label corresponding to the multimodal feature sample vector, iteratively updating the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function, and constructing a multimodal semantic alignment model; inputting the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical large model to obtain a prediction result output by the integrated enhanced medical large model, wherein the integrated enhanced medical large model is trained by a prediction distribution matrix obtained by predicting single-modal medical sample data using multiple source medical models and a true distribution matrix corresponding to each source medical model.

[0007] According to one aspect of the embodiments of the present specification, the method further includes: setting a fully connected linear projection layer between the multimodal semantic alignment model and the integrated enhanced medical large model to match the output of the multimodal semantic alignment model with the feature dimension of the input of the integrated enhanced medical large model; optimizing the multimodal semantic alignment model according to the loss function of the masked language model to determine the output of the multimodal semantic alignment model and the prediction results of the output of the integrated enhanced medical large model.

[0008] According to one aspect of the embodiments of this specification, the characterization extraction sample vector is determined according to the following formula: v = u T (X-μ); where v represents the characterization sample extraction vector, X represents the multimodal feature sample vector, μ represents the feature center vector, and u represents the main feature direction vector.

[0009] According to one aspect of an embodiment of this specification, a multimodal matching loss function is constructed in the following manner: by maximizing the similarity of a multimodal feature positive sample vector pair and minimizing the similarity of a multimodal feature negative sample vector pair, a contrast loss function is determined; based on the multimodal feature sample vector and the corresponding semantic label, a cosine distance loss function is determined; based on the contrast loss function and the cosine distance loss function, a multimodal matching loss function is constructed, and the formula is as follows:

[0010] L match =α·L InfoNCE +(1-α)·L cos , where L match represents the multimodal matching loss function, L InfoNCE Represents the contrast loss function of information aggregation, L cos represents the cosine distance loss function, and α represents the regularization coefficient.

[0011] According to one aspect of the embodiments of the present specification, the integrated enhanced medical big model is constructed in the following manner: inputting single-modality medical sample data into multiple source medical models to obtain the predicted distribution matrix output by each source medical model; calculating the cross entropy between the predicted distribution matrix of each source medical model and the true distribution matrix corresponding to each source medical model, and determining the weight of each predicted distribution matrix; weighting the predicted distribution matrix output by each source medical model according to the weight to obtain a fused probability distribution sample matrix; selecting one of the multiple source medical models as the initial medical big model, and according to the loss value between the predicted distribution matrix output by the initial medical big model, the fused probability distribution matrix and the true distribution matrix, iteratively training the parameters of the initial medical big model according to the loss value to construct the initial integrated enhanced medical big model; fine-tuning the initial integrated enhanced medical big model to obtain the integrated enhanced medical big model.

[0012] According to one aspect of an embodiment of the present specification, the loss value between the predicted distribution matrix output by the initial medical large model, the fused probability distribution matrix and the true distribution matrix is ​​calculated in the following manner: a fusion loss function is determined based on the difference between the true distribution matrix and the fused probability distribution matrix, and the difference between the fused probability distribution matrix and the predicted distribution matrix.

[0013] According to one aspect of the embodiments of this specification, the method further includes: using singular value decomposition to decompose the weight matrix pre-frozen in the initial integrated enhanced medical large model to obtain a decomposed singular scale vector and a singular direction matrix; freezing the singular direction matrix, generating multiple basic low-rank matrices and multiple intermediate matrices, controlling the number of low-rank matrices and the values ​​of low-rank matrices, and decomposing the singular direction matrix into at least one low-rank matrix; adjusting the parameters of the low-rank matrix to obtain a low-rank adjusted singular value matrix; merging the singular scale vector, the adjusted singular direction matrix, and the singular direction matrix after parameter quantization to generate an updated weight matrix.

[0014] The embodiment of the present specification provides a multimodal medical big model application device, the device comprising: an acquisition unit, used to acquire multimodal feature data; a prediction unit, used to input the multimodal feature data into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model comprises a multimodal semantic alignment model and an integrated enhanced medical big model, the multimodal semantic alignment model is obtained by training an initial multimodal semantic alignment model with multimodal feature sample vectors, representation extraction sample vectors and semantic labels corresponding to the multimodal feature sample vectors, and the integrated enhanced medical big model is obtained by training a plurality of source medical models based on a predicted distribution matrix obtained by predicting single-modal medical sample data and a true distribution matrix corresponding to each source medical model.

[0015] The embodiments of the present specification also provide a multimodal medical large model fine-tuning device, the device comprising: a first prediction result acquisition unit, used to input a multimodal feature sample vector, a representation extraction sample vector and a semantic label corresponding to the multimodal feature sample vector into an initial multimodal semantic alignment model to obtain a prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features in the multimodal feature sample vector; a multimodal semantic alignment model construction unit, used to calculate a multimodal matching loss function according to the prediction result and the semantic label, iteratively update the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function, and construct a multimodal semantic alignment model; a second prediction result acquisition unit, used to input the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical large model to obtain a prediction result output by the integrated enhanced medical large model, wherein the integrated enhanced medical large model is trained by a prediction distribution matrix obtained by predicting single-modal medical sample data using multiple source medical models and a true distribution matrix corresponding to each source medical model.

[0016] The embodiments of the present specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the multimodal medical large model application and fine-tuning method is implemented when the processor executes the computer program.

[0017] The embodiments of the present specification also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multimodal medical large model application and fine-tuning method is implemented.

[0018] The present invention reduces dependence on data resources by integrating the knowledge and capabilities of multiple source medical models, effectively improves the generalization ability of the model, and enhances the accuracy and reliability of diagnosis; by extracting multimodal features and optimizing their semantic alignment effects, the problems of information loss and semantic drift are avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of a multi-modal medical large model application method according to an embodiment of this specification;

[0021] Figure 2 It is a flowchart of a multi-modal medical large model fine-tuning method according to an embodiment of this specification;

[0022] Figure 3 Shown is a flow chart of a method for optimizing a multimodal semantic alignment model according to an embodiment of this specification;

[0023] Figure 4 The figure is a flow chart of a method for constructing a large medical model according to an embodiment of the present specification;

[0024] Figure 5 The figure is a flow chart of a method for fine-tuning an initial integrated enhanced medical large model according to an embodiment of the present specification;

[0025] Figure 6 The figure shows a schematic diagram of the structure of a multi-modal medical large model application device according to an embodiment of the present specification;

[0026] Figure 7 The figure shows a schematic diagram of the structure of a multi-modal medical large model fine-tuning device according to an embodiment of the present specification;

[0027] Figure 8 It is a schematic diagram of building an integrated enhanced medical large model based on a medical source model according to an embodiment of this specification;

[0028] Fig. 9 It is a schematic diagram of updating the weight matrix of the integrated enhanced medical model according to an embodiment of this specification;

[0029] Fig.10 FIG. 1 is a schematic diagram of a multimodal semantic alignment model based on adaptive fusion according to an embodiment of the present specification;

[0030] Fig.11 The figure shows a schematic diagram of the structure of a computer device according to an embodiment of the present specification.

[0031] Description of the accompanying symbols:

[0032] 601, acquisition unit;

[0033] 602. prediction unit;

[0034] 701. A first prediction result obtaining unit;

[0035] 702. Multimodal semantic alignment model construction unit;

[0036] 703. A second prediction result obtaining unit;

[0037] 1102. Computer equipment;

[0038] 1104. Processor;

[0039] 1106. Memory;

[0040] 1108, driving mechanism;

[0041] 1110, input / output module;

[0042] 1112. Input device;

[0043] 1114. Output device;

[0044] 1116. Presentation equipment;

[0045] 1118. Graphical user interface;

[0046] 1120. Network interface;

[0047] 1122. Communication link;

[0048] 1124. Communication bus. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0050] It should be noted that the terms "first", "second", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0051] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel.

[0052] It should be noted that the multimodal medical large model application and fine-tuning method of this specification can be used in the field of natural language processing technology, and can also be used in the field of medical technology. This specification does not limit the application field of the multimodal medical large model application and fine-tuning method.

[0053] Figure 1 The flowchart of a multimodal medical large model application method of the embodiment of this specification is shown, which specifically includes the following steps:

[0054] Step 101, obtaining multimodal feature data.

[0055] In this step, the multimodal feature data is in vector form, so it can also be called a multimodal feature vector. In this step, multimodal medical data is obtained, and the multimodal medical data is pre-fused into a multimodal feature vector to obtain multimodal feature data.

[0056] Step 102, input the multimodal feature data into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model includes a multimodal semantic alignment model and an integrated enhanced medical big model, the multimodal semantic alignment model is obtained by training an initial multimodal semantic alignment model with multimodal feature sample vectors, representation extraction sample vectors and semantic labels corresponding to the multimodal feature sample vectors, and the integrated enhanced medical big model is trained with a predicted distribution matrix obtained by predicting single-modal medical sample data using multiple source medical models and a true distribution matrix corresponding to each source medical model.

[0057] In this step, the multimodal semantic alignment model and the integrated enhanced medical model are both pre-trained models. Among them, by integrating the knowledge and capabilities of multiple source medical models, adopting the fusion strategy of predictive distribution representation, training and building into the integrated enhanced learning model, the dependence on data resources is significantly reduced, ensuring the model's learning of complex disease characteristics, improving the generalization ability of the integrated enhanced learning model, and enhancing the accuracy and reliability of diagnosis. In order for the integrated enhanced medical model to understand the multimodal feature data and perform subsequent diagnostic tasks, it is necessary to semantically align the multimodal feature data with the input space of the integrated enhanced medical model. In this specification, the multimodal semantic alignment model can effectively separate irrelevant redundant data in the multimodal feature data, thereby achieving accurate alignment of the multimodal feature data with the data semantic labels corresponding to the multimodal feature data.

[0058] Figure 2 The flowchart of a multi-modal medical large model fine-tuning method according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0059] Step 201, inputting a multimodal feature sample vector and a representation extraction sample vector into an initial multimodal semantic alignment model to obtain a prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features from the multimodal feature sample vector.

[0060] In this step, a multimodal semantic alignment model is constructed. This alignment model integrates the pre-fusion driven multimodal semantic alignment mechanism and the semantic alignment method based on adaptive representation calculation, aiming to semantically align the multimodal feature sample vector with the input space of the large medical model.

[0061] In this step, firstly, multimodal medical sample data is obtained, and the multimodal medical sample data is pre-fused into a multimodal feature sample vector, and the multimodal semantic label corresponding to the multimodal feature sample vector is determined, and the multimodal feature sample vector and the multimodal semantic label are combined in pairs to participate in the pre-training of the initial multimodal semantic alignment model to learn the association between multimodal features and multimodal semantic labels. Among them, the initial multimodal semantic alignment model adopts a Transformer structure, which includes a feature encoder and a label encoder. Specifically, the multimodal feature sample vector and the representation extraction sample vector are input into the feature encoder, and through the interaction of the cross-attention mechanism, a more semantically relevant prediction result is extracted and generated, which is the multimodal feature sample representation. Fig.10 As shown, Fig.10 The multimodal fusion feature vector and the representation extraction vector are input into the feature encoder, wherein the multimodal fusion feature vector and the representation extraction vector are jointly operated in the cross-attention layer of the feature encoder, and the operation results are output in the feedforward layer.

[0062] In the embodiments of this specification, the multimodal feature sample vector may be a vector composed of data in different modalities such as text, voice, and video. For example, a frame of a liver surgery video clip containing live sound and text description may have a corresponding semantic label that can be key lesion information in the surgery video clip, including but not limited to: lesion location, lesion shape, color characteristics, lesion name, etc.

[0063] In the embodiments of this specification, the characterization extraction sample vector is a random initial sample vector used to extract features of the multimodal feature sample vector. It can be any one or a combination of text features, voice features or video features in the above liver surgery video clip example. Furthermore, the characterization extraction sample vector is learnable and can be optimized. Specifically, the characterization extraction sample vector is optimized by the following formula:

[0064] v=u T (X-μ); where v represents the characterization sample extraction vector, X represents the multimodal feature sample vector, μ represents the feature center sample vector, and u represents the main feature direction sample vector. The feature center sample vector μ is the mean of the multimodal feature sample vector X, and the calculation formula is: The main eigendirection sample vector u is the first column of the left singular value matrix obtained by performing singular value decomposition on X-μ. The main eigendirection sample vector corresponds to the eigenvector with the largest eigenvalue, capturing the direction with the largest variance in the data.

[0065] In this step, the representation extraction sample vector and the multimodal feature sample vector are input into the initial multimodal semantic alignment model together to effectively separate irrelevant redundant data in the multimodal feature sample vector.

[0066] Step 202: Calculate a multimodal matching loss function according to the prediction result and the semantic label corresponding to the multimodal feature sample vector, iteratively update the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function, and construct a multimodal semantic alignment model.

[0067] In this step, if Fig.10 As shown in the figure, the multimodal semantic label is used as the input of the label encoder to generate a vector that matches the output of the feature encoder. Furthermore, the constraints of the multimodal matching loss function are introduced on the output results of the feature encoder and the label encoder to learn the mapping relationship that better matches the multimodal feature sample representation with the semantic label, so that the representation extraction vector can effectively capture the features in the multimodal sample vector.

[0068] Among them, the multimodal matching loss function combines contrast loss and cosine distance loss. Specifically, the core of contrast loss is to maximize the similarity of multimodal feature positive sample vector pairs and minimize the similarity of negative sample pairs, while further enhancing the consistency of feature vectors through cosine distance.

[0069] Specifically, the contrast loss function is determined by maximizing the similarity of multimodal feature positive sample vector pairs and minimizing the similarity of multimodal feature negative sample vector pairs;

[0070] Determine the cosine distance loss function according to the multimodal feature sample vector and the corresponding semantic label;

[0071] According to the contrast loss function and the cosine distance loss function, a multimodal matching loss function is constructed, and the formula is as follows:

[0072] L match =α·L InfoNCE +(1-α)·L cos , where L match represents the multimodal matching loss function, L InfoNCE Represents the contrast loss function of information aggregation, L cos represents the cosine distance loss function, and α represents the regularization coefficient.

[0073] In this process, a multimodal semantic alignment model is constructed by optimizing the representation extraction vector. The multimodal semantic alignment model can effectively separate irrelevant redundant data in the multimodal feature sample vector, thereby achieving accurate alignment of the multimodal feature sample vector and the semantic label.

[0074] Step 203, input the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical big model to obtain the prediction result of the integrated enhanced medical big model output, wherein the integrated enhanced medical big model is trained by the predicted distribution matrix obtained by predicting single-modal medical sample data by multiple source medical models and the real distribution matrix corresponding to each source medical model.

[0075] In this step, the integrated enhanced medical big model can realize the diagnosis of multiple diseases. For a detailed description of building an integrated enhanced medical big model, see Figure 4 .

[0076] In this specification, the multimodal semantic alignment model ensures the efficient collaboration between multimodal data and the medical big model by matching the embedding dimension of the medical big model, greatly improving the performance of the diagnostic task. The multimodal semantic alignment model integrates the pre-fusion driven multimodal semantic alignment mechanism and the semantic alignment method based on adaptive representation calculation. Different from the traditional semantic technology that only supports single modality and text alignment, this application can first perform feature fusion on the multimodal medical data, and then perform semantic alignment on the fused features to achieve flexible support for any modality combination. At the same time, it shows higher applicability and robustness in complex multimodal medical scenarios, greatly improving the integrated enhanced medical big model's ability to understand and integrate multimodal information, thereby enhancing the accuracy and reliability of disease diagnosis.

[0077] The method in this manual is suitable for the simultaneous input of multi-modal data, making full use of the complementary information between the modalities, improving the comprehensive understanding of multi-modal data, and significantly enhancing the performance of the model. This multi-modal feature fusion strategy effectively captures the deep semantic associations across modalities, making the processing of multi-modal data in complex medical scenarios more accurate.

[0078] Figure 3 The flowchart of a method for optimizing a multimodal semantic alignment model according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0079] Step 301, a fully connected linear projection layer is set between the multimodal semantic alignment model and the integrated enhanced medical model to match the output of the multimodal semantic alignment model with the feature dimension of the input of the integrated enhanced medical model.

[0080] In the embodiments of the present specification, the output of the multimodal semantic alignment model is the aligned multimodal feature vector. In order to ensure that the aligned multimodal feature vector matches the input feature dimension of the integrated enhanced medical model, the multimodal semantic alignment model is connected to the initial integrated enhanced medical model through a fully connected linear projection layer. The fully connected linear projection layer is used to project the output of the multimodal semantic alignment model (i.e., the aligned multimodal feature vector) into a vector with the same embedding feature dimension as the integrated enhanced medical model, so that the result of the multimodal semantic alignment model is input into the integrated enhanced medical model in the form of a high-order embedding vector.

[0081] Step 302, according to the loss function of the masked language model, optimize the multimodal semantic alignment model, determine the output of the multimodal semantic alignment model and the prediction result of the integrated enhanced medical large model output.

[0082] In this specification, the Masked Language Model (MLM) is a deep learning technology widely used in natural language processing (NLP) tasks, which can be used in model training based on the Transformer architecture. The Masked Language Model learns the deep representation of language by randomly masking or replacing some words in the input text, and trains the model to predict these masked or replaced words based on the context, thereby achieving bidirectional modeling.

[0083] In this step, the initial integrated enhanced medical model is trained by predicting the masked words through the loss function of the masked language model to improve the model's ability to understand multimodal data. The performance of the multimodal semantic alignment model is further optimized to make it work more effectively with the integrated enhanced medical model, improve the integrated enhanced medical model's understanding and application of multimodal information, and greatly improve the performance of diagnostic tasks. The loss function of masked language modeling is as follows:

[0084]

[0085] Among them, L MLM represents the loss value of masked language modeling, i represents the i-th index value, M represents the position index set of the masked word, and x i represents the actual target vocabulary corresponding to the i-th index value, x masked represents the partially masked input word sequence, p(x i ∣x masked ) indicates that the model is masked In the case of i probability.

[0086] The calculation formula of the loss function of the masked language model is as follows: For each masked position, the masked language model needs to calculate the probability of each word at that position and select the word with the highest probability as the prediction result. The loss function helps the model learn the representation of language in the pre-training stage, thereby improving its performance in downstream tasks.

[0087] Figure 4 The flowchart of a method for constructing a large medical model according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0088] Step 401 : inputting single-modality medical sample data into multiple source medical models, and obtaining prediction distribution matrices output by each source medical model.

[0089] This manual needs to handle multiple disease scenarios, and different pre-trained source medical models have their own advantages in specific disease fields. These source medical models have accumulated rich medical knowledge. Therefore, this step can use multiple source medical models that have been pre-trained in various medical fields such as disease diagnosis and treatment, medical textbooks, and medical databases to train and build an integrated enhanced medical model. By using the knowledge learned from these source medical models and further fine-tuning them in specific fields (such as major neuropsychiatric diseases), a more accurate integrated enhanced medical model based on predictive distribution representation can be constructed. Therefore, by integrating the existing medical knowledge of multiple source medical models and using multimodal fusion feature data to complete continuous training and fine-tuning optimization, the diagnosis of multiple diseases can be achieved.

[0090] In this step, the data in the multi-disease medical corpus is first used as the input of multiple source medical models. Among them, the data in the multi-disease medical corpus is mainly single-modal medical-related data. In order to build an integrated enhanced medical large model, this manual collects sample data from the multi-disease medical corpus and inputs them into multiple open source source medical models respectively, wherein the open source source medical models include but are not limited to: such as LLaVA-Med, MedPaLM, BioGPT, etc.; the collected sample data is called single-modal medical sample data. After the single-modal medical sample data is input into multiple source medical models, the corresponding prediction distribution matrix of each source medical model is obtained.

[0091] Since these source medical models use different word segmenters, the dimensions of the prediction distribution matrices may not be consistent. In order to ensure the consistency and completeness of the information, this manual uses a vocabulary alignment method based on the minimum matching cost to standardize each prediction distribution matrix, unify the size format, structure and semantics of each prediction distribution matrix, and provide a consistent input data structure for subsequent knowledge fusion. Specifically, the similarity between each prediction distribution matrix is ​​measured by comparing the cosine distance between each prediction distribution matrix as the matching cost. The word pairs with the lowest matching cost are selected for alignment one by one to ensure the consistency of the distribution matrix in structure and semantics. Among them, the smaller the minimum matching cost, the more similar the two prediction distribution matrices are; conversely, the larger the distance, the less similar the prediction distribution matrices are.

[0092] Step 402 , calculate the cross entropy between the predicted distribution matrix of each source medical model and the true distribution matrix corresponding to each source medical model, and determine the weight of each predicted distribution matrix.

[0093] Based on the standardized prediction distribution matrix, this step adopts a fusion strategy based on average cross entropy to calculate the cross entropy value between each standardized prediction distribution matrix and its corresponding label, and determines the weight value based on the cross entropy value. Among them, the label describes the true distribution matrix corresponding to the input unimodal medical sample data of each source medical model, which can be obtained at the same time as the unimodal medical sample data is collected.

[0094] Step 403, according to the weights, weight the prediction distribution matrices output by each source medical model to obtain a fused probability distribution matrix. After obtaining the weights of each prediction distribution matrix in step 302, the prediction distribution matrices of each source medical model are fused in a weighted manner to obtain a fused probability distribution matrix. This improves the accuracy and robustness of the model in disease diagnosis tasks.

[0095] Step 404, select one of the source medical models from the multiple source medical models as the initial large medical model, based on the loss value between the predicted distribution matrix output by the initial large medical model, the fused probability distribution matrix and the true distribution matrix.

[0096] Step 405, iteratively train the parameters of the initial medical model according to the loss value to construct an initial integrated enhanced medical model.

[0097] A large model is selected from multiple source medical models as the initial medical large model, and the predicted probability distribution output of each training of the initial medical large model is calculated through the loss function, so as to train and adjust the parameters of the initial medical large model so that it can integrate the knowledge of each source model to form a unified large model after knowledge fusion. According to the predicted distribution matrix output by the initial medical large model, the fused probability distribution matrix and the true distribution matrix, a fusion loss function is constructed and the loss value is calculated, and the parameters of the initial medical large model are iteratively trained according to the loss value until the number of iterations of the initial medical large model reaches a preset value; or the loss value converges to a preset threshold, it is determined that the initial medical large model has reached the training end condition, and the initial integrated enhanced medical large model is constructed.

[0098] In the embodiments of this specification, the fusion loss function aims to improve the medical semantic understanding ability of the initial medical large model by optimizing the loss function, and enhance its performance in multi-disease medical scenarios. Specifically, the fusion loss function is determined based on the difference between the true distribution matrix and the fusion probability distribution matrix, and the difference between the fusion probability distribution matrix and the predicted distribution matrix. The fusion loss function is constructed by the following formula, and the loss value of the initial medical large model training is calculated.

[0099] The formula of the fusion loss function is as follows:

[0100]

[0101] in, represents the difference measure between two matrices, implemented using relative entropy,

[0102] represents the expected value of text t in the multi-disease medical corpus C, O t represents the true distribution matrix of the initial medical model for text t, Q t represents the prediction distribution matrix of the initial medical model for the output of text t, P t Represents the fused probability distribution matrix. The fusion loss function combines the difference between the true distribution matrix and the fused probability distribution matrix, and the difference between the fused probability distribution matrix and the predicted distribution matrix of the initial medical model, thereby achieving the fitting of the true label distribution and the learning of the fused distribution in the optimization process.

[0103] Step 406, fine-tune the initial integrated enhanced medical large model to obtain an integrated enhanced medical large model.

[0104] In this specification, an adaptive reduced-rank quantization fine-tuning training strategy is used to reduce the number of parameters that need to be adjusted for fine-tuning the initial integrated enhanced medical model by performing low-rank decomposition and quantization on the model parameters, thereby optimizing the integrated enhanced medical model. For a detailed description of fine-tuning the initial integrated enhanced medical model in this step, see Figure 5 .

[0105] Figure 5 The flowchart of a method for fine-tuning an integrated enhanced medical model according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0106] Step 501, using singular value decomposition to decompose the pre-frozen weight matrix of the initial integrated enhanced medical large model to obtain the decomposed singular scale vector and singular direction matrix.

[0107] In this step, first Figure 4 The weight matrix W0 of the integrated enhanced medical model obtained by pre-training is frozen. After freezing, the weight matrix W0 is decomposed into a singular scale vector m and a singular direction matrix by singular value decomposition, as shown in the following formula:

[0108] W0=U·diag(m)·V, where m represents the singular scale vector, U and V represent the left singular value matrix and the right singular value matrix obtained by singular value decomposition, respectively, where the right singular value matrix is ​​also called the singular direction matrix. diag(·) represents the diagonal matrix operator. In this step, the weight matrix of the initial integrated enhanced medical large model is decomposed to achieve independent adjustment of scale weights and direction weights, capture the global characteristics of the weight matrix, reveal the low-rank structure and directional orthogonality of the weight matrix, improve numerical stability, and be more efficient and reliable.

[0109] Step 502, freeze the singular direction matrix, generate multiple basic low-rank matrices and multiple intermediate matrices, control the number of low-rank matrices and the values ​​of low-rank matrices, and decompose the singular direction matrix into at least one low-rank matrix.

[0110] Before freezing the singular direction matrix in this step, the method further includes quantizing the right singular value matrix (singular direction matrix). Specifically, considering the huge number of parameters of the singular direction matrix V, parameter quantization is used to quantize it, as shown in the following formula:

[0111] Where V represents the singular direction matrix, represents the singular direction matrix after parameter quantization, s0 represents the scale factor, and its value is z0 represents the bias quantization value, which is 7 in 4-bit quantization. clamp(x,A,B) limits the value of x to between A and B. [·] is the rounding operation. This formula means The value is limited to 0 to 2 4 -1.

[0112] After quantizing the parameters, the singular direction matrix After freezing, low-rank matrix decomposition is further achieved by controlling the number of low-rank matrices n and the value of the rank r of each low-rank matrix. Through this method, two basic low-rank matrices A and B are first generated, and n intermediate matrices X are introduced on this basis. i (i=1,...,n), the decomposed matrix is ​​much smaller than the dimension of the original weight matrix, thus reflecting the low-rank characteristic.

[0113] Step 503: adjust the parameters of the low-rank matrix to obtain an adjusted singular value matrix.

[0114] By adjusting A, B, X i The parameters of the matrix without having to adjust the entire The parameters of the matrix reduce the computational cost. The adjusted singular direction matrix is

[0115]

[0116] Decomposed parameter matrices A, B, X i The parameter values ​​are initialized by Gaussian distribution. This decomposition method allows the degree of freedom of decomposition to be flexibly controlled by adjusting the number of intermediate matrices n, and significantly reduces the storage and computational costs of the weight matrix.

[0117] Step 504, merging the singular scale vector, the singular direction matrix after low rank adjustment, and the singular direction matrix after parameter quantization to generate an updated weight matrix.

[0118] The singular scale vector m , the low-rank adjusted singular direction matrix And the parameterized singular direction matrix Merge to generate the final updated weight matrix W′. The updated weight matrix not only contains the main information of the original weight matrix, but also effectively introduces the parameter optimization captured by low-rank decomposition. The specific formula for parameter update is as follows:

[0119] W′ represents the updated weight matrix, represents the low-rank adjusted singular direction matrix, represents the low-rank adjusted singular direction matrix.

[0120] In this specification, an adaptive reduced-rank quantization fine-tuning training strategy is used to reduce the number of parameters that need to be adjusted for fine-tuning the integrated enhanced medical large model by performing low-rank decomposition and quantization on the model parameters, optimize the integrated enhanced medical large model, greatly reduce the consumption of computing resources and storage space, and improve the efficiency of fine-tuning training. In a resource-constrained environment, it can significantly shorten the fine-tuning time and computing cost while maintaining model performance. It greatly reduces the hardware requirements in the optimization process of the integrated enhanced medical large model, improves the application efficiency of the integrated enhanced medical large model in a dynamic medical environment, and makes large-scale deployment and clinical application more sustainable.

[0121] like Figure 6 The figure shows a schematic diagram of the structure of a multimodal medical large model application device according to an embodiment of the present specification. The figure describes the basic structure of the multimodal medical large model application device, wherein the functional units and modules can be implemented by software, or by using a general chip or a specific chip to implement the multimodal medical large model application. The device specifically includes:

[0122] An acquisition unit 601 is used to acquire multimodal feature data;

[0123] The prediction unit 602 is used to input the multimodal feature data into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model includes a multimodal semantic alignment model and an integrated enhanced medical big model. The multimodal semantic alignment model is obtained by training an initial multimodal semantic alignment model with multimodal feature sample vectors, representation extraction sample vectors and semantic labels corresponding to the multimodal feature sample vectors. The integrated enhanced medical big model is trained with a predicted distribution matrix obtained by predicting single-modal medical sample data using multiple source medical models and a true distribution matrix corresponding to each source medical model.

[0124] like Figure 7 The figure shows a schematic diagram of the structure of a multimodal medical large model fine-tuning device in an embodiment of the present specification. The figure describes the basic structure of the multimodal medical large model fine-tuning device, wherein the functional units and modules can be implemented by software, or a general chip or a specific chip can be used to implement the construction of a multimodal medical large model. The device specifically includes:

[0125] The first prediction result acquisition unit 701 is used to input the multimodal feature sample vector, the representation extraction sample vector and the semantic label corresponding to the multimodal feature sample vector into the initial multimodal semantic alignment model to obtain the prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features in the multimodal feature sample vector;

[0126] A multimodal semantic alignment model construction unit 702 is used to calculate a multimodal matching loss function according to the prediction result and the semantic label, and iteratively update the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function to construct a multimodal semantic alignment model;

[0127] The second prediction result acquisition unit 703 is used to input the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical big model to obtain the prediction result output by the integrated enhanced medical big model, wherein the integrated enhanced medical big model is trained by the prediction distribution matrix obtained by predicting single-modal medical sample data with multiple source medical models and the real distribution matrix corresponding to each source medical model.

[0128] Figure 8 The figure shows a schematic diagram of constructing an initial integrated enhanced medical model based on a medical source model in an embodiment of the specification. In the figure, the medical model A, the medical model B and the medical model C are three pre-trained medical knowledge models in various medical fields such as disease diagnosis and treatment, medical teaching materials, and medical databases, and multimodal fusion feature data are used to complete continuous training and fine-tuning optimization to achieve diagnosis of multiple diseases, so as to build a more accurate medical diagnosis model.

[0129] Fig. 9 Shown is a schematic diagram of an updated weight matrix of an integrated enhanced medical large model according to an embodiment of this specification. The weight matrix of the initial integrated enhanced medical large model is decomposed into a singular scale vector and a singular direction matrix by singular value decomposition. Secondly, two basic low-rank matrices A and B are generated, and n intermediate matrices are introduced on this basis. Low-rank matrix decomposition is achieved by controlling the number of introduced intermediate matrices and the rank of each low-rank matrix. The degree of freedom of decomposition is flexibly controlled to reduce the storage and computational costs of the weight matrix. Finally, the singular scale vector, the singular direction matrix and the singular direction matrix updated with low-rank parameters are merged to generate the final updated weight matrix. The updated weight matrix not only contains the main information of the original weight matrix, but also effectively introduces the parameter optimization captured by low-rank decomposition, thereby achieving fine-tuning of the initial integrated enhanced medical large model.

[0130] Fig.10 The figure shows a schematic diagram of a multimodal semantic alignment model based on adaptive fusion in an embodiment of the present specification. The model integrates a pre-fusion driven multimodal semantic alignment mechanism and a semantic alignment method based on adaptive representation calculation, so that the multimodal feature vector is semantically aligned with the input space of the integrated enhanced medical large model.

[0131] Fig.11The diagram shows a schematic diagram of a computer device provided by an embodiment of the present specification. The multimodal medical large model application and fine-tuning method described in the present application can be applied to the computer device. The computer device 1102 may include one or more processors 1104, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 1102 may also include any memory 1106, which is used to store any kind of information such as code, settings, data, etc. Non-limiting, for example, the memory 1106 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 1102. In one case, when the processor 1104 executes an associated instruction stored in any memory or combination of memories, the computer device 1102 may perform any operation of the associated instruction. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0132] The computer device 1102 may also include an input / output module 1110 (I / O) for receiving various inputs (via input devices 1112) and for providing various outputs (via output devices 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface (GUI) 1118. In other embodiments, the input / output module 1110 (I / O), input device 1112, and output device 1114 may not be included, and the computer device 1102 may be used as a computer device in a network. The computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.

[0133] The communication link 1122 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0134] Corresponds to Figures 1 to 5 Method, the embodiment of the specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0135] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 5 The method shown.

[0136] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0137] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0138] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0142] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0144] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A multimodal medical large model application method, characterized in that: The method comprises: Acquire multimodal feature data; The multimodal feature data is input into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model includes a multimodal semantic alignment model and an integrated enhanced medical big model. The multimodal semantic alignment model is obtained by training an initial multimodal semantic alignment model with multimodal feature sample vectors, representation extraction sample vectors and semantic labels corresponding to the multimodal feature sample vectors. The integrated enhanced medical big model is obtained by training a plurality of source medical models based on a predicted distribution matrix obtained by predicting single-modal medical sample data and a true distribution matrix corresponding to each source medical model.

2. A multimodal medical large model fine-tuning method, characterized in that: The method comprises: Inputting the multimodal feature sample vector and the representation extraction sample vector into the initial multimodal semantic alignment model to obtain the prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features from the multimodal feature sample vector; Calculating a multimodal matching loss function according to the prediction result and the semantic label corresponding to the multimodal feature sample vector, iteratively updating the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function, and constructing a multimodal semantic alignment model; The output of the multimodal semantic alignment model is input into a pre-trained integrated enhanced medical big model to obtain the prediction result of the integrated enhanced medical big model output, wherein the integrated enhanced medical big model is trained by the predicted distribution matrix obtained by predicting single-modal medical sample data by multiple source medical models and the true distribution matrix corresponding to each source medical model.

3. The method according to claim 2, characterized in that The method further comprises: A fully connected linear projection layer is set between the multimodal semantic alignment model and the integrated enhanced medical model to match the output of the multimodal semantic alignment model with the feature dimension of the input of the integrated enhanced medical model; According to the loss function of the masked language model, the multimodal semantic alignment model is optimized to determine the output of the multimodal semantic alignment model and the prediction results of the integrated enhanced medical large model output.

4. The method according to claim 2, characterized in that: Determine the representation extraction sample vector according to the following formula: v=u T (X-μ); where v represents the characterization sample extraction vector, X represents the multimodal feature sample vector, μ represents the feature center vector, and u represents the main feature direction vector.

5. The method according to claim 2, characterized in that: The multimodal matching loss function is constructed as follows: Determine the contrast loss function by maximizing the similarity of multimodal feature positive sample vector pairs and minimizing the similarity of multimodal feature negative sample vector pairs; Determine the cosine distance loss function according to the multimodal feature sample vector and the corresponding semantic label; According to the contrast loss function and the cosine distance loss function, a multimodal matching loss function is constructed, and the formula is as follows: L match =α·L InfoNCE +(1-α)·L cos , where L match represents the multimodal matching loss function, L InfoNCE Represents the contrast loss function of information aggregation, L cos represents the cosine distance loss function, and α represents the regularization coefficient.

6. The method according to claim 2, characterized in that The integrated enhanced medical model is constructed in the following way: Inputting the single-modality medical sample data into multiple source medical models, and obtaining the prediction distribution matrix output by each source medical model; Calculate the cross entropy between the predicted distribution matrix of each source medical model and the true distribution matrix corresponding to each source medical model, and determine the weight of each predicted distribution matrix; According to the weights, the prediction distribution matrices output by each source medical model are weighted to obtain a fusion probability distribution matrix; From the multiple source medical models, one of the source medical models is selected as the initial medical large model, and the loss value between the predicted distribution matrix output by the initial medical large model, the fused probability distribution matrix and the true distribution matrix is ​​determined; Iteratively training the parameters of the initial medical model according to the loss value to construct an initial integrated enhanced medical model; The initial integrated enhanced medical model is fine-tuned to obtain an integrated enhanced medical model.

7. The method according to claim 6, characterized in that The loss value between the predicted distribution matrix output by the initial medical large model, the fused probability distribution matrix and the true distribution matrix is ​​calculated as follows: A fusion loss function is determined according to the difference between the true distribution matrix and the fused probability distribution matrix, and the difference between the fused probability distribution sample matrix and the predicted distribution matrix.

8. The method according to claim 6, characterized in that Fine-tuning the initial integrated enhanced medical model to obtain an integrated enhanced medical model includes: Decomposing the weight matrix of the initial integrated enhanced medical large model in advance by singular value to obtain the decomposed singular scale vector and singular direction matrix; Freezing the singular direction matrix, generating a plurality of basic low-rank matrices and a plurality of intermediate matrices, controlling the number of low-rank matrices and the values ​​of the low-rank matrices, and decomposing the singular direction matrix into at least one low-rank matrix; Adjusting the parameters of the low-rank matrix to obtain a low-rank adjusted singular value matrix; The singular scale vector, the adjusted singular direction matrix, and the parameter-quantized singular direction matrix are combined to generate an updated weight matrix.

9. A multi-modal medical large model application device, characterized in that: The device comprises: An acquisition unit, used for acquiring multimodal feature data; A prediction unit is used to input multimodal feature data into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model includes a multimodal semantic alignment model and an integrated enhanced medical big model, the multimodal semantic alignment model is obtained by training an initial multimodal semantic alignment model with multimodal feature sample vectors, representation extraction sample vectors and semantic labels corresponding to the multimodal feature sample vectors, and the integrated enhanced medical big model is obtained by training a plurality of source medical models based on a predicted distribution matrix obtained by predicting single-modal medical sample data and a true distribution matrix corresponding to each source medical model.

10. A multi-modal medical large model fine-tuning device, characterized in that: The device comprises: A first prediction result acquisition unit is used to input a multimodal feature sample vector, a representation extraction sample vector and a semantic label corresponding to the multimodal feature sample vector into an initial multimodal semantic alignment model to obtain a prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features from the multimodal feature sample vector; A multimodal semantic alignment model construction unit, used to calculate a multimodal matching loss function according to the prediction result and the semantic label, iteratively update the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function, and construct a multimodal semantic alignment model; The second prediction result acquisition unit is used to input the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical big model to obtain the prediction result output by the integrated enhanced medical big model, wherein the integrated enhanced medical big model is trained by the prediction distribution matrix obtained by predicting single-modal medical sample data by multiple source medical models and the real distribution matrix corresponding to each source medical model.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images