Multimodal medical large model application and fine-tuning method and device and computer equipment

Abstract

The present specification relates to the technical field of natural language processing, in particular to a multi-modal medical large model application and fine-tuning method, device and computer equipment, the method comprising: acquiring multi-modal feature data; inputting the multi-modal feature data into a pre-trained multi-modal medical large model to obtain an output result, wherein the multi-modal medical large model comprises a multi-modal semantic alignment model and an integrated enhanced medical large model, and the multi-modal semantic alignment model is obtained by training an initial multi-modal semantic alignment model with a multi-modal feature sample vector, a feature extraction sample vector and a semantic label corresponding to the multi-modal feature sample vector. The present application integrates the knowledge and ability of multiple source medical models, reduces the dependence on data resources, improves the generalization ability of the model, and enhances the accuracy and reliability of diagnosis; multi-modal features are extracted and the semantic alignment effect is optimized to avoid information loss and semantic drift.

Description

Technical Field

[0001] This specification relates to the field of natural language processing technology, in particular to multimodal medical large model application and fine-tuning methods, devices and computer equipment. Background Art

[0002] Artificial intelligence technology has shown broad application prospects in the field of healthcare. As a representative of artificial intelligence in the medical field, multimodal medical big models have the potential to process complex data and provide precise diagnosis and treatment plans. They can integrate a variety of medical data, including imaging, genes, electronic medical records, etc., to support disease diagnosis, risk prediction and personalized treatment. With the rapid development of medical big models, their parameter scale has expanded from hundreds of millions to tens of billions or even hundreds of billions, enabling a deeper understanding of the mechanisms and individual differences of neuropsychiatric diseases, discovering new disease markers and predicting disease risks and prognosis. This technological advancement provides strong support for precision medicine and personalized intervention for neuropsychiatric diseases, greatly promotes the progress of medical research and clinical applications, and also brings new possibilities for the early detection, diagnosis and treatment of neuropsychiatric diseases.

[0003] However, the training and fine-tuning of large medical models also faces unprecedented challenges. As the model scales, the required computing and data resources increase exponentially. High-dimensional medical data often has complex structures and diversity, requiring a large amount of computing power to process. Especially in resource-limited environments, obtaining sufficient high-performance computing devices, such as GPUs, becomes increasingly difficult. In addition, the scarcity and high annotation costs of multimodal medical data further increase the difficulty of building large medical models. Therefore, how to optimize the use of computing resources and effectively fine-tune large medical models without sacrificing model performance has become an important research direction. The continued development of large multimodal medical models will bring more possibilities to precision medicine, help deepen the understanding of disease mechanisms, discover new diagnostic markers, and promote the application of personalized medicine and intervention methods. Summary of the Invention

[0004] To solve the problems in the prior art, this specification provides a multimodal medical large model application and fine-tuning method, device and computer equipment.

[0005] An embodiment of the present specification provides a method for applying a multimodal medical big model, the method comprising: obtaining multimodal feature data; inputting the multimodal feature data into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model comprises a multimodal semantic alignment model and an integrated enhanced medical big model, the multimodal semantic alignment model being obtained by training an initial multimodal semantic alignment model using multimodal feature sample vectors, representation extraction sample vectors, and semantic labels corresponding to the multimodal feature sample vectors, and the integrated enhanced medical big model being obtained by training a plurality of source medical models based on a predicted distribution matrix obtained by predicting single-modal medical sample data and a true distribution matrix corresponding to each source medical model.

[0006] An embodiment of the present specification provides a method for fine-tuning a multimodal medical large model, the method comprising: inputting a multimodal feature sample vector and a representation extraction sample vector into an initial multimodal semantic alignment model to obtain a prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features in the multimodal feature sample vector; calculating a multimodal matching loss function based on the prediction result and the semantic label corresponding to the multimodal feature sample vector, iteratively updating the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function, and constructing a multimodal semantic alignment model; inputting the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical large model to obtain a prediction result output by the integrated enhanced medical large model, wherein the integrated enhanced medical large model is trained by a prediction distribution matrix obtained by predicting single-modal medical sample data by multiple source medical models and a true distribution matrix corresponding to each source medical model.

[0007] According to one aspect of the embodiments of this specification, the method further includes: setting a fully connected linear projection layer between the multimodal semantic alignment model and the integrated enhanced medical large model to match the output of the multimodal semantic alignment model with the feature dimensions of the input of the integrated enhanced medical large model; optimizing the multimodal semantic alignment model according to the loss function of the masked language model to determine the prediction results of the output of the multimodal semantic alignment model and the output of the integrated enhanced medical large model.

[0008] According to one aspect of the embodiment of this specification, the characterization extraction sample vector is determined according to the following formula: v = u T (X-μ); where v represents the characterization sample extraction vector, X represents the multimodal feature sample vector, μ represents the feature center vector, and u represents the main feature direction vector.

[0009] According to one aspect of an embodiment of this specification, a multimodal matching loss function is constructed in the following manner: a contrast loss function is determined by maximizing the similarity of a multimodal feature positive sample vector pair and minimizing the similarity of a multimodal feature negative sample vector pair; a cosine distance loss function is determined based on the multimodal feature sample vector and the corresponding semantic label; a multimodal matching loss function is constructed based on the contrast loss function and the cosine distance loss function, and the formula is as follows:

[0010] L match =α·L InfoNCE +(1-α)·L cos , where L match represents the multimodal matching loss function, L InfoNCE Represents the contrast loss function of information aggregation, L cos represents the cosine distance loss function, and α represents the regularization coefficient.

[0011] According to one aspect of the embodiments of this specification, the integrated enhanced medical big model is constructed in the following manner: single-modality medical sample data is input into multiple source medical models to obtain the predicted distribution matrix output by each source medical model; the cross entropy between the predicted distribution matrix of each source medical model and the true distribution matrix corresponding to each source medical model is calculated, and the weight of each predicted distribution matrix is ​​determined; according to the weight, the predicted distribution matrix output by each source medical model is weighted to obtain a fused probability distribution sample matrix; from the multiple source medical models, one of the source medical models is selected as the initial medical big model, and according to the loss value between the predicted distribution matrix output by the initial medical big model, the fused probability distribution matrix and the true distribution matrix, the parameters of the initial medical big model are iteratively trained according to the loss value to construct the initial integrated enhanced medical big model; the initial integrated enhanced medical big model is fine-tuned to obtain the integrated enhanced medical big model.

[0012] According to one aspect of an embodiment of the present specification, the loss value between the predicted distribution matrix output by the initial medical large model, the fused probability distribution matrix and the true distribution matrix is ​​calculated in the following manner: a fusion loss function is determined based on the difference between the true distribution matrix and the fused probability distribution matrix, and the difference between the fused probability distribution matrix and the predicted distribution matrix.

[0013] According to one aspect of the embodiments of this specification, the method further includes: using singular value decomposition to decompose the pre-frozen weight matrix of the initial integrated enhanced medical large model to obtain the decomposed singular scale vector and singular direction matrix; freezing the singular direction matrix, generating multiple basic low-rank matrices and multiple intermediate matrices, controlling the number of low-rank matrices and the values ​​of low-rank matrices, and decomposing the singular direction matrix into at least one low-rank matrix; adjusting the parameters of the low-rank matrix to obtain a low-rank adjusted singular value matrix; merging the singular scale vector, the adjusted singular direction matrix, and the singular direction matrix after parameter quantization to generate an updated weight matrix.

[0014] An embodiment of the present specification provides a multimodal medical big model application device, which includes: an acquisition unit for acquiring multimodal feature data; a prediction unit for inputting the multimodal feature data into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model includes a multimodal semantic alignment model and an integrated enhanced medical big model, the multimodal semantic alignment model is obtained by training an initial multimodal semantic alignment model with multimodal feature sample vectors, representation extraction sample vectors and semantic labels corresponding to the multimodal feature sample vectors, and the integrated enhanced medical big model is obtained by training multiple source medical models based on the predicted distribution matrix obtained by predicting single-modal medical sample data and the true distribution matrix corresponding to each source medical model.

[0015] An embodiment of the present specification also provides a multimodal medical large model fine-tuning device, which includes: a first prediction result acquisition unit, which is used to input a multimodal feature sample vector, a representation extraction sample vector and a semantic label corresponding to the multimodal feature sample vector into an initial multimodal semantic alignment model to obtain a prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features in the multimodal feature sample vector; a multimodal semantic alignment model construction unit, which is used to calculate a multimodal matching loss function based on the prediction result and the semantic label, iteratively update the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function, and construct a multimodal semantic alignment model; a second prediction result acquisition unit, which is used to input the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical large model to obtain a prediction result output by the integrated enhanced medical large model, wherein the integrated enhanced medical large model is trained by a prediction distribution matrix obtained by predicting single-modal medical sample data by multiple source medical models and a true distribution matrix corresponding to each source medical model.

[0016] An embodiment of this specification also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multimodal medical large model application and fine-tuning method is implemented.

[0017] The embodiments of this specification also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the multimodal medical large model application and fine-tuning method.

[0018] By integrating the knowledge and capabilities of multiple source medical models, the present invention reduces dependence on data resources, effectively improves the generalization ability of the model, and enhances the accuracy and reliability of diagnosis; by extracting multimodal features and optimizing their semantic alignment, it avoids the problems of information loss and semantic deviation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 Shown is a flow chart of a multimodal medical large model application method according to an embodiment of this specification;

[0021] Figure 2 Shown is a flow chart of a multimodal medical large model fine-tuning method according to an embodiment of this specification;

[0022] Figure 3 Shown is a flow chart of a method for optimizing a multimodal semantic alignment model according to an embodiment of this specification;

[0023] Figure 4 The figure shows a flow chart of a method for constructing a large medical model according to an embodiment of this specification;

[0024] Figure 5 The figure shows a flow chart of a method for fine-tuning an initial integrated enhanced medical large model according to an embodiment of this specification;

[0025] Figure 6 The figure shows a schematic diagram of the structure of a multimodal medical large model application device according to an embodiment of this specification;

[0026] Figure 7 The figure shows a schematic structural diagram of a multimodal large medical model fine-tuning device according to an embodiment of this specification;

[0027] Figure 8 The figure shows a schematic diagram of building an integrated enhanced medical large model based on a medical source model according to an embodiment of this specification;

[0028] Figure 9 The figure shows a schematic diagram of updating the weight matrix of the integrated enhanced medical model according to an embodiment of the present specification;

[0029] Figure 10 FIG2 is a schematic diagram of a multimodal semantic alignment model based on adaptive fusion according to an embodiment of the present specification;

[0030] Figure 11 The figure shows a schematic diagram of the structure of a computer device according to an embodiment of the present specification.

[0031] Description of the accompanying symbols:

[0032] 601, acquisition unit;

[0033] 602, prediction unit;

[0034] 701. First prediction result obtaining unit;

[0035] 702. Multimodal semantic alignment model construction unit;

[0036] 703. Second prediction result obtaining unit;

[0037] 1102. Computer equipment;

[0038] 1104, processor;

[0039] 1106. Memory;

[0040] 1108, driving mechanism;

[0041] 1110, input / output module;

[0042] 1112. Input device;

[0043] 1114. Output device;

[0044] 1116. Presentation equipment;

[0045] 1118. Graphical User Interface;

[0046] 1120, network interface;

[0047] 1122, communication link;

[0048] 1124. Communication bus. DETAILED DESCRIPTION

[0049] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative work are within the scope of protection of this specification.

[0050] It should be noted that the terms "first," "second," and the like in the description and claims of this specification and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0051] This specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many orderings and does not represent the only execution order. When a system or device product is actually executed, the method can be executed in the order shown in the embodiments or the drawings or in parallel.

[0052] It should be noted that the multimodal medical large model application and fine-tuning method of this specification can be used in the field of natural language processing technology, as well as in the field of medical technology. This specification does not limit the application field of the multimodal medical large model application and fine-tuning method.

[0053] Figure 1 The flowchart of a multimodal medical large model application method according to an embodiment of this specification is shown, which specifically includes the following steps:

[0054] Step 101: Acquire multimodal feature data.

[0055] In this step, the multimodal feature data is in vector form and therefore can also be referred to as a multimodal feature vector. In this step, multimodal medical data is acquired and pre-fused into a multimodal feature vector to obtain multimodal feature data.

[0056] Step 102: Input the multimodal feature data into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model includes a multimodal semantic alignment model and an integrated enhanced medical big model. The multimodal semantic alignment model is obtained by training an initial multimodal semantic alignment model with multimodal feature sample vectors, representation extraction sample vectors, and semantic labels corresponding to the multimodal feature sample vectors. The integrated enhanced medical big model is obtained by training a predicted distribution matrix obtained by predicting single-modal medical sample data using multiple source medical models and a true distribution matrix corresponding to each source medical model.

[0057] In this step, the multimodal semantic alignment model and the integrated enhanced medical model are both pre-trained models. Among them, by integrating the knowledge and capabilities of multiple source medical models, adopting a fusion strategy of predictive distribution representation, training and building into the integrated enhanced learning model, the dependence on data resources is significantly reduced, ensuring the model's learning of complex disease characteristics, improving the generalization ability of the integrated enhanced learning model, and enhancing the accuracy and reliability of diagnosis. In order for the integrated enhanced medical model to understand the multimodal feature data and perform subsequent diagnostic tasks, it is necessary to semantically align the multimodal feature data with the input space of the integrated enhanced medical model. In this specification, the multimodal semantic alignment model can effectively separate irrelevant redundant data in the multimodal feature data, thereby achieving accurate alignment of the multimodal feature data with the data semantic labels corresponding to the multimodal feature data.

[0058] Figure 2 The flowchart of a multimodal medical large model fine-tuning method according to an embodiment of this specification is shown, which specifically includes the following steps:

[0059] Step 201: Input the multimodal feature sample vector and the representation extraction sample vector into the initial multimodal semantic alignment model to obtain the prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features from the multimodal feature sample vector.

[0060] In this step, a multimodal semantic alignment model is constructed. This alignment model integrates the pre-fusion-driven multimodal semantic alignment mechanism and the semantic alignment method based on adaptive representation calculation, aiming to semantically align the multimodal feature sample vector with the input space of the large medical model.

[0061] In this step, first obtain multimodal medical sample data, pre-fuse the multimodal medical sample data into a multimodal feature sample vector, determine the multimodal semantic label corresponding to the multimodal feature sample vector, and combine the multimodal feature sample vector and the multimodal semantic label in pairs to participate in the pre-training of the initial multimodal semantic alignment model to learn the association between multimodal features and multimodal semantic labels. Among them, the initial multimodal semantic alignment model adopts a Transformer structure, which includes a feature encoder and a label encoder. Specifically, the multimodal feature sample vector and the representation extraction sample vector are input into the feature encoder, and through the interaction of the cross-attention mechanism, a more semantically relevant prediction result is extracted and generated, which is the multimodal feature sample representation. Figure 10 As shown, Figure 10 The multimodal fusion feature vector and the representation extraction vector are input into the feature encoder, wherein the multimodal fusion feature vector and the representation extraction vector participate in the operation together in the cross attention layer of the feature encoder, and then the operation result is output in the feedforward layer.

[0062] In the embodiments of this specification, a multimodal feature sample vector can be a vector composed of data from different modalities, such as text, voice, and video. For example, a frame of a liver surgery video clip containing live audio and text descriptions can have a corresponding semantic label that includes key lesion information in the surgical video clip, including but not limited to: lesion location, lesion shape, color characteristics, and lesion name.

[0063] In the embodiments of this specification, the characterization extraction sample vector is a random initial sample vector used to extract features of the multimodal feature sample vector. It can be any one or a combination of text features, voice features, or video features in the above-mentioned liver surgery video clip example. Furthermore, the characterization extraction sample vector is learnable and can be optimized. Specifically, the characterization extraction sample vector is optimized by the following formula:

[0064] v=u T (X-μ); where v represents the characterization sample extraction vector, X represents the multimodal feature sample vector, μ represents the feature center sample vector, and u represents the main feature direction sample vector. The feature center sample vector μ is the mean of the multimodal feature sample vector X, and the calculation formula is: The main eigendirection sample vector u is the first column of the left singular value matrix obtained by performing singular value decomposition on X-μ. The main eigendirection sample vector corresponds to the eigenvector with the largest eigenvalue and captures the direction with the largest variance in the data.

[0065] In this step, the representation extraction sample vector and the multimodal feature sample vector are input into the initial multimodal semantic alignment model together to effectively separate irrelevant redundant data in the multimodal feature sample vector.

[0066] Step 202: Calculate a multimodal matching loss function based on the prediction result and the semantic label corresponding to the multimodal feature sample vector, iteratively update the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function, and construct a multimodal semantic alignment model.

[0067] In this step, if Figure 10 As shown in the figure, the multimodal semantic label is used as the input of the label encoder to generate a vector that matches the output of the feature encoder. Furthermore, a multimodal matching loss function is introduced on the output of the feature encoder and label encoder to learn a mapping relationship that better matches the multimodal feature sample representation with the semantic label, so that the representation vector can effectively capture the characteristics of the multimodal sample vector.

[0068] The multimodal matching loss function combines contrastive loss and cosine distance loss. Specifically, the core of contrastive loss is to maximize the similarity of multimodal feature positive sample vector pairs and minimize the similarity of negative sample pairs, while further enhancing the consistency of feature vectors through cosine distance.

[0069] Specifically, the contrast loss function is determined by maximizing the similarity of multimodal feature positive sample vector pairs and minimizing the similarity of multimodal feature negative sample vector pairs;

[0070] Determine the cosine distance loss function based on the multimodal feature sample vector and the corresponding semantic label;

[0071] Based on the contrast loss function and the cosine distance loss function, a multimodal matching loss function is constructed, and the formula is as follows:

[0072] L match =α·L InfoNCE +(1-α)·L cos , where L match represents the multimodal matching loss function, L InfoNCE Represents the contrast loss function of information aggregation, L cos represents the cosine distance loss function, and α represents the regularization coefficient.

[0073] In this process, a multimodal semantic alignment model is constructed by optimizing the representation extraction vector. The multimodal semantic alignment model can effectively separate irrelevant redundant data in the multimodal feature sample vector, thereby achieving precise alignment of the multimodal feature sample vector and the semantic label.

[0074] Step 203: Input the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical big model to obtain the prediction result output by the integrated enhanced medical big model, wherein the integrated enhanced medical big model is trained by the predicted distribution matrix obtained by predicting single-modal medical sample data by multiple source medical models and the true distribution matrix corresponding to each source medical model.

[0075] In this step, the integrated enhanced medical model can realize the diagnosis of multiple diseases. For a detailed description of building the integrated enhanced medical model, see Figure 4 .

[0076] In this specification, the multimodal semantic alignment model ensures the efficient collaboration of multimodal data and medical big models by matching the embedding dimensions of the medical big model, which greatly improves the performance of the diagnostic task. The multimodal semantic alignment model integrates the pre-fusion driven multimodal semantic alignment mechanism and the semantic alignment method based on adaptive representation calculation. Different from the traditional semantic technology that only supports single modality and text alignment, this application can first perform feature fusion on multimodal medical data, and then perform semantic alignment on the fused features to achieve flexible support for any modality combination. At the same time, it shows higher applicability and robustness in complex multimodal medical scenarios, greatly improving the integrated enhanced medical big model's ability to understand and integrate multimodal information and enhance the accuracy and reliability of disease diagnosis.

[0077] The method described in this manual accommodates the simultaneous input of multimodal data, fully leveraging the complementary information between modalities to enhance comprehensive understanding of multimodal data and significantly enhance model performance. This multimodal feature fusion strategy effectively captures deep semantic connections across modalities, enabling more accurate processing of multimodal data in complex medical scenarios.

[0078] Figure 3 The flowchart of a method for optimizing a multimodal semantic alignment model according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0079] Step 301: A fully connected linear projection layer is set between the multimodal semantic alignment model and the integrated enhanced medical model to match the feature dimensions of the output of the multimodal semantic alignment model with the input of the integrated enhanced medical model.

[0080] In the embodiments of this specification, the output of the multimodal semantic alignment model is the aligned multimodal feature vector. In order to ensure that the aligned multimodal feature vector matches the input feature dimension of the integrated enhanced medical model, the multimodal semantic alignment model and the initial integrated enhanced medical model are connected through a fully connected linear projection layer. The fully connected linear projection layer is used to project the output of the multimodal semantic alignment model (i.e., the aligned multimodal feature vector) into a vector with the same embedding feature dimension as the integrated enhanced medical model, so that the result of the multimodal semantic alignment model is input into the integrated enhanced medical model in the form of a high-order embedding vector.

[0081] Step 302: Optimize the multimodal semantic alignment model according to the loss function of the masked language model, and determine the output of the multimodal semantic alignment model and the prediction results of the integrated enhanced medical model output.

[0082] In this document, the Masked Language Model (MLM) is a deep learning technique widely used in natural language processing (NLP) tasks and can be used to train models based on the Transformer architecture. The MLM randomly masks or replaces words in the input text and trains the model to predict these masked or replaced words based on context, thereby learning a deep representation of language and achieving bidirectional modeling.

[0083] In this step, the initial ensemble augmented medical model is trained by predicting masked words using the loss function of the masked language model, improving the model's ability to understand multimodal data. The performance of the multimodal semantic alignment model is further optimized to enable it to work more effectively with the ensemble augmented medical model, enhancing its understanding and application of multimodal information and significantly improving the performance of diagnostic tasks. The loss function for masked language modeling is as follows:

[0084]

[0085] Among them, L MLM Represents the loss value of masked language modeling, i represents the i-th index value, M represents the position index set of the masked word, x i Indicates the target actual vocabulary corresponding to the i-th index value, x masked represents the partially masked input word sequence, p(x i ∣x masked ) indicates that the model is masked In the case of i probability.

[0086] The loss function of the masked language model is calculated as follows: For each masked position, the masked language model calculates the probability of each word at that position and selects the word with the highest probability as the prediction result. This loss function helps the model learn language representations during pre-training, thereby improving its performance in downstream tasks.

[0087] Figure 4 The flowchart of a method for constructing a large medical model according to an embodiment of this specification is shown, which specifically includes the following steps:

[0088] Step 401: Input single-modality medical sample data into multiple source medical models to obtain the prediction distribution matrix output by each source medical model.

[0089] This manual needs to handle multiple disease scenarios, and different pre-trained source medical models have their own advantages in specific disease fields. These source medical models have accumulated rich medical knowledge. Therefore, this step can use multiple source medical models that have been pre-trained in various medical fields such as disease diagnosis and treatment, medical textbooks, and medical databases to train and build an integrated enhanced medical model. By utilizing the knowledge learned from these source medical models and further fine-tuning them in specific fields (such as major neuropsychiatric diseases), a more accurate integrated enhanced medical model based on predictive distribution representation can be constructed. Therefore, by integrating the existing medical knowledge of multiple source medical models and using multimodal fusion feature data to complete continuous training and fine-tuning optimization, the diagnosis of multiple diseases can be achieved.

[0090] This step first uses the data in the multi-disease medical corpus as the input of multiple source medical models. Among them, the data in the multi-disease medical corpus is mainly single-modal medical-related data. In order to build an integrated enhanced medical large model, this manual collects sample data from the multi-disease medical corpus and inputs them into multiple open source source medical models respectively, among which the open source source medical models include but are not limited to: such as LLaVA-Med, MedPaLM, BioGPT, etc.; the collected sample data is called single-modal medical sample data. After the single-modal medical sample data is input into multiple source medical models, the corresponding prediction distribution matrix of each source medical model is obtained.

[0091] Since these source medical models use different word segmenters, the dimensions of the prediction distribution matrices may not be consistent. In order to ensure the consistency and integrity of the information, this manual uses a vocabulary alignment method based on minimum matching cost to standardize each prediction distribution matrix, unify the size format, structure and semantics of each prediction distribution matrix, and provide a consistent input data structure for subsequent knowledge fusion. Specifically, the similarity between each prediction distribution matrix is ​​measured by comparing the cosine distance between each prediction distribution matrix as the matching cost. The word pairs with the lowest matching cost are selected for alignment one by one to ensure the consistency of the distribution matrix in structure and semantics. Among them, the smaller the minimum matching cost, the more similar the two prediction distribution matrices are; conversely, the larger the distance, the more dissimilar the prediction distribution matrices are.

[0092] Step 402 : Calculate the cross entropy between the predicted distribution matrix of each source medical model and the true distribution matrix corresponding to each source medical model, and determine the weight of each predicted distribution matrix.

[0093] Based on the standardized prediction distribution matrix, this step uses a fusion strategy based on average cross-entropy to calculate the cross-entropy value between each standardized prediction distribution matrix and its corresponding label, and then determines the weight value based on the cross-entropy value. The label describes the true distribution matrix corresponding to each source medical model for the input unimodal medical sample data, which can be obtained at the same time as the unimodal medical sample data is collected.

[0094] In step 403, the prediction distribution matrices output by each source medical model are weighted according to the weights to obtain a fused probability distribution matrix. After obtaining the weights for each prediction distribution matrix in step 302, the prediction distribution matrices of each source medical model are fused in a weighted manner to obtain a fused probability distribution matrix. This improves the accuracy and robustness of the model in disease diagnosis tasks.

[0095] Step 404: Select one of the source medical models from the multiple source medical models as the initial large medical model, and calculate the loss value between the predicted distribution matrix output by the initial large medical model, the fused probability distribution matrix, and the true distribution matrix.

[0096] Step 405: Iteratively train the parameters of the initial medical model according to the loss value to construct an initial integrated enhanced medical model.

[0097] A large model is selected from multiple source medical models as the initial large medical model. The predicted probability distribution output by the initial large medical model during each training session is calculated using a loss function, thereby training and adjusting the parameters of the initial large medical model so that it can integrate the knowledge of each source model to form a unified large model after knowledge fusion. A fusion loss function is constructed and a loss value is calculated based on the predicted distribution matrix output by the initial large medical model, the fused probability distribution matrix, and the true distribution matrix. The parameters of the initial large medical model are iteratively trained based on the loss value until the number of iterations of the initial large medical model reaches a preset value; or the loss value converges to a preset threshold, determining that the initial large medical model has met the training end condition, and thus constructing the initial integrated enhanced medical model.

[0098] In the embodiments of this specification, the fusion loss function is intended to improve the medical semantic understanding ability of the initial medical large model by optimizing the loss function, and enhance its performance in multi-disease medical scenarios. Specifically, the fusion loss function is determined based on the difference between the true distribution matrix and the fusion probability distribution matrix, and the difference between the fusion probability distribution matrix and the predicted distribution matrix. The fusion loss function is constructed using the following formula, and the loss value of the initial medical large model training is calculated.

[0099] The formula of the fusion loss function is as follows:

[0100]

[0101] in, Represents the difference measure between two matrices, implemented using relative entropy,

[0102] represents the expected value of text t in the multi-disease medical corpus C, O t Represents the true distribution matrix of the initial medical model for text t, Q t Represents the prediction distribution matrix of the initial medical model for the text t, P t represents the fused probability distribution matrix. The fusion loss function combines the difference between the true distribution matrix and the fused probability distribution matrix, as well as the difference between the fused probability distribution matrix and the predicted distribution matrix of the initial medical model, thereby achieving fitting of the true label distribution and learning of the fused distribution during the optimization process.

[0103] Step 406: fine-tune the initial integrated enhanced medical model to obtain an integrated enhanced medical model.

[0104] In this manual, the adaptive reduced-rank quantization fine-tuning training strategy is used to reduce the number of parameters that need to be adjusted for fine-tuning the initial integrated enhanced medical model by performing low-rank decomposition and quantization on the model parameters, thereby optimizing the integrated enhanced medical model. Figure 5 .

[0105] Figure 5 The flowchart of a method for fine-tuning an integrated enhanced medical model according to an embodiment of this specification is shown, which specifically includes the following steps:

[0106] Step 501: Decompose the pre-frozen weight matrix of the initial integrated enhanced medical model using singular value decomposition to obtain decomposed singular scale vectors and singular direction matrices.

[0107] In this step, first Figure 4 The weight matrix W0 of the integrated enhanced medical model obtained by pre-training is frozen. After freezing, the weight matrix W0 is decomposed into a singular scale vector m and a singular direction matrix using the singular value decomposition method, as shown in the following formula:

[0108] W0 = U·diag(m)·V, where m represents the singular scale vector, U and V represent the left and right singular value matrices, respectively, obtained by singular value decomposition. The right singular value matrix is ​​also called the singular direction matrix. diag(·) represents the diagonal matrix operator. This step decomposes the weight matrix of the initial integrated augmented medical model to achieve independent adjustment of scale and direction weights, capture the global properties of the weight matrix, reveal its low-rank structure and directional orthogonality, improve numerical stability, and achieve greater efficiency and reliability.

[0109] Step 502 : freeze the singular direction matrix, generate multiple basic low-rank matrices and multiple intermediate matrices, control the number of low-rank matrices and the values ​​of low-rank matrices, and decompose the singular direction matrix into at least one low-rank matrix.

[0110] Before freezing the singular direction matrix in this step, the method further includes quantizing the right singular value matrix (singular direction matrix). Specifically, considering the huge number of parameters of the singular direction matrix V, parameter quantization is used to quantize it, as shown in the following formula:

[0111] Where V represents the singular direction matrix, Represents the singular direction matrix after parameter quantization, s0 represents the scale factor, and its value is z0 represents the bias quantization value, which is 7 in 4-bit quantization. clamp(x,A,B) limits the value of x to between A and B. [·] is the rounding operation. This formula means The value is limited to 0 to 2 4 -1.

[0112] The singular direction matrix after quantizing the parameters After freezing, the low-rank matrix decomposition is further achieved by controlling the number of low-rank matrices n and the rank r of each low-rank matrix. Through this method, two basic low-rank matrices A and B are first generated, and n intermediate matrices X are introduced on this basis. i (i=1,...,n), the decomposed matrix is ​​much smaller than the dimension of the original weight matrix, thus reflecting the low-rank characteristic.

[0113] Step 503: Adjust the parameters of the low-rank matrix to obtain an adjusted singular value matrix.

[0114] By adjusting A, B, X i The parameters of the matrix without having to adjust the entire The parameters of the matrix reduce the computational cost. The adjusted singular direction matrix is

[0115]

[0116] Decomposed parameter matrices A, B, X i The parameter values ​​are initialized using a Gaussian distribution. This decomposition method allows for flexible control of the degree of freedom of decomposition by adjusting the number of intermediate matrices n, and significantly reduces the storage and computational costs of the weight matrix.

[0117] Step 504 , merging the singular scale vector, the singular direction matrix after low rank adjustment, and the singular direction matrix after parameter quantization to generate an updated weight matrix.

[0118] The singular scale vector m , the low-rank adjusted singular direction matrix And the parameterized singular direction matrix Merge to generate the final updated weight matrix W′. The updated weight matrix not only contains the main information of the original weight matrix, but also effectively introduces the parameter optimization captured by low-rank decomposition. The specific formula for parameter update is as follows:

[0119] W′ represents the updated weight matrix, represents the low-rank adjusted singular direction matrix, represents the low-rank adjusted singular direction matrix.

[0120] In this specification, an adaptive reduced-rank quantization fine-tuning training strategy is used to reduce the number of parameters that need to be adjusted for fine-tuning the integrated enhanced medical large model by performing low-rank decomposition and quantization on the model parameters, optimize the integrated enhanced medical large model, greatly reduce the consumption of computing resources and storage space, and improve the efficiency of fine-tuning training. In a resource-constrained environment, it can significantly shorten the fine-tuning time and computing cost while maintaining model performance. This greatly reduces the hardware requirements during the optimization process of the integrated enhanced medical large model, improves the application efficiency of the integrated enhanced medical large model in a dynamic medical environment, and makes large-scale deployment and clinical application more sustainable.

[0121] like Figure 6 The figure shows a schematic diagram of the structure of a multimodal medical large model application device according to an embodiment of this specification. The figure describes the basic structure of the multimodal medical large model application device. The functional units and modules therein can be implemented in software, or a general-purpose chip or a specific chip can be used to implement the multimodal medical large model application. The device specifically includes:

[0122] An acquisition unit 601 is used to acquire multimodal feature data;

[0123] The prediction unit 602 is used to input the multimodal feature data into a pre-trained multimodal medical big model to obtain an output result, wherein the multimodal medical big model includes a multimodal semantic alignment model and an integrated enhanced medical big model. The multimodal semantic alignment model is obtained by training an initial multimodal semantic alignment model with multimodal feature sample vectors, representation extraction sample vectors and semantic labels corresponding to the multimodal feature sample vectors. The integrated enhanced medical big model is trained by a predicted distribution matrix obtained by predicting single-modal medical sample data using multiple source medical models and a true distribution matrix corresponding to each source medical model.

[0124] like Figure 7 The figure shows a schematic diagram of the structure of a multimodal medical model fine-tuning device according to an embodiment of this specification. The figure describes the basic structure of the multimodal medical model fine-tuning device. The functional units and modules therein can be implemented in software, or a general-purpose chip or a specific chip can be used to implement the multimodal medical model construction. The device specifically includes:

[0125] A first prediction result acquisition unit 701 is configured to input a multimodal feature sample vector, a representation extraction sample vector, and a semantic label corresponding to the multimodal feature sample vector into an initial multimodal semantic alignment model to obtain a prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features from the multimodal feature sample vector;

[0126] A multimodal semantic alignment model construction unit 702 is configured to calculate a multimodal matching loss function based on the prediction result and the semantic label, and iteratively update the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function to construct a multimodal semantic alignment model.

[0127] The second prediction result acquisition unit 703 is used to input the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical model to obtain the prediction result output by the integrated enhanced medical model, wherein the integrated enhanced medical model is trained by the prediction distribution matrix obtained by predicting single-modal medical sample data by multiple source medical models and the true distribution matrix corresponding to each source medical model.

[0128] Figure 8 A schematic diagram illustrating an embodiment of this specification illustrating the construction of an initial integrated enhanced medical macromodel based on a medical source model. The diagram utilizes three pre-trained models—Medical Macromodel A, Medical Macromodel B, and Medical Macromodel C—based on a wealth of medical knowledge accumulated across various medical fields, such as disease diagnosis and treatment, medical textbooks, and medical databases. Multimodal fusion feature data is used to perform continuous training and fine-tuning optimization, enabling diagnosis of multiple diseases and building a more accurate medical diagnostic model.

[0129] Figure 9 Shown is a schematic diagram of an updated weight matrix of an integrated enhanced medical large model according to an embodiment of this specification. The weight matrix of the initial integrated enhanced medical large model is decomposed into a singular scale vector and a singular direction matrix by using singular value decomposition. Secondly, two basic low-rank matrices A and B are generated, and n intermediate matrices are introduced on this basis. By controlling the number of introduced intermediate matrices and the rank of each low-rank matrix, low-rank matrix decomposition is achieved. The degree of freedom of decomposition is flexibly controlled to reduce the storage and computational costs of the weight matrix. Finally, the singular scale vector, the singular direction matrix and the singular direction matrix updated with low-rank parameters are merged to generate the final updated weight matrix. The updated weight matrix not only contains the main information of the original weight matrix, but also effectively introduces the parameter optimization captured by low-rank decomposition, thereby achieving fine-tuning of the initial integrated enhanced medical large model.

[0130] Figure 10 Figure 2 shows a schematic diagram of a multimodal semantic alignment model based on adaptive fusion, according to an embodiment of this specification. This model integrates a pre-fusion-driven multimodal semantic alignment mechanism and a semantic alignment method based on adaptive representation computation, enabling semantic alignment of multimodal feature vectors with the input space of an integrated augmented medical model.

[0131] Figure 11Schematic diagram of a computer device provided in accordance with an embodiment of this specification. The multimodal medical large model application and fine-tuning method described herein can be applied to the computer device. The computer device 1102 may include one or more processors 1104, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 1102 may also include any memory 1106 for storing any type of information, such as code, settings, data, and the like. For example, and without limitation, the memory 1106 may include any one or more combinations of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, and the like. More generally, any memory may use any technology to store information. Furthermore, any memory may provide volatile or non-volatile retention of information. Furthermore, any memory may represent a fixed or removable component of the computer device 1102. In one embodiment, when the processor 1104 executes associated instructions stored in any memory or combination of memories, the computer device 1102 may perform any operation of the associated instructions. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like.

[0132] The computer device 1102 may also include an input / output module 1110 (I / O) for receiving various inputs (via input devices 1112) and for providing various outputs (via output devices 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface (GUI) 1118. In other embodiments, the input / output module 1110 (I / O), input devices 1112, and output devices 1114 may not be included, and the computer device 1102 may simply be a computer device in a network. The computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.

[0133] The communication link 1122 may be implemented in any manner, for example, via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0134] Corresponding to Figures 1 to 5 Method, an embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0135] The embodiment of this specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 5 The method shown.

[0136] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0137] It should also be understood that in the embodiments of this specification, the term "and / or" is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " in this specification generally indicates that the associated objects are in an "or" relationship.

[0138] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0139] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be an electrical, mechanical or other form of connection.

[0141] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0142] In addition, the functional units in the various embodiments of this specification may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0144] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, based on the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this specification.

Claims

1. A multimodal medical large model fine-tuning method, characterized in that: in, The multimodal medical big model includes a multimodal semantic alignment model and an integrated enhanced medical big model, and the method includes: The multimodal feature sample vector and the representation extraction sample vector are input into the initial multimodal semantic alignment model to obtain the prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features from the multimodal feature sample vector, which includes: determining the representation extraction sample vector according to the following formula: ; Where v represents the sample extraction vector, represents the multimodal feature sample vector, represents the feature center vector, u represents the main feature direction vector; A multimodal matching loss function is calculated based on the semantic labels corresponding to the prediction results and the multimodal feature sample vectors, and the parameters in the initial multimodal semantic alignment model are iteratively updated based on the multimodal matching loss function to construct a multimodal semantic alignment model. The multimodal matching loss function is constructed in the following manner: Determine the contrast loss function by maximizing the similarity of multimodal feature positive sample vector pairs and minimizing the similarity of multimodal feature negative sample vector pairs; Determine the cosine distance loss function based on the multimodal feature sample vector and the corresponding semantic label; Based on the contrast loss function and the cosine distance loss function, a multimodal matching loss function is constructed, and the formula is as follows: in, represents the multimodal matching loss function, represents the contrast loss function for information aggregation, represents the cosine distance loss function, represents the regularization coefficient; The output of the multimodal semantic alignment model is input into a pre-trained integrated enhanced medical model to obtain a prediction result output by the integrated enhanced medical model, wherein the integrated enhanced medical model is trained by a prediction distribution matrix obtained by predicting single-modal medical sample data using multiple source medical models and a true distribution matrix corresponding to each source medical model, wherein the integrated enhanced medical model is constructed by: Inputting single-modality medical sample data into multiple source medical models, and obtaining the prediction distribution matrix output by each source medical model; Calculate the cross entropy between the predicted distribution matrix of each source medical model and the true distribution matrix corresponding to each source medical model, and determine the weight of each predicted distribution matrix; According to the weights, the prediction distribution matrices output by each source medical model are weighted to obtain a fusion probability distribution matrix; Selecting one of the source medical models from the multiple source medical models as an initial large medical model, and calculating a loss value between a predicted distribution matrix output by the initial large medical model, the fused probability distribution matrix, and a true distribution matrix; Iteratively training the parameters of the initial medical model according to the loss value to construct an initial integrated enhanced medical model; The initial integrated enhanced medical model is fine-tuned to obtain an integrated enhanced medical model.

2. The method according to claim 1, characterized in that The method further comprises: Setting a fully connected linear projection layer between the multimodal semantic alignment model and the integrated enhanced medical model to match the feature dimensions of the output of the multimodal semantic alignment model with the input of the integrated enhanced medical model; According to the loss function of the masked language model, the multimodal semantic alignment model is optimized to determine the output of the multimodal semantic alignment model and the prediction results of the integrated enhanced medical model output.

3. The method according to claim 2, characterized in that The loss value between the predicted distribution matrix output by the initial medical large model, the fused probability distribution matrix and the true distribution matrix is ​​calculated as follows: A fusion loss function is determined based on the difference between the true distribution matrix and the fused probability distribution matrix, and the difference between the fused probability distribution matrix and the predicted distribution matrix.

4. The method according to claim 3, characterized in that Fine-tuning the initial integrated enhanced medical model to obtain the integrated enhanced medical model includes: Decomposing the pre-frozen weight matrix of the initial integrated enhanced medical large model using singular value decomposition to obtain decomposed singular scale vectors and singular direction matrices; Freezing the singular direction matrix, generating multiple basic low-rank matrices and multiple intermediate matrices, controlling the number of low-rank matrices and the values ​​of the low-rank matrices, and decomposing the singular direction matrix into at least one low-rank matrix; Adjusting the parameters of the low-rank matrix to obtain a low-rank adjusted singular value matrix; The singular scale vector, the adjusted singular direction matrix, and the parameter quantized singular direction matrix are combined to generate an updated weight matrix.

5. A multimodal medical large model application method, characterized in that: The method comprises: Acquire multimodal feature data; The multimodal feature data is input into the multimodal medical large model obtained by the method according to any one of claims 1 to 4 to obtain an output result.

6. A multimodal medical large model application device, characterized in that: The device comprises: an acquisition unit, used for acquiring multimodal feature data; A prediction unit is used to input the multimodal feature data into the multimodal medical large model obtained by the method according to any one of claims 1 to 4 to obtain an output result.

7. A multimodal medical model fine-tuning device, characterized in that: in, The multimodal medical big model includes a multimodal semantic alignment model and an integrated enhanced medical big model, and the device includes: The first prediction result acquisition unit is used to input the multimodal feature sample vector, the representation extraction sample vector, and the semantic label corresponding to the multimodal feature sample vector into the initial multimodal semantic alignment model to obtain the prediction result output by the initial multimodal semantic alignment model, wherein the representation extraction sample vector is used to extract features from the multimodal feature sample vector, which includes: determining the representation extraction sample vector according to the following formula: ; Where v represents the sample extraction vector, represents the multimodal feature sample vector, represents the feature center vector, u represents the main feature direction vector; A multimodal semantic alignment model construction unit is configured to calculate a multimodal matching loss function based on the prediction result and the semantic label, and iteratively update the parameters in the initial multimodal semantic alignment model based on the multimodal matching loss function to construct a multimodal semantic alignment model; wherein the multimodal matching loss function is constructed in the following manner: Determine the contrast loss function by maximizing the similarity of multimodal feature positive sample vector pairs and minimizing the similarity of multimodal feature negative sample vector pairs; Determine the cosine distance loss function based on the multimodal feature sample vector and the corresponding semantic label; Based on the contrast loss function and the cosine distance loss function, a multimodal matching loss function is constructed, and the formula is as follows: in, represents the multimodal matching loss function, represents the contrast loss function for information aggregation, represents the cosine distance loss function, represents the regularization coefficient; The second prediction result acquisition unit is configured to input the output of the multimodal semantic alignment model into a pre-trained integrated enhanced medical model to obtain a prediction result output by the integrated enhanced medical model, wherein the integrated enhanced medical model is trained by a prediction distribution matrix obtained by predicting single-modal medical sample data using multiple source medical models and a true distribution matrix corresponding to each source medical model, wherein the integrated enhanced medical model is constructed in the following manner: Inputting single-modality medical sample data into multiple source medical models, and obtaining the prediction distribution matrix output by each source medical model; Calculate the cross entropy between the predicted distribution matrix of each source medical model and the true distribution matrix corresponding to each source medical model, and determine the weight of each predicted distribution matrix; According to the weights, the prediction distribution matrices output by each source medical model are weighted to obtain a fusion probability distribution matrix; Selecting one of the source medical models from the multiple source medical models as an initial large medical model, and calculating a loss value between a predicted distribution matrix output by the initial large medical model, the fused probability distribution matrix, and a true distribution matrix; Iteratively training the parameters of the initial medical model according to the loss value to construct an initial integrated enhanced medical model; The initial integrated enhanced medical model is fine-tuned to obtain an integrated enhanced medical model.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images