Medical multi-mode large model intelligent system and method based on federated learning

Abstract

The invention discloses a medical multi-modal large model intelligent system and method based on federated learning, relates to the technical field of medical information, and solves the technical problem that existing federated learning still has defects in the aspects of processing heterogeneous data quality, ensuring model interpretability and promoting multi-party effective cooperation. The system comprises a federated architecture module, a federated reasoning module and a model collaboration module, the federal architecture module is used for adjusting modal weight in real time according to data quality; the federated reasoning module is used for constructing a knowledge graph; introducing the knowledge graph into a medical multi-modal large model; setting an atlas constraint loss function; the model collaboration module is used for optimizing the medical multi-modal large model through distributed training and localized deployment; according to the method, multi-modal data can be effectively integrated, and the model training and reasoning process is optimized by utilizing medical knowledge, so that the accuracy and reliability of medical diagnosis are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical information, and relates to a multi-modal large language model technology, in particular to a medical multi-modal large model intelligent system and method based on federated learning. BACKGROUND

[0002] With the development of medical informatization, medical institutions have accumulated a large amount of multi-modal medical data, which is crucial for training high-precision and multi-functional medical artificial intelligence models; medical data covers multiple types such as images, genetic sequencing, and physiological signals; medical multi-modal large models can simultaneously analyze cross-modal information and mine potential correlations between data; meanwhile, multi-modal fusion can make up for the shortcomings of single data and improve the sensitivity and specificity of diagnosis; through deep analysis of patient historical data, the model can predict disease development trends and patient prognosis, providing a basis for doctors to develop treatment plans.

[0003] Data privacy protection regulations in the prior art limit direct data sharing, resulting in a serious "data island" phenomenon; federated learning, as a distributed machine learning framework, allows participants to jointly train models without sharing raw data, which provides a possibility for solving this problem; however, existing federated learning still has deficiencies in handling heterogeneous data quality, ensuring model interpretability, and promoting effective collaboration among multiple parties. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art; for this purpose, the present application proposes a medical multi-modal large model intelligent system and method based on federated learning, which is used to solve the technical problem that existing federated learning still has deficiencies in handling heterogeneous data quality, ensuring model interpretability, and promoting effective collaboration among multiple parties.

[0005] To achieve the above-mentioned purpose, the first aspect of the present application provides a medical multi-modal large model intelligent system based on federated learning, comprising a federated architecture module, a federated inference module, and a model collaboration module; The federated architecture module is used to adjust the modality weight in real time according to the data quality; The federated inference module is used to construct a knowledge graph, introduce the knowledge graph into the medical multi-modal large model, and set a graph constraint loss function; The model collaboration module is used to optimize the medical multi-modal large model through distributed training and localized deployment.

[0006] Based on the above modules, the modal weight is adjusted in real time according to the data quality, so that the model can dynamically adapt to different sources and different quality data; the weight of each modal data in the model can be automatically identified and adjusted, ensuring that the model is always trained and inferred based on reliable and valuable data, and improving the processing capability of the model for complex medical data; the real-time weight adjustment mechanism helps the model to maintain stable performance when facing data fluctuations or abnormalities. When some modal data has quality problems or is missing, the model can reduce its weight and reduce the negative impact on overall performance, thereby enhancing the robustness of the model in complex and variable environments and providing more reliable basis for medical diagnosis and decision-making; a knowledge graph is constructed and introduced into the medical multi-modal large model, providing the model with a structured medical knowledge system; various concepts, relationships and facts in the medical field can be integrated to improve the understanding and explanation capability of the model for medical problems; a graph constraint loss function is set to guide the model to follow the logic and rules contained in the knowledge graph during the training process; it helps the model to generate reasoning results that conform to medical common sense and logic, reduces errors and unreasonable predictions, and improves the accuracy of medical diagnosis and treatment, providing more valuable reference for clinical practice; distributed training is adopted to distribute the training task to multiple computing nodes for parallel processing, fully utilizing the computing resources and greatly shortening the model training time. At the same time, local deployment enables the model to run independently on different medical institutions or devices, further optimizing and adjusting the model according to local data, and improving the pertinence and practicality of the model.

[0007] Preferably, the real-time adjustment of the modal weight according to the data quality comprises: evaluating the data quality by using the modal health quantification unit to obtain a modal health index; analyzing the weight coefficients of each modal data by using the gating routing network according to the modal health index; and screening high-value modal features according to the weight coefficients of each modal data.

[0008] Preferably, the evaluation of the data quality by using the modal health quantification unit to obtain a modal health index comprises: obtaining the modal quality of the local data; wherein the modal quality comprises image clarity, text integrity and diagnosis report accuracy; calculating the modal health index by the formula ; wherein, , and are weight coefficients; and image clarity, text integrity and diagnosis report accuracy represent the contribution degree of image clarity, text integrity and diagnosis report accuracy, respectively; is the image clarity; is the text integrity; is the diagnosis report accuracy.

[0009] Preferably, the construction of the knowledge graph includes: Acquire medical knowledge data; this includes medical books, clinical guidelines, and expert experience. Medical knowledge data is categorized and integrated according to medical classifications to obtain professional information; a knowledge graph is then constructed based on this professional information.

[0010] Preferably, the step of introducing knowledge graphs into the medical multimodal large model includes: Add knowledge graphs to the output of large language models; The knowledge graph is encoded into a vector through the embedding layer of the large language model and concatenated to the input vector; then the input vector is embedded into the federated learning framework.

[0011] Preferably, the setting of the graph constraint loss function includes: Through formula Calculate the graph constraint loss function; where, It is the loss of a conventional large model; It is a graph constraint loss; This is the balance coefficient.

[0012] Preferably, the federated inference module further includes: Represent the initial features generated by the local model; The reasoning path of the global model is verified based on the knowledge graph; The contribution report will be fed back to all participating parties.

[0013] Preferably, the optimization of the medical multimodal large model through distributed training and local deployment includes: It retrieves and integrates data from multiple imaging modalities to perform pre-training and supervised fine-tuning of examination items for multiple parts of the body; and sets up communication protocols and asynchronous update mechanisms for large-scale medical multimodal models.

[0014] The second aspect of this application provides an intelligent method for large multimodal medical models based on federated learning, including: Modal weights are adjusted in real time based on data quality. Constructing knowledge graphs; introducing knowledge graphs into large-scale multimodal medical models; Set the graph constraint loss function; Optimize large-scale medical multimodal models through distributed training and localized deployment.

[0015] A third aspect of this application provides a computer-readable storage medium storing instructions for performing the steps of the method described in one possible implementation of the second aspect.

[0016] Compared with the prior art, the beneficial effects of this application are: 1. This application measures the modal quality of local data from three key dimensions: image clarity, text completeness, and diagnostic report accuracy. It comprehensively and objectively assesses the value of different modalities. The Modal Health Index (MHI) is calculated to quantify and synthesize the quality indicators across various dimensions. This quantification method makes the quality of different modalities comparable, providing clear numerical basis for subsequent data processing and model training, and avoiding biases caused by subjective judgment. The gated routing network can dynamically analyze the weight coefficients of each modality based on the Modal Health Index. The dynamic adjustment mechanism allows the model to respond in real time to changes in data quality; when the quality of a modality changes, its weight coefficient is adjusted accordingly, ensuring that the model is always trained and inferred based on the optimal data combination. Selecting high-value modal features reduces data redundancy during model training, preventing the model from wasting computational resources and time on low-quality or irrelevant features. Using only high-value features for training accelerates model convergence, improves training efficiency, and shortens the model development cycle.

[0017] 2. This application acquires diverse medical knowledge data, including medical books, clinical guidelines, and expert experience, and categorizes and integrates them to obtain professional information, thereby constructing a knowledge graph. It comprehensively gathers professional knowledge from different sources and levels in the medical field, covering basic theoretical knowledge, clinical practice experience, and authoritative expert insights, providing the model with a rich, accurate, and structured knowledge foundation. The knowledge graph is added to the output of the large language model, and its encoding into vectors is performed through the embedding layer of the large language model. These vectors are then concatenated to the input vector, which is then embedded into a federated learning framework. This allows the model to utilize both the original input data and the structured knowledge in the knowledge graph during training and inference, enriching the model's input information and providing more comprehensive and in-depth background knowledge support. A graph constraint loss function is calculated to guide the model to follow the medical logic and rules inherent in the knowledge graph during training. This helps the model generate inference results that are more consistent with medical common sense and clinical practice, avoiding erroneous predictions that contradict medical knowledge and improving the model's accuracy and reliability; it represents the initial features generated by the local model and allows the global model to verify the inference path based on the knowledge graph; the collaborative verification mechanism can ensure that the inference results generated by the local model conform to the global knowledge framework and medical logic, guaranteeing the consistency and quality of the model among different participants; it provides contribution reports to each participant, enabling them to understand their contribution to the federated learning process; it helps incentivize participants to actively participate in model training and optimization, and also allows them to identify their own problems and shortcomings based on the contribution reports, enabling targeted improvements and optimizations. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the module connection structure of this application; Figure 2 This is a schematic diagram illustrating the working principle of the federal architecture module in this application; Figure 3 This is a schematic diagram illustrating the working principle of the federated reasoning module in this application; Figure 4 This is a schematic diagram of the overall application process. Detailed Implementation

[0020] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] Please see Figure 1 The first aspect of this application provides a medical multimodal large model intelligent system based on federated learning, including: a federated architecture module, a federated inference module, and a model collaboration module; The federated architecture module is used to adjust modal weights in real time based on data quality; The federated reasoning module is used to construct a knowledge graph; introduce the knowledge graph into a large multimodal medical model; and set a graph constraint loss function. The model collaboration module is used to optimize large medical multimodal models through distributed training and localized deployment.

[0022] In one possible implementation, the medical multimodal large model is optimized through distributed training and localized deployment, including: It retrieves and integrates data from multiple imaging modalities to perform pre-training and supervised fine-tuning of examination items for multiple parts of the body; and sets up communication protocols and asynchronous update mechanisms for large-scale medical multimodal models.

[0023] Based on the above modules, the modal weights are adjusted in real time according to data quality, enabling the model to dynamically adapt to data from different sources and of different quality; it can automatically identify and adjust the weights of each modality in the model, ensuring that the model is always trained and inferred based on reliable and valuable data, thereby improving the model's ability to process complex medical data; the real-time weight adjustment mechanism helps the model maintain stable performance when faced with data fluctuations or anomalies. When certain modal data exhibits quality issues or is missing, the model can reduce its weights to minimize the negative impact on overall performance, thereby enhancing the model's robustness in complex and ever-changing environments and providing a more reliable basis for medical diagnosis and decision-making. Constructing a knowledge graph and incorporating it into a large-scale medical multimodal model provides the model with a structured medical knowledge system. This integrates various concepts, relationships, and facts in the medical field, improving the model's understanding and interpretation of medical problems. Setting a graph-constrained loss function guides the model to follow the logic and rules inherent in the knowledge graph during training. This helps the model generate reasoning results that are more consistent with medical common sense and logic, reducing errors and unreasonable predictions, improving the accuracy of medical diagnosis and treatment, and providing more valuable references for clinical practice. A distributed training approach distributes training tasks across multiple computing nodes for parallel processing, fully utilizing computing resources and significantly shortening model training time. Simultaneously, localized deployment allows the model to run independently in different medical institutions or on different devices, enabling further optimization and adjustments based on local data, improving the model's relevance and practicality.

[0024] Please see Figure 2 The federated architecture module includes: Acquire the modal quality of local data; where modal quality includes: image sharpness, text integrity, and diagnostic report accuracy; Through formula The modal health index was calculated; among which, , and These are weighting coefficients; representing the contributions of image clarity, text integrity, and diagnostic report accuracy, respectively. For image clarity; For text completeness; To ensure the accuracy of the diagnostic report.

[0025] Based on the modal health index, the weight coefficients of each modal data are analyzed using a gated routing network; high-value modal features are then selected based on the weight coefficients of each modal data.

[0026] Example: Suppose there are three hospitals (Hospital A, Hospital B, and Hospital C) participating in federated learning. Each hospital has local medical data, including imaging data (such as X-rays, CT scans, etc.), text data (such as medical records), and diagnostic reports. Hospital A used a professional image quality assessment algorithm to evaluate local CT images and found that most images had high resolution, but a small number of images were blurry. After quantitative calculation, the average image clarity score was 8 points. Hospital B assessed the completeness of medical records by checking whether they included key elements such as basic patient information, symptom descriptions, past medical history, and examination and test results. Some medical records were found to have missing information, with an average text completeness score of 7. Hospital C organized experts to conduct a sample review of local diagnostic reports, assessing accuracy based on the degree of consistency between the diagnostic results and the actual patient conditions. The average accuracy score for the diagnostic reports was 8.5 points. Assuming weighting coefficients α=0.5, β=0.3, and γ=0.2, representing the contributions of image clarity, text integrity, and diagnostic report accuracy, respectively; Based on the formula, the modal health indices for hospitals A, B, and C were calculated to be 7.8, 7.6, and 7.9, respectively. Each hospital inputs the calculated Modal Health Index (MHI) into the gated routing network. The gated routing network dynamically analyzes the weighting coefficients of the three modalities—image, text, and diagnostic report—based on the MHI values. For example, the gated routing network analysis for Hospital A yields an image modality weight of 0.6, a text modality weight of 0.25, and a diagnostic report modality weight of 0.15. Hospital A screened out high-value modal features. For the image modality, it selected image region features with high resolution and good contrast; for the text modality, it extracted text fragment features containing key symptoms and medical history information; for the diagnostic report modality, it selected features related to diagnostic conclusions and key evidence.

[0027] Similarly, Hospital B and Hospital C perform the same operation.

[0028] Based on the above steps, the federated architecture module measures the modal quality of local data from three key dimensions: image clarity, text integrity, and diagnostic report accuracy. It comprehensively and objectively assesses the value of different modalities and calculates the Modal Health Index (MHI), quantifying and synthesizing the quality indicators across various dimensions. This quantification method makes the quality of different modalities comparable, providing clear numerical basis for subsequent data processing and model training, and avoiding biases caused by subjective judgment. The gated routing network can dynamically analyze the weight coefficients of each modality based on the Modal Health Index. The dynamic adjustment mechanism allows the model to respond in real time to changes in data quality; when the quality of a modality changes, its weight coefficient is adjusted accordingly, ensuring that the model is always trained and inferred based on the optimal data combination. Selecting high-value modal features reduces data redundancy during model training, preventing the model from wasting computational resources and time on low-quality or irrelevant features. Using only high-value features for training accelerates model convergence, improves training efficiency, and shortens the model development cycle.

[0029] Please see Figure 3 The federated reasoning module includes: Acquire medical knowledge data, including medical books, clinical guidelines, and expert experience; classify and integrate the medical knowledge data according to medical categories to obtain professional information; and construct a knowledge graph based on the professional information.

[0030] The knowledge graph is added to the output of the large language model; the knowledge graph is encoded into a vector through the embedding layer of the large language model and concatenated to the input vector; and the input vector is embedded into the federated learning framework.

[0031] Through formula Calculate the graph constraint loss function; where, It is the loss of a conventional large model; It is a graph constraint loss; This is the balance coefficient.

[0032] The system generates initial features for the local model; verifies the reasoning path of the global model based on the knowledge graph; and provides contribution reports to all participants.

[0033] Example: The three hospitals jointly collected medical books (such as "Internal Medicine" and "Surgery"), clinical guidelines (such as national guidelines for the diagnosis and treatment of various diseases), and summaries of clinical experience from their own experts; The collected medical knowledge is categorized and integrated according to disease type, symptoms, treatment methods, drugs, etc. For example, "lung cancer" is used as the central node, connecting related symptoms (such as cough, hemoptysis), diagnostic methods (such as CT scan, pathological biopsy), treatment methods (such as surgery, chemotherapy, radiotherapy), etc., to construct a complete medical knowledge graph; Each hospital adds its constructed knowledge graph to the output of the large language model. The embedding layer of the large language model encodes the nodes and edges in the knowledge graph into vectors, which are then concatenated into the input vector. For example, when processing a case about lung cancer diagnosis, the knowledge graph vector related to lung cancer is concatenated to the text input vector of the case. The concatenated input vector is then embedded into a federated learning framework, enabling the local models of each hospital to collaboratively utilize the knowledge graph for training and inference while protecting data privacy. Assume that the conventional large model loss is obtained based on the accuracy of each token word output by the supervised model, while the graph constraint loss is obtained based on the accuracy of the medical facts output by the supervised model. Set the balance coefficient λ = 0.3; During training, each hospital calculates the total loss function according to the formula. For example, in a certain round of training, Hospital A's loss for the standard large model is 0.2; the loss for the graph constraint is 0.1; then the total loss function is 0.23.

[0034] Each hospital's local model generates initial feature representations based on the input data for subsequent inference and validation. The global model in the federated learning framework uses a knowledge graph to validate the inference paths of each local model. For example, when Hospital A's local model diagnoses a case as lung cancer, the global model checks whether the diagnostic inference process conforms to the lung cancer diagnostic logic in the knowledge graph, such as whether key symptoms and examination results were considered. Based on the validation results, the contribution of each participant (hospital) to the federated learning process is calculated, and contribution reports are fed back to each hospital. For example, if Hospital B's data and model inference results contribute significantly to improving the overall model's performance, its contribution report will have a higher score. This helps incentivize hospitals to actively participate in federated learning and jointly improve the model's quality and performance.

[0035] Based on the above steps, the federated inference module acquires diverse medical knowledge data, including medical books, clinical guidelines, and expert experience, and categorizes and integrates this data to obtain professional information, thereby constructing a knowledge graph. This comprehensively gathers professional knowledge from different sources and levels within the medical field, covering basic theoretical knowledge, clinical practice experience, and insights from authoritative experts, providing the model with a rich, accurate, and structured knowledge foundation. The knowledge graph is added to the output of the large language model, and its embedding layer encodes it into a vector, which is then concatenated to the input vector. The input vector is then embedded into the federated learning framework. This allows the model to utilize both the original input data and the structured knowledge in the knowledge graph during training and inference, enriching the model's input information and providing more comprehensive and in-depth background knowledge support. Finally, a graph constraint loss function is calculated to guide the model to follow the medical logic and rules inherent in the knowledge graph during training. This helps the model generate inference results that are more consistent with medical common sense and clinical practice, avoiding erroneous predictions that contradict medical knowledge and improving the model's accuracy and reliability; it represents the initial features generated by the local model and allows the global model to verify the inference path based on the knowledge graph; the collaborative verification mechanism can ensure that the inference results generated by the local model conform to the global knowledge framework and medical logic, guaranteeing the consistency and quality of the model among different participants; it provides contribution reports to each participant, enabling them to understand their contribution to the federated learning process; it helps incentivize participants to actively participate in model training and optimization, and also allows them to identify their own problems and shortcomings based on the contribution reports, enabling targeted improvements and optimizations.

[0036] Please see Figure 4 The second aspect of this application provides an intelligent method for medical multimodal large models based on federated learning, including: Modal weights are adjusted in real time based on data quality. Constructing knowledge graphs; introducing knowledge graphs into large-scale multimodal medical models; Set the graph constraint loss function; Optimize large-scale medical multimodal models through distributed training and localized deployment.

[0037] A third aspect of this application provides a computer-readable storage medium storing instructions for performing the steps of the method described in a possible implementation of a second aspect embodiment.

[0038] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0039] The working principle of this application is as follows: This application includes a federated architecture module, a federated inference module, and a model collaboration module; the federated architecture module is used to adjust modal weights in real time according to data quality; the federated inference module is used to construct a knowledge graph; introduce the knowledge graph into a large medical multimodal model; and set a graph constraint loss function; the model collaboration module is used to optimize the large medical multimodal model through distributed training and localized deployment.

[0040] The above embodiments are only used to illustrate the technical methods of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of this application without departing from the spirit and scope of the technical methods of this application.

Claims

1. A large-scale intelligent medical multimodal model system based on federated learning, characterized in that: include: The module includes a federated architecture module, a federated inference module, and a model collaboration module. The federated architecture module is used to adjust modal weights in real time based on data quality; The federated reasoning module is used to construct a knowledge graph; introduce the knowledge graph into a large multimodal medical model; and set a graph constraint loss function. The model collaboration module is used to optimize large medical multimodal models through distributed training and localized deployment.

2. The intelligent system for medical multimodal large models based on federated learning according to claim 1, characterized in that, The real-time adjustment of modal weights based on data quality includes: The modal health index is obtained by using a modal health quantification unit to assess data quality. Based on the modal health index, the weight coefficients of each modal data are analyzed using a gated routing network; high-value modal features are then selected based on the weight coefficients of each modal data.

3. The intelligent system for large-scale medical multimodal models based on federated learning according to claim 1, characterized in that, The method of using a modal health quantification unit to evaluate data quality and obtain a modal health index includes: Acquire the modal quality of local data; where modal quality includes: image sharpness, text integrity, and diagnostic report accuracy; Through formula The modal health index was calculated; among which, , and These are weighting coefficients; representing the contributions of image clarity, text integrity, and diagnostic report accuracy, respectively. For image clarity; For text completeness; To ensure the accuracy of the diagnostic report.

4. The intelligent system for large-scale medical multimodal models based on federated learning according to claim 1, characterized in that, The construction of the knowledge graph includes: Acquire medical knowledge data; this includes medical books, clinical guidelines, and expert experience. Medical knowledge data is categorized and integrated according to medical classifications to obtain professional information; a knowledge graph is then constructed based on this professional information.

5. The intelligent system for large-scale medical multimodal models based on federated learning according to claim 1, characterized in that, The introduction of knowledge graphs into a large-scale multimodal medical model includes: Add knowledge graphs to the output of large language models; The knowledge graph is encoded into a vector through the embedding layer of the large language model and concatenated to the input vector; then the input vector is embedded into the federated learning framework.

6. The intelligent system for medical multimodal large models based on federated learning according to claim 1, characterized in that, The setting of the graph constraint loss function include: Through formula Calculate the graph constraint loss function; where, It is the loss of a conventional large model; It is a graph constraint loss; This is the balance coefficient.

7. The intelligent system for medical multimodal large models based on federated learning according to claim 1, characterized in that, The federated inference module also includes: Represent the initial features generated by the local model; The reasoning path of the global model is verified based on the knowledge graph; The contribution report will be fed back to all participating parties.

8. The intelligent system for large-scale medical multimodal models based on federated learning according to claim 1, characterized in that, The optimization of the large-scale medical multimodal model through distributed training and local deployment includes: It retrieves and integrates data from multiple imaging modalities to perform pre-training and supervised fine-tuning of examination items for multiple parts of the body; and sets up communication protocols and asynchronous update mechanisms for large-scale medical multimodal models.

9. A federated learning-based intelligent method for large-scale medical multimodal models, applied to the federated learning-based intelligent system for large-scale medical multimodal models as described in any one of claims 1-8, characterized in that, include: Modal weights are adjusted in real time based on data quality. Constructing knowledge graphs; Introducing knowledge graphs into large-scale multimodal medical models; Set the graph constraint loss function; Optimize large-scale medical multimodal models through distributed training and localized deployment.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions for performing the steps of the method described in claim 9.

Images