Method and system for cooperative medical image diagnosis based on MID-LLM framework
By introducing blockchain and LLM technologies into the MID-LLM framework, decentralized collaboration and secure medical image diagnosis are achieved, and the problems of data privacy leakage and single point of failure in the existing technology are solved, improving the accuracy and security of diagnosis.
Patent Information
- Application Number
- CN202510181592.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-19
- Publication Date
- 2025-06-06
AI Technical Summary
The existing collaborative medical image diagnosis method based on the MID-LLM framework has problems such as data privacy leakage risks, single point of failure and confrontational attacks, and has not fully utilized the advantages of LLM and blockchain technology.
Blockchain and LLM are used for federated learning, decentralized collaboration is achieved through interstellar file systems and shared smart contracts, integrated image processing modules for local training, and improved aggregation algorithms and blockchain consensus mechanisms are used to ensure the security and accuracy of the model.
Improve the accuracy and reliability of target analysis, enhance data security and work efficiency, avoid single point of failure and data breach risks, and prevent malicious attacks through verification mechanisms.
Smart Images
Figure CN120108699A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing and artificial intelligence, and in particular to a method and system for collaborative medical image diagnosis based on a MID-LLM framework. Background Art
[0002] In recent years, with the rapid growth of medical imaging data, traditional centralized systems face severe challenges in diagnostic accuracy, data privacy, and computing efficiency.
[0003] In order to solve these problems, researchers have explored a variety of technical solutions: Deep learning technology has made significant progress in the field of medical image analysis, such as brain tumor segmentation and lesion detection. However, centralized systems need to collect a large amount of data, which poses a risk of data privacy leakage; traditional federated learning frameworks allow multiple devices to train the framework locally and collaborate through framework parameters to protect data privacy and reduce communication costs. However, traditional federated learning frameworks are vulnerable to single point failures and adversarial attacks, such as model poisoning and bias attacks, which affect model performance and data privacy; LLM performs well in natural language processing and can provide a deeper understanding of context, thereby improving the accuracy of medical image analysis. Blockchain technology has the characteristics of decentralization, anonymity, immutability and traceability, which can enhance the security and reliability of federated learning systems, but existing federated learning frameworks rarely consider the application of LLM and blockchain technology. Summary of the invention
[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.
[0005] In view of the problems existing in the existing methods and systems for collaborative medical image diagnosis based on the MID-LLM framework, the present invention is proposed.
[0006] Therefore, the purpose of the present invention is to provide a method and system for collaborative medical image diagnosis based on the MID-LLM framework, which utilizes blockchain and LLM to perform target analysis of federated learning, thereby improving the accuracy and reliability of target analysis while improving data security and work efficiency.
[0007] To solve the above technical problems, the present invention provides the following technical solutions: a method for collaborative medical image diagnosis based on the MID-LLM framework, comprising the following steps:
[0008] Initialize the global model parameters, obtain the initial model of federated learning, upload the initial model of federated learning to the InterPlanetary File System, and store the generated hash link in the shared smart contract;
[0009] Retrieving the initial parameters in the Interplanetary File System through the hash link to start the federated learning process;
[0010] Integrate the image processing module into the federated learning model, and locally train the federated learning model in combination with the medical image data of the local institution to obtain updated local model parameters, store the local model parameters and the average reward score in the InterPlanetary File System, and record the hash link of the stored model parameters on the blockchain through a shared smart contract;
[0011] The shared smart contract downloads the local model parameters from the InterPlanetary File System and verifies the local model parameters;
[0012] Aggregate the local model parameters of multiple local institutions at the collaborator, perform weighted aggregation on the local model parameters, update the global parameters of the federated learning model, upload the global parameters of the federated learning model to the InterPlanetary File System, record the hash link of the federated learning model on the blockchain through a shared smart contract, and reward honest participants and punish dishonest participants based on the consistency of the aggregated parameters;
[0013] The process from local model training to global model update is iteratively trained, and the consensus mechanism of the blockchain is used to verify and update the global model parameters.
[0014] As a preferred solution of the method for collaborative medical image diagnosis based on the MID-LLM framework described in the present invention, the image processing module is integrated into the federated learning model, and the federated learning model is locally trained in combination with the medical image data of the local institution to obtain updated local model parameters, including the following steps:
[0015] Obtain medical image data from a local institution as input data, and use an image encoder to extract key features from the input data;
[0016] The query processor interprets the query tokens associated with the input data;
[0017] The projection layer generates soft hints based on the extracted key features and query tokens;
[0018] Use LLM to interpret soft prompts and output LLM insights;
[0019] The insights from LLM are used to update the local model parameters to obtain a locally trained federated learning model.
[0020] As a preferred solution of the method for collaborative medical image diagnosis based on the MID-LLM framework of the present invention, the local institution parameters and the average reward score are stored in the Interplanetary File System, and the calculation formula of the average reward score is:
[0021]
[0022] in, represents the average reward score, N represents the total number of episodes, Q i represents the cumulative reward of the ith episode, represents the learning parameters used in computing these cumulative rewards.
[0023] As a preferred solution of the method for collaborative medical image diagnosis based on the MID-LLM framework described in the present invention, wherein: the global model parameters include the federated learning model parameters and hyperparameters initialized by the local organization using random values; the local organization parameters include global parameters, specific parameters and reward scores.
[0024] As a preferred solution of the method for collaborative medical image diagnosis based on the MID-LLM framework of the present invention, the formula for verifying the local institution parameters by the shared smart contract is:
[0025]
[0026] in, represents the verification result, σ represents the verification trade-off factor, t represents the specific time, k represents the local institution, represents the local model evaluation result of the local institution at a specific time, MER(W t ) represents the overall model evaluation result of the global model at a specific time.
[0027] As a preferred solution of the method for collaborative medical image diagnosis based on the MID-LLM framework described in the present invention, wherein: the local mechanism parameters are weighted aggregated to obtain the local mechanism aggregation weight, and the calculation formula for weighted aggregation of the local mechanism parameters is:
[0028]
[0029] in, represents the aggregation weight of the local institution, represents the model parameters corresponding to the local institution (k) in round t, represents the number of participations of the local agency k in the collaborative training session before round t, represents the rounds in which local institution k participates in the aggregation process, represents the average attributable reward score of local institution k in round t, and Summarizes the total contribution of local organization k before round t, where T represents the number of training rounds;
[0030] The calculation formula is:
[0031]
[0032] in, represents the contribution of local organization k in the tth round, T represents the number of training rounds, represents the contribution of local institution k in round t-1;
[0033] The calculation formula is:
[0034]
[0035] in, represents the contribution of local institution k in round t, SAW t represents the sum of all aggregate weights assigned to effective parameters in round t, Represents the aggregation weight of the local institution; SAW t The calculation formula is:
[0036]
[0037] Among them, D t represents the local authority that considers the parameters valid in round t, SAW t represents the sum of all aggregate weights assigned to effective parameters in round t, Represents the aggregation weight of the local institution.
[0038] As a preferred solution of the method for collaborative medical image diagnosis based on the MID-LLM framework of the present invention, the expression of the federated learning model parameters is:
[0039]
[0040] Among them, SAW t represents the sum of all aggregate weights assigned to effective parameters in round t, represents the aggregation weight of the local institution, represents the model parameters corresponding to the local institution k in round t, W t+1 represents the updated federated learning model parameters, W t Represents the updated federated learning model parameters.
[0041] As a preferred solution of the method for collaborative medical image diagnosis based on the MID-LLM framework of the present invention, the update formula of the blockchain network by updating the hash link of the federated learning model is:
[0042]
[0043] Among them, GHL t represents the hash link of the federated learning model in round t, GHL t+1 represents the hash link of the federated learning model in round t+1, w represents the collaborator, HL represents the hash link of the aggregated parameters submitted by the collaborator, and L t represents the group of participants that submit the aggregated results to the shared smart contract for round T, w kt Indicates the aggregation result of the collaborator's uploads, H indicates that the results show consistent aggregation parameters L t collaborators in .
[0044] A collaborative medical image diagnosis system based on the MID-LLM framework, comprising: an initial parameter storage module, a federated learning startup module, a local institution parameter storage module, a parameter verification module, a weighted aggregation module, and a participant reward and punishment module;
[0045] The initial parameter storage module is used to initialize the global model parameters, obtain the initial model of federated learning, upload the initial model of federated learning to the Interstellar File System, and store the generated hash link in the shared smart contract;
[0046] A federated learning startup module, used to retrieve the initial parameters in the Interstellar File System through the hash link to start the federated learning process;
[0047] A local training module, which is used to integrate the image processing module into the federated learning model, and locally train the federated learning model in combination with the medical image data of the local institution to obtain updated local model parameters, store the local model parameters and the average reward score in the InterPlanetary File System, and record the hash link of the stored model parameters on the blockchain through a shared smart contract;
[0048] A parameter verification module, in which the shared smart contract downloads the local model parameters from the InterPlanetary File System and verifies the local model parameters;
[0049] The collaborator aggregation module is used to aggregate the local model parameters of multiple local institutions at the collaborator to obtain the local institution aggregation weight, update the global parameters of the federated learning model, upload the global parameters of the federated learning model to the InterPlanetary File System, record the hash link of the federated learning model on the blockchain through a shared smart contract, and reward honest participants and punish dishonest participants based on the consistency of the aggregation parameters;
[0050] The global model update module is used to iteratively train the process from local model training to global model update, and use the consensus mechanism of the blockchain to verify and update the global model parameters.
[0051] Beneficial effects of the present invention:
[0052] 1. The present invention uses blockchain technology to achieve decentralized collaboration, avoid single point failure and data leakage risks, and improve the security and reliability of the system;
[0053] 2. The present invention utilizes an improved aggregation algorithm to improve the model convergence speed and performance and ensure the unbiasedness of the global model;
[0054] 3. The present invention uses the InterPlanetary File System storage technology to achieve efficient and secure model parameter sharing, reduce communication costs and improve collaboration efficiency;
[0055] 4. The present invention detects and prevents attacks by malicious participants through a verification mechanism, ensuring the accuracy and integrity of model parameters;
[0056] 5. The present invention utilizes the powerful language understanding capability of LLM to extract richer information from medical images and provide more accurate diagnosis results. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:
[0058] Figure 1 It is a flow chart of a method for collaborative medical image diagnosis based on the MID-LLM framework of the present invention;
[0059] Figure 2 It is an overall schematic diagram of the collaborative medical image diagnosis system based on the MID-LLM framework of the present invention. DETAILED DESCRIPTION
[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.
[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0062] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.
[0063] Secondly, the present invention is described in detail with reference to the schematic diagram. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.
[0064] Reference Figure 1-Figure 2 :
[0065] Embodiment 1:
[0066] A method for collaborative medical image diagnosis based on the MID-LLM (Medicals Image Dignose-Large Language Model) framework, such as Figure 1 As shown, the following steps are included:
[0067] S100, initialize global model parameters, obtain the initial model of federated learning, upload the initial model of federated learning to the Interstellar File System, and store the generated hash link in the shared smart contract;
[0068] S200, retrieving the initial parameters in the Interstellar File System through the hash link, and starting the federated learning process;
[0069] S300, integrating the image processing module into the federated learning model, and locally training the federated learning model in combination with the medical image data of the local institution to obtain updated local model parameters, storing the local model parameters and the average reward score in the InterPlanetary File System, and recording the hash link of the stored model parameters on the blockchain through a shared smart contract;
[0070] S400, the shared smart contract downloads the local model parameters from the Interplanetary File System and verifies the local model parameters;
[0071] S500, aggregate the local model parameters of multiple local institutions at the collaborator, perform weighted aggregation on the local model parameters, update the global parameters of the federated learning model, upload the global parameters of the federated learning model to the InterPlanetary File System, record the hash link of the federated learning model on the blockchain through a shared smart contract, and reward honest participants and punish dishonest participants based on the consistency of the aggregated parameters;
[0072] S600, iterative training is performed from local model training to global model update, and the consensus mechanism of the blockchain is used to verify and update the global model parameters.
[0073] The present invention utilizes blockchain technology to realize decentralized collaboration, avoids single point failure and data leakage risks, and improves the security and reliability of the system; utilizes an improved aggregation algorithm to improve the model convergence speed and performance, and ensures the unbiasedness of the global model; utilizes Interstellar File System storage technology to realize efficient and secure model parameter sharing, reduce communication costs and improve collaboration efficiency; detects and prevents attacks by malicious participants through a verification mechanism, and ensures the accuracy and integrity of model parameters; utilizes LLM's powerful language comprehension capabilities to extract richer information from medical images and provide more accurate diagnostic results.
[0074] Step S100 initializes the global model parameters, obtains the federated learning initial model, uploads the federated learning initial model to the InterPlanetary File System, and stores the generated hash link in the shared smart contract, including the following three steps:
[0075] S110: The MID-LLM framework includes local institutions (hospitals in this embodiment) and collaborators (research centers in this embodiment), and participants deposit a certain amount of encrypted assets (such as cryptocurrency) into the reward smart contract as a commitment to participate in the process;
[0076] S120: After the hospital and research center are registered to the MID-LLM framework through the registration smart contract, each participant will obtain a pair of private and public keys for secure communication in the blockchain network;
[0077] S130: After registration is completed, the participating local institutions use random values to initialize the federated learning model parameters and hyperparameters (such as the number of local iterations, learning rate, etc.), upload the initial parameters to IPFS (Interplanetary File System), and store the generated hash link in the shared smart contract.
[0078] In step S200, the initial parameters in IPFS are retrieved through the hash link to start the federated learning process. Specifically:
[0079] To begin the collaborative learning process, the local agency first retrieves the global model parameters stored in IPFS, which they do by using a hash link written in a shared smart contract.
[0080] In step S300, the image processing module is integrated into the federated learning model, and the federated learning model is locally trained in combination with the medical image data of the local institution to obtain updated local model parameters, the local model parameters and the average reward score are stored in the InterPlanetary File System, and the hash link of the stored model parameters is recorded on the blockchain through a shared smart contract, including the following steps:
[0081] S310: Integrate the image processing module into the federated learning model, where multiple local institutions handle various medical image analysis tasks and collaborate to train the image processing model. Each local institution locally trains the federated learning model on the medical image data it collects to obtain local institution parameters. Specifically,
[0082] The image processing module is integrated into the federated learning model, and the federated learning model is locally trained in combination with the local institution image data to obtain the local institution parameters, including the following steps:
[0083] Obtain local organization image data as input data, and use an image encoder to extract key features from the input data;
[0084] The query processor interprets the query tokens associated with the input data;
[0085] The projection layer generates soft hints based on the extracted key features and query tokens;
[0086] Use LLM to interpret soft prompts and output LLM insights;
[0087] Using the insights of LLM to update local model parameters to obtain a locally trained federated learning model, wherein the local institutional parameters include global parameters, specific parameters, and reward scores;
[0088] S320: The local mechanism parameters and the average reward score obtained by calculation are stored in the Interplanetary File System, and the hash link of the storage model parameters is recorded on the blockchain through a shared smart contract. Specifically,
[0089] The average reward score is calculated as follows:
[0090]
[0091] in, represents the average reward score, N represents the total number of episodes, Q i represents the cumulative reward of the ith episode, represents the learning parameters used in computing these cumulative rewards.
[0092] Step S400: The shared smart contract downloads the local model parameters from the InterPlanetary File System and verifies the local model parameters, including the following steps:
[0093] S410: The shared smart contract randomly selects a group of local organizations and downloads the local organization parameters from IPFS;
[0094] S420: Verify local mechanism model parameters;
[0095] The formula for verifying the local institution parameters by the shared smart contract is:
[0096]
[0097] in, represents the verification result, σ represents the verification trade-off factor, t represents the specific time, k represents the local institution, represents the local model evaluation result of the local institution at a specific time, MER(W t ) represents the overall model evaluation result of the global model at a specific time.
[0098] In step S500, the local model parameters of multiple local institutions are aggregated at the collaborator, the local model parameters are weightedly aggregated, the global parameters of the federated learning model are updated, the global parameters of the federated learning model are uploaded to the InterPlanetary File System, the hash link of the federated learning model is recorded on the blockchain through a shared smart contract, and honest participants are rewarded and dishonest participants are punished according to the consistency of the aggregated parameters, including the following steps:
[0099] S510: Aggregating the local model parameters of multiple local institutions at the collaborator, the calculation formula for weighted aggregation of the local model parameters of the local institutions is:
[0100]
[0101] in, represents the aggregation weight of the local institution, represents the model parameters corresponding to the local institution (k) in round t, represents the number of participations of the local agency k in the collaborative training session before round t, represents the rounds in which local institution k participates in the aggregation process, represents the average attributable reward score of local institution k in round t, and Summarizes the total contribution of local organization k before round t, where T represents the number of training rounds;
[0102] The calculation formula is:
[0103]
[0104] in, represents the contribution of local organization k in the tth round, T represents the number of training rounds, represents the contribution of local institution k in round t-1,
[0105] The calculation formula is:
[0106]
[0107] in, represents the contribution of local institution k in round t, SAW t represents the sum of all aggregate weights assigned to effective parameters in round t, represents the aggregation weight of the local institution;
[0108] SAW t The calculation formula is:
[0109]
[0110] Among them, D t represents the local authority that considers the parameters valid in round t, SAW t represents the sum of all aggregate weights assigned to effective parameters in round t, represents the aggregation weight of the local institution;
[0111] S520: Update the federated learning model parameters, upload the obtained federated learning model parameters to the InterPlanetary File System, and record the federated learning model hash link on the blockchain through a shared smart contract. The expression of the federated learning model parameters is:
[0112]
[0113] Among them, SAW t represents the sum of all aggregate weights assigned to effective parameters in round t, represents the aggregation weight of the local institution, represents the model parameters corresponding to the local institution k in round t, W t+1 represents the updated federated learning model parameters, W t Represents the updated federated learning model parameters.
[0114] In step S600, the process from local model training to global model update is iteratively trained, and the consensus mechanism of the blockchain is used to verify and update the global model parameters. The blockchain network updates the hash link of the federated learning model by the following update formula:
[0115]
[0116] Among them, GHL t represents the hash link of the federated learning model in round t, GHL t+1 represents the hash link of the federated learning model in round t+1, w represents the collaborator, HL represents the hash link of the aggregated parameters submitted by the collaborator, and L t represents the group of participants that submit the aggregated results to the shared smart contract for round T, w kt Indicates the aggregation result of the collaborator's uploads, H indicates that the results show consistent aggregation parameters L t collaborators in .
[0117] The present invention utilizes blockchain technology to realize decentralized collaboration, avoids single point failure and data leakage risks, and improves the security and reliability of the system; utilizes an improved aggregation algorithm to improve the model convergence speed and performance, and ensures the unbiasedness of the global model; utilizes Interstellar File System storage technology to realize efficient and secure model parameter sharing, reduce communication costs and improve collaboration efficiency; detects and prevents attacks by malicious participants through a verification mechanism, and ensures the accuracy and integrity of model parameters; utilizes LLM's powerful language comprehension capabilities to extract richer information from medical images and provide more accurate diagnostic results.
[0118] Embodiment 2:
[0119] A collaborative medical image diagnosis system based on the MID-LLM framework, such as Figure 2 As shown, it includes an initial parameter storage module 100, a federated learning startup module 200, a local training module 300, a parameter verification module 400, a collaborator aggregation module 500, and a global model update module 600.
[0120] The initial parameter storage module is used to initialize the global model parameters, obtain the initial model of federated learning, upload the initial model of federated learning to the Interstellar File System, and store the generated hash link in the shared smart contract;
[0121] A federated learning startup module, used to retrieve the initial parameters in the Interstellar File System through the hash link to start the federated learning process;
[0122] A local training module, which is used to integrate the image processing module into the federated learning model, and locally train the federated learning model in combination with the medical image data of the local institution to obtain updated local model parameters, store the local model parameters and the average reward score in the InterPlanetary File System, and record the hash link of the stored model parameters on the blockchain through a shared smart contract;
[0123] A parameter verification module, in which the shared smart contract downloads the local model parameters from the InterPlanetary File System and verifies the local model parameters;
[0124] The collaborator aggregation module is used to aggregate the local model parameters of multiple local institutions at the collaborator to obtain the local institution aggregation weight, update the global parameters of the federated learning model, upload the global parameters of the federated learning model to the InterPlanetary File System, record the hash link of the federated learning model on the blockchain through a shared smart contract, and reward honest participants and punish dishonest participants based on the consistency of the aggregation parameters;
[0125] The global model update module is used to iteratively train the process from local model training to global model update, and use the consensus mechanism of the blockchain to verify and update the global model parameters.
[0126] Various changes and modifications can be made without departing from the spirit and scope of the present invention, and all equivalent technical solutions also belong to the scope of the present invention.
[0127] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0128] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0129] The present invention is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0132] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A method for collaborative medical image diagnosis based on the MID-LLM framework, characterized in that: The following steps are involved: Initialize the global model parameters, obtain the initial model of federated learning, upload the initial model of federated learning to the InterPlanetary File System, and store the generated hash link in the shared smart contract; Retrieving the initial parameters in the Interplanetary File System through the hash link to start the federated learning process; Integrate the image processing module into the federated learning model, and locally train the federated learning model in combination with the medical image data of the local institution to obtain updated local model parameters, store the local model parameters and the average reward score in the InterPlanetary File System, and record the hash link of the stored model parameters on the blockchain through a shared smart contract; The shared smart contract downloads the local model parameters from the InterPlanetary File System and verifies the local model parameters; Aggregate the local model parameters of multiple local institutions at the collaborator, perform weighted aggregation on the local model parameters, update the global parameters of the federated learning model, upload the global parameters of the federated learning model to the InterPlanetary File System, record the hash link of the federated learning model on the blockchain through a shared smart contract, and reward honest participants and punish dishonest participants based on the consistency of the aggregated parameters; The process from local model training to global model update is iteratively trained, and the consensus mechanism of the blockchain is used to verify and update the global model parameters.
2. The method for collaborative medical image diagnosis based on the MID-LLM framework according to claim 1, characterized in that: Integrate the image processing module into the federated learning model and locally train the federated learning model with the medical image data of the local institution to obtain updated local model parameters, including the following steps: Obtain medical image data from a local institution as input data, and use an image encoder to extract key features from the input data; The query processor interprets the query tokens associated with the input data; The projection layer generates soft hints based on the extracted key features and query tokens; Use LLM to interpret soft prompts and output LLM insights; The insights from LLM are used to update the local model parameters to obtain a locally trained federated learning model.
3. The method for collaborative medical image diagnosis based on the MID-LLM framework according to claim 2, characterized in that: The local institution parameters and the average reward score are stored in the Interplanetary File System. The calculation formula for the average reward score is: in, represents the average reward score, N represents the total number of episodes, Q i represents the cumulative reward of the ith episode, represents the learning parameters used in computing these cumulative rewards.
4. The method for collaborative medical image diagnosis based on the MID-LLM framework according to claim 3 is characterized in that: The global model parameters include the federated learning model parameters and hyperparameters initialized by the local institution using random values; the local institution parameters include global parameters, specific parameters and reward scores.
5. The method for collaborative medical image diagnosis based on the MID-LLM framework according to claim 1, characterized in that: The formula for verifying the local institution parameters by the shared smart contract is: in, represents the verification result, σ represents the verification trade-off factor, t represents the specific time, k represents the local institution, represents the local model evaluation result of the local institution at a specific time, MER(W t ) represents the overall model evaluation result of the global model at a specific time.
6. The method for collaborative medical image diagnosis based on the MID-LLM framework according to claim 5, characterized in that: The local institution parameters are weighted and aggregated to obtain the local institution aggregation weight. The calculation formula for weighted aggregation of local institution parameters is: in, represents the aggregation weight of the local institution, represents the model parameters corresponding to the local institution (k) in round t, represents the number of participations of the local agency k in the collaborative training session before round t, represents the rounds in which local institution k participates in the aggregation process, represents the average attributable reward score of local institution k in round t, and Summarizes the total contribution of local organization k before round t, where T represents the number of training rounds; The calculation formula is: in, represents the contribution of local organization k in the tth round, T represents the number of training rounds, represents the contribution of local institution k in round t-1; The calculation formula is: in, represents the contribution of local institution k in round t, SAW t represents the sum of all aggregate weights assigned to effective parameters in round t, Represents the aggregation weight of the local institution; SAW t The calculation formula is: Among them, D t represents the local authority that considers the parameters valid in round t, SAW t represents the sum of all aggregate weights assigned to effective parameters in round t, Represents the aggregation weight of the local institution.
7. The method for collaborative medical image diagnosis based on the MID-LLM framework according to claim 1, characterized in that: The expression of the federated learning model parameters is: Among them, SAW t represents the sum of all aggregate weights assigned to effective parameters in round t, represents the aggregation weight of the local institution, represents the model parameters corresponding to the local institution k in round t, W t+1 represents the updated federated learning model parameters, W t Represents the updated federated learning model parameters.
8. The method for collaborative medical image diagnosis based on the MID-LLM framework according to claim 1, characterized in that: The update formula of the blockchain network by updating the hash link of the federated learning model is: Among them, GHL t represents the hash link of the federated learning model in round t, GHL t+1 represents the hash link of the federated learning model in round t+1, w represents the collaborator, HL represents the hash link of the aggregated parameters submitted by the collaborator, and L t represents the group of participants that submit the aggregated results to the shared smart contract for round T, Indicates the aggregation result of the collaborator's uploads, H indicates that the results show consistent aggregation parameters L t collaborators in .
9. A collaborative medical image diagnosis system based on the MID-LLM framework, characterized in that: include: Initial parameter storage module, federated learning startup module, local institution parameter storage module, parameter verification module, weighted aggregation module, participant reward and punishment module; The initial parameter storage module is used to initialize the global model parameters, obtain the initial model of federated learning, upload the initial model of federated learning to the Interstellar File System, and store the generated hash link in the shared smart contract; A federated learning startup module, used to retrieve the initial parameters in the Interstellar File System through the hash link to start the federated learning process; A local training module, which is used to integrate the image processing module into the federated learning model, and locally train the federated learning model in combination with the medical image data of the local institution to obtain updated local model parameters, store the local model parameters and the average reward score in the InterPlanetary File System, and record the hash link of the stored model parameters on the blockchain through a shared smart contract; A parameter verification module, in which the shared smart contract downloads the local model parameters from the InterPlanetary File System and verifies the local model parameters; The collaborator aggregation module is used to aggregate the local model parameters of multiple local institutions at the collaborator to obtain the local institution aggregation weight, update the global parameters of the federated learning model, upload the global parameters of the federated learning model to the InterPlanetary File System, record the hash link of the federated learning model on the blockchain through a shared smart contract, and reward honest participants and punish dishonest participants based on the consistency of the aggregation parameters; The global model update module is used to iteratively train the process from local model training to global model update, and use the consensus mechanism of the blockchain to verify and update the global model parameters.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.