Medical image classification task-oriented clustering federal learning method based on model isomerism of super network

Abstract

The invention belongs to the field of medical image classification, and particularly discloses a medical image classification task-oriented model heterogeneous clustering federated learning method based on a super network, and the method comprises the steps: generating model parameters needed by medical institutions participating in federated training through employing the super network, enabling the medical institutions as clients to select personalized models, and enabling the personalized models to be more personalized. When the local medical image data is private, the model architecture of the user is not disclosed, the user only needs to inform the server of the parameter quantity required by the personalized model, and after the server generates the parameter and sends the parameter to the client, the client only needs to return the gradient of the corresponding number and dimension. In addition, the invention further provides a clustering method based on the hypernetwork gradient and client local medical image data distribution, the method enables the distribution of the medical image data in the clustered client group to be similar, reduces the model isomerism influence as much as possible, improves the federal learning performance, and improves the user experience. And it is ensured that a medical institution can obtain a personalized model with better performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image classification and federated learning, and in particular relates to a clustering federated learning method based on a hypernetwork and model heterogeneity for medical image classification tasks. Background Art

[0002] Federated Learning (FL) enables each client to train a model on its own data and communicate with other local models in the server to aggregate into a more general global model without collecting any privacy-sensitive raw data from the client. Taking the most representative FL framework FedAvg as an example, each client and server collaboratively trains a homogeneous model with the same architecture throughout the training process. More specifically, the server updates the global model by aggregating local model parameters and periodically sends the average global model parameters to all clients in each round of communication. As can be seen from the above methods, the classic FL framework usually requires giving clients the same model architecture for parameter aggregation. However, when each client designs its personalized model, the traditional federated learning framework cannot meet the needs.

[0003] To overcome these problems, Personalized Federated Learning (PFL) aims to personalize the global model for each client in the federated devices. However, so far, there has been little research on the problem of model heterogeneity in personalized federated learning. In the traditional federated learning framework, all clients jointly train the global model. However, considering different application scenarios, each client may need to build a different local model, which cannot be met in the current federated learning framework. For example, some clients may deploy the federated learning model on edge devices, and some clients require the trained model to adapt to local data. Therefore, the network architecture needs to be designed separately.

[0004] To achieve a more general training framework in FL, Federated Distillation (FD) has been proposed to aggregate local soft predictions in servers instead of local model parameters to achieve collaborative learning with robustness and efficient communication. Specifically, Jeong et al. proposed FD to perform class soft prediction aggregation and knowledge distillation, enabling multiple clients to collaboratively train heterogeneous models in a privacy-preserving manner. The original FD aggregates soft predictions for each label from clients and sends the global average per-label soft prediction back to each client. It requires that each client must have the complete labels of the task, which is impractical in FL because the data distribution is usually non-IID. Then, Li et al. proposed a public dataset-based method, called FedMD, to transfer knowledge between clients. After initial pre-training on a labeled public dataset (stored on a centralized server and periodically distributed to clients), each client trains a local model based on its own private data and uploads the soft predictions synthesized from the public dataset to the centralized server to achieve knowledge transfer. Since FedMD trains client models on labeled public and private datasets simultaneously, the model classifier must include all classes from both datasets. Another similar approach relies on unlabeled datasets such as DS-FL, FedAUX, FedDF, FedAD, and FCCL. However, both of these public datasets need to be manually selected before training begins, which reduces the generality and robustness of the training algorithm. In addition, choosing different public datasets may lead to different distillation performance.

[0005] Therefore, it is crucial to study a more flexible method to perform the knowledge extraction phase. Since hypernetworks can map the embedding of the target task to the corresponding model parameters to generate the parameters of other neural networks, inspired by this, hypernetworks can be used to generate heterogeneous client models. For example, Shamsian et al. were the first to apply hypernetworks in FL, which can generate effective personalized model parameters for each client. In this method, each client has an ID, and the embedding vector is generated by inputting the ID into the embedding layer, and then the embedding vector is input into the hypernetwork to generate the corresponding personalized model. The embedding layer and the hypernetwork use the client's weight update to obtain the gradient through chain derivation and then update. Another method using hypernetworks is pFedLA, which assumes that the model has similarities between layers, so different weights need to be used when aggregating between model layers. It maintains a hypernetwork for each client on the server to generate weights between model layers when aggregating. However, these methods are only discussed on the basis of the same client model architecture.

[0006] The privacy-preserving nature of federated learning makes it well-used in the medical field. In the medical field, medical data such as electronic medical records and medical images play a foundational role in the realization of precision health. On the other hand, these data are often scattered everywhere and are difficult to share due to their sensitivity. For example, smart watches collect personal real-time heartbeat data. Researchers try to use these data, combined with electronic medical records, to train models to assist doctors in making judgments. However, smart watch manufacturers and hospital medical records refuse to disclose data for privacy reasons. Using federated learning can ensure that sensitive patient data remains in the hands of local institutions or individual consumers and is not leaked to model trainers during the federated machine learning process. In addition, if only data from a specific population is used for training, the model may be biased towards a specific data distribution. A more reliable method is to be able to train from an electronic medical record data set that is more representative of the global population, but this data goes beyond a single hospital or location and requires data aggregation, which is almost impossible to achieve. Therefore, federated learning is particularly important.

[0007] Federated learning is currently being discussed on the basis of the same model architecture. When medical institutions require not only that medical data be private but also that personalized models be used and that the model architecture be private, current federated learning cannot meet the requirements. Summary of the invention

[0008] The purpose of the present invention is to propose a hypernetwork-based model heterogeneous federated learning method for medical image classification tasks, which is helpful to solve the problem that medical institutions need to disclose model architecture and model heterogeneity when participating in federated learning, while protecting the privacy of medical image data, meeting the personalization requirements of medical institution models and the privacy requirements of model architecture.

[0009] In order to achieve the above object, the present invention adopts the following technical scheme:

[0010] A hyper-network-based model heterogeneous federated learning method for medical image classification tasks, wherein the federated learning system includes a server and multiple medical institutions as clients;

[0011] The hypernetwork-based model heterogeneous federated learning method for medical image classification tasks comprises the following steps:

[0012] Step I.1. Initialize the client, determine the number of clients n and the number of parameters K required for each client;

[0013] Step I.2. Initialize the server and create an embedding layer and a hypernetwork of corresponding depth according to the information uploaded by the client. The depth of the embedding layer is Where N represents the dimension of the hypernetwork generation parameters;

[0014] Step I.3. Use the hypernetwork to generate the required parameters for all clients participating in the training;

[0015] Step I.4. The client loads the model parameters generated by the hypernetwork and trains a private model based on the local medical image dataset;

[0016] Step I.5. The client collects the gradients corresponding to the number of parameters during the training process and uploads them to the server;

[0017] Step I.6. The server receives the gradient uploaded by the client, and obtains the gradient of the embedding layer and the hypernetwork through chain derivation, thereby updating the embedding layer and the hypernetwork in the server;

[0018] Step I.7. Repeat steps I.3 to I.6 until the training is completed; after the training is completed, the medical institution obtains a personalized model for medical image classification by loading the personalized model parameters generated by the server hypernetwork.

[0019] In addition, based on the above-mentioned federated learning method, in order to alleviate the problems of model heterogeneity among medical institutions and non-independent and identically distributed medical image data, the present invention further proposes a clustering federated learning method based on model heterogeneity of a hypernetwork for medical image classification tasks. This method performs clustering based on the hypernetwork gradient and the distribution of client medical image data, which can make clients with similar medical image data distribution and less affected by model heterogeneity in the same group, effectively improving the performance of the federated learning method and ensuring that medical institutions can obtain personalized models with better performance.

[0020] In order to achieve the above object, the present invention adopts the following technical scheme:

[0021] A hyper-network-based model heterogeneous clustering federated learning method for medical image classification tasks, wherein the federated learning system includes a server and multiple medical institutions as clients;

[0022] The clustering federated learning method based on hypernetwork model heterogeneity for medical image classification tasks includes the following steps:

[0023] Step II.1. Initialize the client, determine the number of clients n and the number of parameters K required for each client;

[0024] Step II.2. Initialize the server and create an embedding layer and a hypernetwork of corresponding depth according to the information uploaded by the client. The depth of the embedding layer is And initialize the global small model parameters; where N represents the dimension of the hypernetwork generation parameters;

[0025] Step II.3. Generate the required parameters for the corresponding client using the super network of the corresponding cluster;

[0026] Step II.4. The client loads the model parameters generated by the super network of the corresponding cluster and trains the private model based on the local medical image dataset; at the same time, the client loads the global small model of the corresponding cluster and trains it based on the local medical image dataset;

[0027] Step II.5. The client collects the gradients of the corresponding number of parameters during the training of the private model, calculates the weight update of the local small model, and uploads it to the server;

[0028] Step II.6. The server receives the gradient and weight update uploaded by the client, and uses the client gradient to obtain the gradient of the embedding layer and the gradient of the super network of the corresponding cluster through chain derivation;

[0029] Step II.7. Check whether the client group of each cluster meets the clustering conditions based on the hypernetwork gradient or the local small model weight update; when one of the clustering conditions is met, perform K-means clustering to divide the cluster where the current client group is located into two clusters; and create a hypernetwork with the same hypernetwork parameters as the current cluster, and then assign a hypernetwork to each of the two newly divided clusters; when the current cluster does not meet the clustering conditions, do not execute the clustering algorithm;

[0030] Step II.8. The server updates the parameters of the corresponding embedding layer of each client, and then uses weighted aggregation to update the hypernetwork parameters of each cluster and the parameters of the global small model of each cluster;

[0031] Step II.9. Repeat steps II.3 to II.8 until the training is completed. After the training is completed, the medical institutions participating in the training are divided into different clusters on the server side. Each cluster has a hypernetwork that conforms to its medical image data characteristics and model characteristics. Therefore, the medical institutions obtain a personalized model for medical image classification by loading the personalized model parameters generated by the hypernetwork that conforms to the characteristics.

[0032] The present invention has the following advantages:

[0033] As described above, the method of the present invention aims at the problem that medical institutions are forced to use a unified model in traditional federated learning and must disclose the model architecture, and proposes a clustering federated learning method based on a hypernetwork with heterogeneous models for medical image classification tasks. Among them, the medical institution as a client needs to inform the server at the beginning of the parameter amount required for its personalized model. The server creates an embedding layer of corresponding depth for the client according to the parameter amount required by the client, and then inputs the embedding vector output by the embedding layer into the hypernetwork to directly generate the model parameters of the corresponding client, which effectively solves the problem of model heterogeneity in federated learning of medical institutions and provides a guarantee for each medical institution to use personalized models. In view of the problem that medical institutions do not want to expose the model architecture in some scenarios, the hypernetwork in the method of the present invention is only responsible for generating the parameter amount required by the client and transmitting it to the client in the form of a one-dimensional vector. The client only needs to return the gradient of the corresponding quantity and dimension, thereby effectively protecting the model architecture information of the client, which is very necessary in the case of medical information security. At the same time, the client can customize the required parameter amount, that is, the client can choose not to participate in the generation of parameters of certain layers, but to update them with local training, which also protects the model architecture of the client to a certain extent and can meet the personalized requirements of the client. In addition, on the basis of the above-mentioned hypernetwork-based model heterogeneity federated learning method, in order to alleviate the problems caused by the model heterogeneity of medical institutions and the non-independent and identically distributed medical image data, the present invention further proposes a clustering federated learning method based on hypernetwork model heterogeneity for medical image classification tasks. This method performs clustering based on hypernetwork gradient and client medical image data distribution, which can make medical institutions with similar medical image data distribution and less affected by model heterogeneity in the same group, effectively improving the performance of the federated learning method and ensuring that medical institutions obtain personalized models with better performance, so that they can use the personalized model to perform specific medical image classification tasks, thereby assisting doctors in making judgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of a federated learning method for model heterogeneity based on a hypernetwork in Example 1 of the present invention;

[0035] Figure 2 This is a flow chart of a clustering federated learning method for model heterogeneity based on a hypernetwork in Example 2 of the present invention;

[0036] Figure 3 Schematic diagram of the clustering method in Example 2 of the present invention;

[0037] Figure 4 This is a schematic diagram of a clustering federated learning method for model heterogeneity based on a hypernetwork in Example 2 of the present invention;

[0038] Figure 5This is a generalization principle diagram of the clustering federated learning method based on hypernetwork model heterogeneity in Example 2 of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0040] Example 1

[0041] like Figure 1 As shown, this embodiment 1 describes a hypernetwork-based model heterogeneous federated learning method for medical image classification tasks, and the federated learning system it targets includes a server and multiple medical institutions as clients.

[0042] The basic principle of this method is that in the process of federated learning, an embedding layer of corresponding depth can be created on the server according to the number of parameters required by the client model. The mapping relationship is a D (D<<N)-dimensional embedding vector. Through the hypernetwork, N-dimensional parameters can be generated, so the corresponding number of model parameters can be generated according to the embedding layer of corresponding depth. After loading the parameter training, the client returns the corresponding number of gradients, and then calculates the gradient of the server embedding layer and the hypernetwork by chain derivation. Since each client has a corresponding embedding layer on the server, the embedding layer can be updated according to the calculated gradient. For the hypernetwork, the weighted aggregation method based on the amount of medical image data is adopted to obtain the aggregated gradient, so as to update the parameters of the hypernetwork in the server.

[0043] A hypernetwork-based model heterogeneous federated learning method for medical image classification tasks includes the following steps:

[0044] Step I.1. Initialize the client, determine the number of clients n and the number of parameters K required for each client.

[0045] Step I.2. Initialize the server, determine the mapping relationship between the embedding vector dimension D and the hypernetwork generation parameter dimension N, and create the corresponding embedding layer e according to the information uploaded by client i i (·), the depth of each client embedding layer is Then create the hypernetwork h(·).

[0046] Step I.3. Use the hypernetwork to generate the required parameters for all clients participating in the training. The formula is as follows:

[0047]

[0048] Among them, e i (·) represents the embedding layer corresponding to client i, ψ i is the parameter of the embedding layer corresponding to client i, vi is the embedding vector output by all embedding rows in the embedding layer, represents the embedding vector output by the τth embedding row in the embedding layer, τ=1,…,d, i=1,…,n; h(·) represents the hypernetwork, are the parameters of the hypernetwork, Represents the parameters generated by the hypernetwork for the i-th client.

[0049] Step I.4. The client loads the model parameters generated by the hypernetwork and trains a private model based on the local medical image dataset.

[0050] Specifically, the client loads the formula of the model parameters generated by the hypernetwork, as shown in formula (4), and trains the formula of the private model based on the local medical image dataset, as shown in formula (5).

[0051]

[0052] in, represents the parameters generated by the hypernetwork for the i-th client, represents the parameters of the client retention layer, θ i represents the local model composed of the parameters generated by the hypernetwork loaded by client i and the parameters of the client reserved layer, η i is the learning rate of client i, Represents the loss function For parameter θ i The gradient of θ i ′ represents the updated parameters after the gradient descent step.

[0053] Step I.5. The client collects the gradients corresponding to the number of parameters during training And upload to the server.

[0054] Step I.6. The server receives the gradient uploaded by the client and obtains the gradient of the embedding layer and the hypernetwork through chain derivation, thereby updating the embedding layer and the hypernetwork in the server. The formula is as follows:

[0055]

[0056] Formula (6) and Formula (7) are expressions of the process of updating the super network in the server; where, is the gradient of client i with respect to the hypernetwork, Indicates the parameters uploaded by client i The gradient of represents the amount of medical image data of client i, represents the sum of the medical image data of all the clients participating in the training; α is the learning rate of the hypernetwork; represents the updated parameters after the gradient descent step.

[0057]

[0058] Formula (8) and Formula (9) are expressions of the process of updating the embedded layer in the server; where, is the gradient of the embedding layer corresponding to client i, β i is the learning rate of the embedding layer, Represents the hypernetwork parameters For client i, the corresponding embedding layer parameter ψ i The gradient of Representation parameters Hypernetwork model parameters The gradient of i is the parameter of the embedding layer corresponding to client i, ψ i ′ represents the updated parameters after the gradient descent step.

[0059] Step I.7. Repeat steps I.3 to I.6 until the training is completed. After the training is completed, the medical institution obtains a high-performance personalized model by loading the personalized model parameters generated by the server hypernetwork.

[0060] This model can be used to classify medical images, thereby accurately assisting doctors in making judgments. In the above-mentioned federated learning method, medical institutions as clients can choose their own models, that is, the models between clients can be heterogeneous, and clients can keep their own model architectures private, which is very necessary under the requirements of medical information security.

[0061] The federated learning method in the embodiment of the present invention solves the problem that medical institutions are forced to use a unified model in traditional federated learning by using a hypernetwork. In this method, medical institutions as clients can choose their own personalized models. Moreover, in this method, the client does not need to disclose its own model architecture, but only needs to inform the server of the required parameter quantity. After the server generates the parameters, the client only needs to return the corresponding number and dimensional gradient. Therefore, for the server, the client's model architecture is unknown, which is very necessary in the scenario of medical information security. At the same time, the client can choose not to participate in the generation of parameters of some layers, which further meets the client's personalization requirements and privacy requirements.

[0062] In addition, while solving the model heterogeneity problem in federated learning, the present invention also proposes a clustering method based on hypernetwork gradient and client medical image data distribution in the following embodiment 2. The medical image data distribution within the clustered group is similar and is less affected by model heterogeneity, which effectively improves the performance of the federated learning method. The hypernetwork can generate model parameters for the client without knowing the model, and can perform well in non-independent and identically distributed scenarios.

[0063] Example 2

[0064] This embodiment 2 describes a clustering federated learning method based on hypernetwork model heterogeneity for medical image classification tasks. In order to alleviate the impact of model heterogeneity among medical institutions and non-independent and identically distributed medical image data, this method adds a clustering method based on hypernetwork model gradient and client medical image data distribution. The clustering process, principle and flow chart are respectively as follows: Figure 2 , Figure 3 , Figure 4 As shown, the medical image data distribution within the clustered groups is similar and is less affected by model heterogeneity, which effectively improves the performance of the federated learning method and ensures that medical institutions obtain personalized models with better performance.

[0065] In this embodiment, the clustering federated learning method based on model heterogeneity of the super network uses the gradient of the super network and the weight update of the local small model of the client to perform K-means clustering. By setting the clustering condition, when triggered, the current client group is divided into two clusters, and a super network with the same parameters as the current super network is assigned to each cluster. Normal training is still carried out within each cluster until the clustering condition is triggered again. After clustering based on the super network gradient, the gradient of the super network of each client in the cluster is closer in numerical size, reducing the impact of the gradient difference of different models on the aggregated gradient. At the same time, after clustering based on the weight update of the local small model of the client, the clients in the cluster will be more similar in the distribution of medical image data, reducing the impact of non-independent and identically distributed on the accuracy of the model, thereby improving the overall operation efficiency and performance of the federated learning method.

[0066] A clustering federated learning method based on hyper-network model heterogeneity for medical image classification tasks includes the following steps:

[0067] Step II.1. Initialize the client, determine the number of clients n and the number of parameters K required for each client.

[0068] Step II.2. Initialize the server, determine the mapping relationship between the embedding vector dimension D and the hypernetwork generation parameter dimension N, and create the corresponding embedding layer e according to the information uploaded by client i i (·), the depth of each client embedding layer is Then the hypernetwork h(·) is created and the global small model parameters w are initialized.

[0069] Initially, all clients are in the same cluster, so:

[0070] c 1 ={1,…,n} (10)

[0071] c(i)=1 i=1,…,n (11)

[0072] Where c(i) represents the cluster where client i is located; initially, all clients are in cluster 1, i.e., c 1 ;

[0073]

[0074] h c(i) (·)=h(·) (13)

[0075] Among them, w is the initialization parameter of the global small model, and h c(i) (·) represents the global small model and super network corresponding to the cluster where client i is located. Initially, all clients are in cluster 1, so they correspond to and h 1 (·).

[0076] Step II.3. Generate the required parameters for the corresponding client using the hypernetwork of the corresponding cluster.

[0077] The process by which each client generates the required parameters from the hypernetwork of its cluster is as follows:

[0078]

[0079] Among them, e i (·) represents the embedding layer corresponding to client i, ψ i is the parameter of the embedding layer corresponding to client i, v i The embedding vectors output for all embedding rows in the embedding layer, represents the embedding vector output by the τth embedding row in the embedding layer, is the parameter of the super network of the cluster corresponding to client i, represents the parameters generated by the super network of the cluster where client i is located for client i, h c(i) (·) represents the super network of the cluster corresponding to client i.

[0080] Step II.4. The client loads the model parameters generated by the super network of the corresponding cluster and trains the private model based on the local medical image dataset; at the same time, the client loads the global small model of the corresponding cluster and trains it based on the local medical image dataset.

[0081] Specifically, the client participating in the training loads the global small model parameters of the cluster and the model parameters generated by the hypernetwork The training process based on the local medical image dataset is expressed as follows:

[0082]

[0083] Formula (17) and Formula (18) express the process of loading the parameters generated by the hypernetwork of the cluster in which the client is located and training based on the local medical image dataset; where θ i It indicates that client i loads the parameters generated by the hypernetwork of the corresponding cluster and combines them with the parameters of the client reserved layer to form a local model. represents the parameters of the client retention layer, η i is the learning rate, Represents the loss function For parameter θ i The gradient of θ i ′ represents the updated parameters after the gradient descent step.

[0084]

[0085] Formula (19) and Formula (20) express the process of loading the parameters of the global small model into the client local small model and training it based on the local medical image dataset. i is the learning rate, Represents the loss function For parameter w i The gradient of i represents the parameters of the global model synchronized by client i to its cluster; w i ′ represents the updated parameters after the gradient descent step.

[0086] Step II.5. The client collects the gradients corresponding to the number of parameters during the private model training process And calculate the weight update of the local small model Upload to the server.

[0087] Step II.6. The server receives the gradient and weight update uploaded by the client, and uses the client gradient to obtain the gradient of the embedding layer and the gradient of the super network of the corresponding cluster through chain derivation. The formula is as follows:

[0088]

[0089] in, is the gradient of the hypernetwork of the cluster to which client i belongs, is the gradient of the embedding layer by client i, Representation parameters Hypernetwork model parameters corresponding to clusters The gradient of Indicates the parameters uploaded by client i The gradient of Representation parameters For client i, the corresponding embedding layer parameter ψ i gradient.

[0090] Step II.7. Check whether the client group of each cluster meets the clustering conditions based on the super network gradient or the local small model weight update; when one of the clustering conditions is met, perform K-means clustering to divide the cluster where the current client group is located into two clusters; and create a super network with the same super network parameters as the current cluster, and then assign a super network to each of the two newly divided clusters; when the current cluster does not meet the clustering conditions, the clustering algorithm will not be executed.

[0091] like Figure 2 As shown, detect each cluster {c j |j=1,…,k} meets the clustering condition based on the distribution of medical image data, that is, satisfies and ∈ 1 ,∈ 2 is the clustering threshold, and k represents the number of clusters.

[0092] When the clustering threshold is reached, the client weight is updated to {Δw i |i∈c j}Execute K-means clustering and cluster the current cluster c j Divided into two clusters, namely c j and c k+1 , and create a hypernetwork h with the same hypernetwork parameters as the current one k+1 =h j , and assign a super network to each cluster, update cluster c k+1 The corresponding relationship of client i in is c(i)=k+1; normal hypernetwork-based model heterogeneous federated learning is still performed within each cluster; if the clustering condition is not met, training continues.

[0093] like Figure 2 As shown, detect each cluster {c j |j=1,…,k} whether the clustering condition based on the hypernetwork gradient is met, that is, and where ε 1 , ε 2 is the clustering threshold.

[0094] When the clustering condition is met, the gradient of the hypernetwork is Perform K-means clustering to divide the clients into two clusters, namely c j and c k+1 , and create a hypernetwork h with the same hypernetwork parameters as the current one k+1 =h j , each cluster is assigned a super network, and the cluster c is updated k+1 The corresponding relationship of client i in is c(i)=k+1; normal hypernetwork-based model heterogeneous federated learning is still performed within each cluster; if the clustering condition is not met, training continues.

[0095] Step II.8. The server updates the parameters of the embedding layer corresponding to each client, and then uses weighted aggregation to update the hypernetwork parameters of each cluster and the parameters of the global small model of each cluster. The process is as follows:

[0096]

[0097] in, represents the global small model corresponding to the j-th cluster, represents the amount of medical image data of client i, It represents the sum of the medical image data of all clients in the cluster. represents the model parameters of the hypernetwork corresponding to the jth cluster, represents the gradient of the j-th cluster client i to the cluster super network, ψ i represents the parameters of the embedding layer corresponding to client i, represents the gradient of the embedding layer corresponding to client i, ψ i ′ represents the updated parameter after the gradient descent step, α j ,γ j is the learning rate of each cluster hypernetwork and global small model, β i is the learning rate of each embedding layer.

[0098] Step II.9. Repeat steps II.3 to II.8 until the training is completed; after the training is completed, the participating medical institutions are divided into different clusters on the server side, and each cluster has a hypernetwork that conforms to its medical image data characteristics and model characteristics. Therefore, the medical institution obtains a personalized model by loading the personalized model parameters generated by the hypernetwork that conforms to the characteristics. This personalized model is then used for daily medical image classification tasks to help medical institution personnel conduct daily research.

[0099] In addition, in order to generalize to more clients, that is, for newly joined medical institutions, this embodiment further proposes a generalization mechanism based on a hypernetwork, which uses a trained hypernetwork for generalization, so that newly joined medical institutions can also obtain a personalized model with good performance. The generalization process is as follows:

[0100] For newly added clients, the server creates an embedding layer corresponding to the number of hypernetworks, that is, each embedding layer corresponds to a hypernetwork, and then uses the hypernetworks of different clusters in the server to generate parameters for the client. The client loads and trains with the local medical image dataset, and returns the gradient of the corresponding parameters in turn, thereby updating each embedding layer; after the training is completed, the accuracy of the model parameters generated by the hypernetworks of each cluster is compared, and the client belongs to the cluster with the highest generated model parameter accuracy. This operation is performed for all newly added clients to obtain the final generalization effect.

[0101] Specifically, if Figure 5 As shown, this embodiment performs the generalization process of clustering federated learning based on the heterogeneous model of the hypernetwork for all newly joined clients as follows:

[0102] Step III.1. Initialize the newly joined client n+1 and determine the required parameter quantity K.

[0103] Step III.2. On the server, based on the information uploaded by the client and the number of hypernetworks in the server, i.e., the number of clusters k, create an embedding layer with the corresponding depth and number, i.e. The depth of each embedding layer is in, They represent the k embedding layers created by the server for client n+1, and each embedding layer corresponds to a hypernetwork.

[0104] Step III.3. At the server, use the supernetwork of all clusters to generate parameters for the clients in turn.

[0105]

[0106] Among them, v j is the embedding vector generated by the jth embedding layer, is the parameter generated by the jth hypernetwork for the client, represents the embedding vector of the τth row output in the jth embedding layer, ψ j represents the parameters of the j-th embedding layer, represents the jth embedding layer, are the model parameters of the jth hypernetwork.

[0107] Step III.4. The client sequentially loads the parameters generated by the hypernetwork and trains based on the local medical image dataset.

[0108]

[0109] in, Parameters of the reserved layer for client n+1, η n+1 is the learning rate of client n+1, It means that client n+1 loads the parameters generated by the jth hypernetwork and combines them with the parameters of the client's reserved layer to form a local model. Represents the loss function Parameters gradient.

[0110] Step III.5. The client collects the gradients of the corresponding parameters during training And upload to the server; among them, Respectively represent the client n+1 upload parameters gradient.

[0111] Step III.6. The server receives the gradient uploaded by client n+1, and uses the client gradient to obtain the gradient of the embedding layer through chain derivation;

[0112]

[0113] in, is the gradient of each embedding layer corresponding to the newly added client n+1, β j is the learning rate for each embedding layer, Representation parameters For the j-th embedding layer parameter ψ j The gradient of Representation parameters For the jth hypernetwork model parameter The gradient of Indicates the client n+1 upload parameter The gradient of j represents the parameters of embedding layer j, ψ j ′ represents the updated parameters after the gradient descent step.

[0114] Step III.7. Repeat Step III.3 to Step III.6 until the training is completed; check which cluster's hypernetwork generates the highest parameter accuracy, and add the newly added client to this cluster;

[0115] Step III.8. Execute steps III.1 to III.7 for all newly joined clients until generalization is completed.

[0116] The method of the present invention uses a hypernetwork to generate the model parameters required by the medical institutions participating in the federated training, thereby solving the problem that in traditional federated learning, when the medical institutions, as clients, participate in the medical image tasks, they are forced to use a unified model and the model architecture is public. In this method, the medical institutions as clients can select personalized models, and while the local medical image data is private, they can also keep their own model architecture private. They only need to inform the server of the number of parameters required for the personalized model. After the server generates the parameters and sends them to the client, the client only needs to return the gradient of the corresponding number and dimension. Therefore, for the server, the medical image data and model architecture of the client are unknown. At the same time, the client can choose not to participate in the generation of the parameters of some layers of the model, further meeting the personalized requirements and privacy requirements of the client. In addition, while solving the problem of model heterogeneity, the present invention also proposes a clustering method based on the hypernetwork gradient and the distribution of local medical image data of the client, so that the distribution of medical image data within the clustered client group is similar, and is less affected by the model heterogeneity, thereby improving the performance of federated learning and ensuring that the medical institution obtains a personalized model with better performance, thereby assisting doctors in making judgments.

[0117] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A hypernetwork-based model heterogeneous federated learning method for medical image classification tasks, wherein the federated learning system includes a server and multiple medical institutions as clients; characterized in that: The hypernetwork-based model heterogeneous federated learning method for medical image classification tasks comprises the following steps: Step I.

1. Initialize the client, determine the number of clients m and the number of parameters K required for each client model; Step I.

2. Initialize the server and create an embedding layer and a hypernetwork of corresponding depth according to the information uploaded by the client. The depth of the embedding layer is Among them, M represents the dimension of hypernetwork generation parameters; Step I.

3. Use the hypernetwork to generate the required model parameters for all clients participating in the training; Step I.

4. The client loads the model parameters generated by the hypernetwork and trains a private model based on the local medical image dataset; Step I.

5. The client collects the gradients corresponding to the number of parameters during the training process and uploads them to the server; Step I.

6. The server receives the gradient uploaded by the client, and obtains the gradient of the embedding layer and the hypernetwork through chain derivation, thereby updating the embedding layer and the hypernetwork in the server; Step I.

7. Repeat steps I.3 to I.6 until the training is completed; after the training is completed, the medical institution obtains a personalized model by loading the personalized model parameters generated by the server hypernetwork.

2. The hypernetwork-based model heterogeneous federated learning method for medical image classification tasks according to claim 1, characterized in that: In step 1.3, the formula for using the hypernetwork to generate the required parameters for all clients participating in the training is expressed as follows: Among them, e i (·) represents the embedding layer corresponding to client i, ψ i is the parameter of the embedding layer corresponding to client i, v i is the embedding vector output by all embedding rows in the embedding layer, represents the embedding vector output by the τth embedding row in the embedding layer, τ=1,…,d, i=1,…,n; h(·) represents the hypernetwork, are the parameters of the hypernetwork, represents the parameters generated by the hypernetwork for the i-th client; In step 1.4, the client loads the formula of the model parameters generated by the hypernetwork, as shown in formula (4), and trains the formula of the private model based on the local medical image dataset, as shown in formula (5); in, represents the parameters generated by the hypernetwork for client i, represents the parameters of the reserved layer of client i, θ i represents the local model composed of the parameters generated by the hypernetwork loaded by client i and the parameters of the client reserved layer, η i is the learning rate of client i, Represents the loss function For parameter θ i The gradient of θ i ′ represents the updated parameters after the gradient descent step.

3. The hypernetwork-based model heterogeneous federated learning method for medical image classification tasks according to claim 1, characterized in that: In step 1.6, the server receives the gradient uploaded by the client, and obtains the gradient of the embedding layer and the hypernetwork through chain derivation, so as to update the embedding layer and the hypernetwork in the server. The formula is expressed as follows: Formula (6) and Formula (7) are expressions of the process of updating the super network in the server; where, is the gradient of client i to the hypernetwork, Representation parameters The gradient of the hypernetwork model parameter i, Indicates the parameters uploaded by client i The gradient of represents the amount of medical image data of client i, represents the sum of the medical image data of all the clients participating in the training; α is the learning rate of the hypernetwork; represents the updated parameters after the gradient descent step; Formula (8) and Formula (9) are expressions of the process of updating the embedded layer in the server; where, is the gradient of the embedding layer corresponding to client i, β i is the learning rate for each embedding layer, Represents the hypernetwork parameters For client i, the corresponding embedding layer parameter ψ i The gradient of Representation parameters Hypernetwork model parameters The gradient of i is the parameter of the embedding layer corresponding to client i, ψ i ′ represents the updated parameters after the gradient descent step.

4. A hypernetwork-based model heterogeneous clustering federated learning method for medical image classification tasks, wherein the federated learning system includes a server and multiple medical institutions as clients; characterized in that: The clustering federated learning method based on hypernetwork model heterogeneity for medical image classification tasks includes the following steps: Step II.

1. Initialize the client, determine the number of clients n and the number of parameters K required for each client; Step II.

2. Initialize the server and create an embedding layer and a hypernetwork of corresponding depth according to the information uploaded by the client. The depth of the embedding layer is And initialize the global small model parameters, N represents the dimension of the hypernetwork generation parameters; Step II.

3. Generate the required parameters for the corresponding client using the super network of the corresponding cluster; Step II.

4. The client loads the model parameters generated by the super network of the corresponding cluster and trains the private model based on the local medical image dataset; at the same time, the client loads the global small model of the corresponding cluster and trains it based on the local medical image dataset; Step II.

5. The client collects the gradients of the corresponding number of parameters during the training of the private model, calculates the weight update of the local small model, and uploads it to the server; Step II.

6. The server receives the gradient and weight update uploaded by the client, and uses the client gradient to obtain the gradient of the embedding layer and the gradient of the super network of the corresponding cluster through chain derivation; Step II.

7. Check whether the client group of each cluster meets the clustering conditions based on the hypernetwork gradient or the local small model weight update; when one of the clustering conditions is met, perform K-means clustering to divide the cluster where the current client group is located into two clusters; and create a hypernetwork with the same hypernetwork parameters as the current cluster, and then assign a hypernetwork to each of the two newly divided clusters; When the current cluster does not meet the clustering conditions, the clustering algorithm will not be executed; Step II.

8. The server updates the parameters of the corresponding embedding layer of each client, and then uses weighted aggregation to update the hypernetwork parameters of each cluster and the parameters of the global small model of each cluster; Step II.

9. Repeat steps II.3 to II.8 until the training is completed. After the training is completed, the medical institutions participating in the training are divided into different clusters on the server side. Each cluster has a hypernetwork that conforms to its medical image data characteristics and model characteristics. Therefore, the medical institutions obtain personalized models by loading personalized model parameters generated by the hypernetwork that conforms to the characteristics.

5. The hypernetwork-based model heterogeneous clustering federated learning method for medical image classification tasks according to claim 4, characterized in that: In step II.2, initially, all clients are in the same cluster, so: c1={1,…,n} (10) c(i)=1 i=1,…,n (11) Where c(i) represents the cluster where client i is located; initially, all clients are in cluster 1, i.e., c1; h c(i) (·)=h(·) (13) Among them, w is the initialization parameter of the global small model, and h c(i) (·) represents the global small model and super network corresponding to the cluster where client i is located; initially, all clients are in cluster 1, so and h c(i) (·) corresponds to and h1(·); In step II.3, the process by which each client generates the required parameters from the super network of its cluster is as follows: Among them, e i (·) represents the embedding layer corresponding to client i, ψ i is the parameter of the embedding layer corresponding to client i, v i The embedding vectors output for all embedding rows in the embedding layer, represents the embedding vector output by the τth embedding row in the embedding layer, is the parameter of the super network of the cluster corresponding to client i, The parameter h represents the hypernetwork of the cluster where client i is located, which is generated for client i. c(i) (·) represents the super network of the cluster corresponding to client i; In step II.4, the client participating in the training loads the global small model parameters of the cluster and the model parameters generated by the hypernetwork The training process based on the local medical image dataset is expressed as follows: Formula (17) and Formula (18) express the process of loading the parameters generated by the hypernetwork of the cluster in which the client is located and training based on the local medical image dataset; where Δ i It indicates that client i loads the parameters generated by the hypernetwork of the corresponding cluster and combines them with the parameters of the client reserved layer to form a local model. represents the parameters of the client retention layer, η i is the learning rate, Represents the loss function For parameter θ i The gradient of θ i ′ represents the updated parameters after the gradient descent step; Formula (19) and Formula (20) express the process of loading the parameters of the global small model into the client local small model and training based on the local medical image dataset; where σ i is the learning rate, Represents the loss function For parameter w i The gradient of i represents the parameters of the global model synchronized by client i to its cluster; w i ′ represents the updated parameters after the gradient descent step; In step II.5, client i calculates the weight update And upload to the server.

6. The hypernetwork-based model heterogeneous clustering federated learning method for medical image classification tasks according to claim 4, characterized in that: In step II.6, the server receives the gradient and weight update uploaded by the client, and uses the client gradient to obtain the gradient of the embedding layer and the gradient of the super network of the corresponding cluster through chain derivation. The formula is expressed as follows: in, is the gradient of the hypernetwork of the cluster to which client i belongs, is the gradient of the embedding layer by client i, Representation parameters Hypernetwork model parameters corresponding to clusters The gradient of Indicates the parameters uploaded by client i The gradient of Representation parameters For client i, the corresponding embedding layer parameter ψ i gradient.

7. The method of clustering federated learning based on hypernetwork model heterogeneity for medical image classification tasks according to claim 4, characterized in that: The step II.7 is specifically as follows: Detect each cluster {c j |j=1,…,k} meets the clustering condition based on the distribution of medical image data, that is, satisfies and ∈1, ∈2 are clustering thresholds, and k represents the number of clusters; When the clustering threshold is reached, the client weight is updated to {Δw i |i∈c j }Execute K-means clustering and cluster the current cluster c j Divided into two clusters, namely c j and c k+1 , and create a hypernetwork h with the same hypernetwork parameters as the current one k+1 =h j , and assign a super network to each cluster, update cluster c k+1 The corresponding relationship of client i in is c(i)=k+1; each cluster still performs normal hyper-network-based model heterogeneous federated learning; If the clustering condition is not met, continue training; Detect each cluster {c j |j=1,…,k} whether the clustering condition based on the hypernetwork gradient is met, that is, and ε1 and ε2 are the clustering occurrence thresholds; When the clustering condition is met, the gradient of the hypernetwork is Perform K-means clustering to divide the clients into two clusters, namely c j and c k+1 , and create a hypernetwork h with the same hypernetwork parameters as the current one k+1 =h j , each cluster is assigned a super network, and the cluster c is updated k+1 The corresponding relationship of client i in is c(i)=k+1; each cluster still performs normal hyper-network-based model heterogeneous federated learning; If the clustering condition is not met, continue training.

8. The hypernetwork-based model heterogeneous clustering federated learning method for medical image classification tasks according to claim 4, characterized in that: In step II.8, the parameters of the embedding layer corresponding to each client are updated, and then the parameters of the hypernetwork of each cluster and the parameters of the global small model of each cluster are updated by weighted aggregation as follows: in, represents the global small model corresponding to the j-th cluster, represents the amount of medical image data of client i, It represents the sum of the medical image data of all clients in the cluster. represents the model parameters of the hypernetwork corresponding to the jth cluster, represents the gradient of the j-th cluster client i to the cluster super network, ψ i represents the parameters of the embedding layer corresponding to client i, represents the gradient of the embedding layer corresponding to client i, ψ i ′ represents the updated parameter after the gradient descent step, α j ,γ j is the learning rate of each cluster super network and the global small model, β i is the learning rate of each embedding layer.

9. The hypernetwork-based model heterogeneous clustering federated learning method for medical image classification tasks according to claim 4, characterized in that: For a newly joined medical institution, it is treated as a new client, and an embedding layer corresponding to the number of hypernetworks is created on the server, that is, each embedding layer corresponds to a hypernetwork. Then, the hypernetworks of different clusters in the server are used in turn to generate parameters for the client. The client is loaded in turn and trained using the local medical image dataset, and the gradients of the corresponding parameters are returned in turn, thereby updating each embedding layer. After the training is completed, the accuracy of the model parameters generated by the hypernetworks of each cluster is compared, and the client belongs to the cluster with the highest generated model parameter accuracy. The medical institution then requests the server to use the hypernetwork in the cluster to generate the required model parameters, thereby obtaining a personalized model and achieving the medical image classification task.

10. The clustering federated learning method based on hypernetwork model heterogeneity for medical image classification tasks according to claim 9, characterized in that: The generalization process of clustering federated learning with heterogeneous models based on hypernetwork is as follows: Step III.

1. Initialize the newly added client n+1 and determine the required parameter quantity K; Step III.

2. On the server, based on the information uploaded by the client and the number of hypernetworks in the server, i.e., the number of clusters k, create an embedding layer with the corresponding depth and number, i.e. The depth of each embedding layer is in, They represent the k embedding layers created by the server for client n+1, and each embedding layer corresponds to a hypernetwork; Step III.

3. On the server, use the hypernetwork of all clusters to generate parameters for the client in turn; Among them, v j is the embedding vector generated by the jth embedding layer, is the parameter generated by the jth hypernetwork for the client, represents the embedding vector of the τth row output in the jth embedding layer, ψ j represents the parameters of the j-th embedding layer, represents the jth embedding layer, represents the model parameters of the jth hypernetwork; Step III.

4. The client loads the parameters generated by the hypernetwork in sequence and trains based on the local medical image dataset; in, Parameters of the reserved layer for client n+1, η n+1 is the learning rate of client n+1, It means that client n+1 loads the parameters generated by the jth hypernetwork and combines them with the parameters of the client's reserved layer to form a local model. Represents the loss function Parameters The gradient, represents the updated parameters after the gradient descent step; Step III.

5. The client collects the gradients of the corresponding parameters during training And upload to the server; among them, Respectively represent the parameters uploaded by client n+1 The gradient of Step III.

6. The server receives the gradient uploaded by client n+1, and uses the client gradient to obtain the gradient of the embedding layer through chain derivation; in, is the gradient of each embedding layer corresponding to the newly added client n+1, β j is the learning rate for each embedding layer, Representation parameters For the j-th embedding layer parameter ψ j The gradient of Representation parameters For the jth hypernetwork model parameter The gradient of Indicates the parameters uploaded by client n+1 The gradient of j represents the parameters of the embedding layer j; ψ j ′ represents the updated parameters after the gradient descent step; Step III.

7. Repeat Step III.3 to Step III.6 until the training is completed; check which cluster's hypernetwork generates the highest parameter accuracy, and add the newly added client to this cluster; Step III.

8. Execute steps III.1 to III.7 for all newly joined clients until generalization is completed.

Images