Federated personalized learning method and system based on knowledge-free distillation and gradient matching

Abstract

The present invention discloses a federated personalized learning method and system based on knowledge-free distillation and gradient matching. The present invention includes constructing an image classification model for each participating center, including a feature encoder, a classifier, a knowledge-free distillation module and a gradient matching module, respectively using the private data of each participating center to train the image classification model and calculate the classification loss, and combining the knowledge-free distillation module and the gradient matching module to calculate the federated variational distillation loss and the gradient matching loss; the classification loss, the federated variational distillation loss and the gradient matching loss are fused to form a total loss to achieve the optimization update of the model parameters of the image classification model of each participating center. The present invention aims to make full use of the multi-center domain invariant information while adding the constraints of domain-specific information, train a personalized federated deep model for each participating center, and achieve the optimal performance in the classification task of its local brain image or other images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a federated personalized learning method and system based on knowledge-free distillation and gradient matching. Background Art

[0002] Although deep learning methods have been widely used in the field of pattern recognition and analysis of brain imaging data and have achieved remarkable results. However, the effectiveness of data-driven deep learning methods is closely related to the amount of training data. Due to the improvement of data security protection regulations and people's concern about data privacy, data sharing and aggregation in distributed centers are becoming increasingly difficult. The dilemma of "data islands" makes it difficult for deep learning methods to further improve their performance in pattern recognition tasks of medical images, and also creates great difficulties for the promotion and application of more advanced deep learning models. The federated deep learning method is a secure multi-center distributed computing method proposed to solve this problem. It can collaboratively train multi-center deep models without transmitting sensitive data. However, due to the serious statistical heterogeneity of multi-center data medical imaging data, this may be due to the domain transfer between centers caused by factors such as different medical imaging acquisition equipment, preprocessing protocols and other intrinsic variables. Although previous federated deep learning methods have made some progress in overcoming the negative impact of data domain differences, the traditional federated deep learning aims to build a unique global federated model for each center. Due to the differences in data distribution among different participating centers, their optimal optimization directions may not be the same, so the only global federated deep learning model cannot achieve satisfactory performance in all participating centers. Therefore, how to extract globally available domain-invariant information from all participating centers in a heterogeneous federated environment, and further train a locally optimal federated personalized deep model for each participating center by learning domain-specific knowledge based on the domain-invariant information according to the specific data distribution of each participating center, has become a key technical problem that needs to be solved urgently. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a federated personalized learning method and system based on knowledge-free distillation and gradient matching are provided. The present invention aims to make full use of the invariant information of multi-center domains while adding constraints on domain-specific information, and train a personalized federated deep model for each participating center to achieve optimal performance in the classification tasks of its local brain images or other images.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A federated personalized learning method based on knowledge-free distillation and gradient matching includes the following steps:

[0006] S1, constructing an image classification model for each participating center respectively, including a feature encoder, a classifier, a knowledge-free distillation module and a gradient matching module, wherein the feature encoder is used to extract encoding features for brain structure magnetic resonance imaging data or images, the classifier is used to classify the encoding features, the knowledge-free distillation module is used to extract common features of domain-invariant information in each center in the federation, and the gradient matching module is used to learn domain-specific information according to the data distribution unique to each center, and the feature encoder, classifier, knowledge-free distillation module and gradient matching module are all trainable embedding networks;

[0007] S2, randomly divide the private data of each participating center into training set and test set;

[0008] S3, use the private data of each participating center to train the image classification model and calculate the classification loss; for each participating center, use the model parameters of the local classifier as the input of the knowledge-free distillation module to build a covariance matrix proprietary to the participating center to capture the personalized distribution information of the participating center, reconstruct the global knowledge information of other participating centers in each participating center after secure sharing to calculate the federated variational distillation loss using mutual information, and use the gradient information of the model parameters of the local classifier as the input of the gradient matching module to calculate the gradient matching loss using the information reconstructed with other participating centers;

[0009] S4, the classification loss, federated variational distillation loss and gradient matching loss are integrated to form the total loss, and the total loss is used to optimize and update the model parameters of the image classification model of each participating center;

[0010] S5, if it is necessary to continue training, jump to step S2; otherwise, end the training of each participating center, save the model parameters to obtain a personalized federated model that integrates global information.

[0011] Optionally, the classification loss in step S3 is a cross entropy loss, and the calculation function expression is:

[0012] ,

[0013] In the above formula, For the i Participating Centers The cross entropy loss is is the expected value, For the i Participating Centers Magnetic resonance imaging data or images of brain structures, For the i Participating Centers brain structure magnetic resonance imaging data or image labels, is the cross entropy function, For the i Participating Centers The prediction results of the image classification model are:

[0014] ,

[0015] In the above formula, For the i Participating Centers The classifier, and are batch normalization layer and activation layer functions respectively, For the i Participating Centers The feature encoder Feature encoder based Model parameters right The extracted coding features, , They represent the length, width and height of the brain structure magnetic resonance imaging data or image respectively.

[0016] Optionally, in step S3, the model parameters of the local classifier are used as the input of the knowledge-free distillation module to construct a covariance matrix specific to the participating center to capture the personalized distribution information of the participating center, and the global knowledge information of other participating centers is reconstructed in each participating center after secure sharing to calculate the federated variational distillation loss using mutual information, including:

[0017] S3.1, for each participating center, the model parameters of the local classifier are used as the input of the knowledge-free distillation module to construct a covariance matrix specific to the participating center to capture the personalized distribution information of the participating center. The knowledge-free distillation module is a graph convolution embedding network model consisting of C graph convolutional networks. , graph convolutional embedding network model The model parameters are , each graph convolutional network consists of L layers, and the function expression of the output of each layer is:

[0018] ,

[0019] In the above formula, and Respectively and The output of the layer, is the activation function; is a degree matrix whose diagonal elements By matrix Elements Sure: ,matrix pass To calculate, is the identity matrix, is the adjacency matrix, the adjacency matrix According to the input of the first layer calculate , the input of the first layer For the classifier The linear layer model parameters corresponding to the C categories in ; and the covariance of the fitted joint distribution is calculated according to the output of C graph convolutional networks according to the following formula :

[0020] ,

[0021] In the above formula, The covariance of the fitted joint distribution is The inverse matrix of represents the latent space features extracted by C graph convolutional networks, ~ are the d-dimensional latent space features extracted by the 1st to Cth graph convolutional networks, is a hyperparameter, is a unit matrix of the same dimension; and the gradient information is calculated according to the following formula :

[0022] ,

[0023] In the above formula, Represents dimension information, is the dimension of the model parameters, is the number of label categories, is the model parameter The gradient of is the crossover function, For the i Participating Centers The classifier, and are batch normalization layer and activation layer functions respectively, For the i Participating Centers The feature encoder Feature encoder based Model parameters right The extracted coding features, For the j Participating Centers Magnetic resonance imaging data or images of brain structures, For the j Participating Centers The brain structure magnetic resonance imaging data or image labels of , gradient information , model parameters Using encryption functions After encryption, it is passed to other participating centers for calculating the federated variational distillation loss;

[0024] S3.2, each participating center calculates its own material distillation loss based on the data transmitted by other participating centers:

[0025] ,

[0026] In the above formula, For the i Participating Centers The material distillation loss, is the expected value, is the encryption function, is the covariance of the fitted joint distribution; For the i Participating Centers The model parameters of the image classification model are: For the j Participating Centers The model parameters of the image classification model are, For the j Participating Centers Gradient information of

[0027] S3.3, each participating center calculates the federated variational distillation loss according to the following formula:

[0028] ,

[0029] In the above formula, For the i Participating Centers The federated variational distillation loss, is the number of participating centers, For the i Participating Centers of material distillation losses.

[0030] Optionally, the encryption function To use Gaussian noise, the function expression is:

[0031] ,

[0032] In the above formula, Indicates the encryption function Information that needs to be shared The encryption result is represents a mean of 0 and a variance of Gaussian noise.

[0033] Optionally, the calculation function expression of the gradient matching loss in step S3 is:

[0034] ,

[0035] ,

[0036] ,

[0037] ,

[0038] In the above formula, For the i Participating Centers Gradient matching loss, is the number of participating centers, For the i Participating Centers Gradient and j Participating Centers Gradient similarities between; represents the expected value, is the encryption function, represents the transpose operation, and Respectively i Participating Centers and j Participating Centers Magnetic resonance imaging data or images of brain structures, and Respectively i Participating Centers and j Participating Centers brain structure magnetic resonance imaging data or image labels, and Respectively i Participating Centers and j Participating Centers The training set, For the i Participating Centers Gradient matching module Model parameters The gradient of is the cross entropy loss, For the i Participating Centers The feature encoder Feature encoder based Model parameters right The extracted coding features, For the j Participating Centers The feature encoder Feature encoder based Model parameters right Extracted encoding features.

[0039] Optionally, in step S4, the calculation function expression of the total loss formed by fusing the classification loss, the federated variational distillation loss and the gradient matching loss is:

[0040] ,

[0041] In the above formula, For the i Participating Centers The total loss, For the i Participating Centers The cross entropy loss is For the i Participating Centers The federated variational distillation loss, For the i Participating Centers Gradient matching loss.

[0042] Optionally, in step S4, the function expression for optimizing and updating the model parameters of the image classification model of each participating center by using the total loss is:

[0043] ,

[0044] In the above formula, To optimize the updated i Participating Centers The model parameters of the image classification model are as follows, " indicates optimized update operation. To optimize the update before i Participating Centers The model parameters of the image classification model are, is the learning rate, is the gradient of the model parameters, For thei Participating Centers The total loss, model parameters Including feature encoder , Classifier , No knowledge distillation module and gradient matching module The model parameters.

[0045] In addition, the present invention also provides a federated personalized learning system based on knowledge-free distillation and gradient matching, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the federated personalized learning method based on knowledge-free distillation and gradient matching.

[0046] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the federated personalized learning method based on knowledge-free distillation and gradient matching through a processor.

[0047] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the federated personalized learning method based on knowledge-free distillation and gradient matching through a processor.

[0048] Compared with the prior art, the present invention has the following advantages: the method of the present invention constructs a knowledge-free distillation module for each center, and uses the knowledge-free distillation method to extract domain-invariant information from the data of multiple participating centers, while ensuring the security of data information, and contributing available information to each center participating in the federation; a gradient matching module is constructed, and gradient information is used to constrain global information, extract domain-specific information for each center, and further make the optimization direction toward the local optimum. The knowledge-free distillation method can use the variational distribution properties of the encoding features to extract the domain-invariant information shared by all participating centers; the gradient matching loss can adjust the optimization direction of the model toward the domain-specific direction based on the matching degree of the gradient information, thereby realizing the personalization of the federated model in different centers, so that it can be optimized based on domain-specific information while aggregating global domain-invariant information, and finally achieve the best performance in a specific data distribution. Therefore, the present invention can effectively and safely utilize the information of each participating center to collaboratively train the federated model, and can further extract domain-specific information and constrain the update direction of the model parameters toward the personalized optimum according to the unique data distribution of each participating center, and achieve the goal of collaboratively training a personalized federated model for each participating center in a multi-center heterogeneous environment, and can significantly improve the performance of the deep learning model in brain image classification in each participating center. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0050] Figure 2 Schematic diagram of the principle of the method of the embodiment of the present invention.

[0051] Figure 3 Schematic diagram for comparing experimental results in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] Federated personalized learning involves multiple federated participating centers. The key to federated personalized learning is to achieve federated learning while maintaining the personalization of each participating center to obtain a personalized federated model that integrates global information. The present invention aims to make full use of the invariant information of the multi-center domain while adding constraints on domain-specific information, training a personalized federated deep model for each participating center to achieve optimal performance in the classification task of its local brain images or other images. The classification of brain structure magnetic resonance imaging data will be taken as an example to further describe the present invention in detail. In addition, the method of the present invention can also be applied to the classification of images, which will not be described in detail here.

[0053] like Figure 1 As shown, the federated personalized learning method based on knowledge-free distillation and gradient matching in this embodiment includes the following steps:

[0054] S1, construct an image classification model for each participating center respectively, including a feature encoder, a classifier, a knowledge-free distillation module and a gradient matching module, the feature encoder is used to extract encoding features for brain structure magnetic resonance imaging data, the classifier is used to classify the encoding features, the knowledge-free distillation module is used to extract common features of domain-invariant information in each center in the federation, and the gradient matching module is used to learn domain-specific information according to the data distribution unique to each center, and the feature encoder, classifier, knowledge-free distillation module and gradient matching module are all trainable embedding networks;

[0055] S2, randomly divide the private data of each participating center into training set and test set;

[0056] S3, use the private data of each participating center to train the image classification model and calculate the classification loss; for each participating center, use the model parameters of the local classifier as the input of the knowledge-free distillation module to build a covariance matrix proprietary to the participating center to capture the personalized distribution information of the participating center, reconstruct the global knowledge information of other participating centers in each participating center after secure sharing to calculate the federated variational distillation loss using mutual information, and use the gradient information of the model parameters of the local classifier as the input of the gradient matching module to calculate the gradient matching loss using the information reconstructed with other participating centers;

[0057] S4, the classification loss, federated variational distillation loss and gradient matching loss are integrated to form the total loss, and the total loss is used to optimize and update the model parameters of the image classification model of each participating center;

[0058] S5, if it is necessary to continue training, jump to step S2; otherwise, end the training of each participating center, save the model parameters to obtain a personalized federated model that integrates global information.

[0059] This embodiment is aimed at the classification task of multi-center brain structure magnetic resonance imaging data. Each center participating in the federated training has a local data set that cannot be transmitted and shared. Each subject in each data set has brain structure magnetic resonance imaging data and a given label. During the training process, the feature encoding image classification model needs to calculate the cross entropy classification loss based on the output of the classifier module, and use the gradient descent method to update the learnable parameters in the embedded network. Repeat the above training update strategy until the preset number of iterations is reached.

[0060] In step S1 of this embodiment, for any i-th participating center of the federation , for which the image classification model built includes a feature encoder , Classifier , No knowledge distillation module and gradient matching module . The feature encoder , Classifier Various known image feature encoders and classifiers can be used as needed. This embodiment does not rely on a specific feature encoder. , Classifier For example, as an optional implementation, the feature encoder in this embodiment It is a parallel network fused with CNN and Transformer, classifier Use a fully connected layer network without a knowledge distillation module Using graph neural network and gradient matching module A multi-layer perceptron network is used.

[0061] In this embodiment, the process of preprocessing brain structure magnetic resonance imaging data is also included before step S2. The preprocessing is completed using SPM8 software. In this embodiment, for each subject, the gray matter and white matter are separated, and only the three-dimensional gray matter signal of each subject is used for subsequent feature encoding and classification. Have a local brain structure magnetic resonance imaging dataset , each subject in the dataset has structural magnetic resonance imaging data and the corresponding labels . The method of this embodiment, under the premise of ensuring data security, collaboratively utilizes data information from multiple centers, extracts domain-invariant information between centers, and further extracts domain-specific information based on the data distribution characteristics of different centers, so that each center can always continuously optimize in the direction of local personalized optimization when obtaining federated global information, thereby training a federated personalized model with the best performance in local data distribution for each participating center. It should be noted that the data set mentioned in this embodiment will be divided into a training set and a test set. The training set is accessible during model training, and the test set is only accessible during model testing. Specifically, in this embodiment, the three-dimensional structural functional magnetic resonance imaging data and its classification labels of the subjects are obtained from medical institutions participating in model training and testing. The brain structure (structural magnetic resonance imaging, sMRI) data of the subjects' brains is obtained from medical institutions, which can be collected on-site using instruments or by inputting pre-collected magnetic resonance imaging data as needed. When collecting the brain structure and functional magnetic resonance imaging data of the subjects, the equipment used is a nuclear magnetic resonance scanner, and the scanning parameters are set according to the reasonable needs of each site. The brain structure MRI dataset in medical institutions needs to be divided into training set and test set. Each subject in the dataset is required to have structural MRI imaging data and the label corresponding to the subject. The size of the dataset depends on the number of subjects collected in the medical institution. The training set and test set are divided by cross-validation.

[0062] The classification loss in step S3 of this embodiment can adopt a desired loss function as needed. For example, as an optional implementation, the classification loss in step S3 of this embodiment is a cross entropy loss, and the calculation function expression is:

[0063] ,

[0064] In the above formula, For the i Participating Centers The cross entropy loss is is the expected value, For the i Participating Centers Magnetic resonance imaging data or images of brain structures, For the i Participating Centers brain structure magnetic resonance imaging data or image labels, is the cross entropy function, For the i Participating Centers The prediction results of the image classification model are:

[0065] ,

[0066] In the above formula, For the i Participating Centers The classifier, and are batch normalization layer and activation layer functions respectively, For the i Participating Centers The feature encoder Feature encoder based Model parameters right The extracted coding features, , They represent the length, width and height of the brain structure magnetic resonance imaging data or image respectively.

[0067] Based on the information theory approach, the domain-invariant knowledge in the federation is preserved by maximizing the variational information of feature distributions between different participating centers. First, the encoding features between different domains are defined as and , , the mutual information between them It is expressed as:

[0068] ,

[0069] ,

[0070] In the above formula, represents the entropy of the encoded feature, Representation based on joint distribution Conditional entropy under the definition. Mutual information It can represent the distribution difference between the encoded features learned by different participating centers. Therefore, by maximizing the mutual information between the data feature encodings of different participating centers by the same encoder, the common knowledge that is invariant in the middle domain can be extracted in the federation.

[0071] The key to calculating mutual information is to calculate the conditional entropy based on the joint distribution. By defining a variational distribution To fit the joint distribution The variational lower bound of can transform the calculation of mutual information into:

[0072] ,

[0073] Since the feature encoding entropy It is from the center The feature encoder Therefore, its distribution is relative to the model parameters can be considered as a constant. Therefore, mutual information can be equivalent to finding The lower bound of . At each center Construct a mean value , the covariance is The Gaussian distribution of To approximate the joint distribution. Therefore, the lower bound of the variational distribution can be determined by minimizing the following formula:

[0074] .

[0075] Since multiple centers participate in the federated collaborative training model, the center Leverage securely shared model parameters The calculated feature encoding is difficult to share directly. Therefore, the Taylor first-order expansion is used to approximate the fit and The relationship between:

[0076] ,

[0077] In the above formula, It is calculated by The gradient information obtained is is the dimension of the model parameters, is the number of label categories. Therefore, the center The material loss due to distillation can be defined as:

[0078] ,

[0079] In the above formula, “ " means that it is defined as, then:

[0080] ,

[0081] in, is the covariance of the fitted joint distribution, and All can be found in the center Calculated, the sharing of sensitive parameter information can be reduced, and the privacy security of private data can be protected to a greater extent. Represents an encryption function used to protect the information shared by each center, which is implemented using Gaussian noise: , Represents the information that needs to be shared, represents a mean of 0 and a variance of Gaussian noise. By controlling the variance of the superimposed Gaussian noise, the degree of contribution to information security can be adjusted. Since the covariance of the variational distribution Hard to be centered Directly obtain, therefore, by constructing a graph convolutional embedding network at each center to model a specific covariance Used to calculate the federated distillation loss. Because the covariance matrix can capture the relationship between different categories, a graph convolutional embedding network model with C nodes is defined It is used to model the covariance matrix between C labels. is a positive semidefinite matrix and can therefore be modeled as:

[0082] ,

[0083] in, Represents the d-dimensional latent space features of each node in the graph convolutional network. is a hyperparameter used to ensure reversibility, is a unit matrix of the same dimension. Specifically, an L-layer graph neural embedding network is constructed in each center as a variational distillation module. , whose parameters are Since the classifier The encoder's encoding features can be classified and predicted, so it can better correspond to the modeling of C category labels. The linear layer model parameters corresponding to the C categories in As Input Therefore, the function expression of the output of each layer of the graph convolutional network is:

[0084] ,

[0085] In the above formula, and Respectively and The output of the layer, is the activation function; is a degree matrix whose diagonal elements are given by The diagonal elements of ,matrix pass To calculate, is the identity matrix. Each institution models the covariance matrix of local features by constructing a specific variational distillation model And pass it to other centers for calculating the federated variational distillation loss, thereby extracting the invariant knowledge of the federated domain. The material distillation losses are calculated by other federal participating centers besides the local one:

[0086] ,

[0087] In the above formula, is the i-th participating center The material distillation loss, is the expected value, is the encryption function, is the covariance of the fitted joint distribution; is the i-th participating center The model parameters of the image classification model are, is the jth participating center The model parameters of the image classification model are, is the jth participating center Therefore, in step S3 of this embodiment, the model parameters of the local classifier are used as the input of the knowledge-free distillation module to construct a covariance matrix that is exclusive to the participating center to capture the personalized distribution information of the participating center. After secure sharing, the global knowledge information of other participating centers is reconstructed in each participating center to calculate the federated variational distillation loss using mutual information, including:

[0088] S3.1, for each participating center, the model parameters of the local classifier are used as the input of the knowledge-free distillation module to construct a covariance matrix specific to the participating center to capture the personalized distribution information of the participating center. The knowledge-free distillation module is a graph convolution embedding network model consisting of C graph convolutional networks. , graph convolutional embedding network model The model parameters are , each graph convolutional network consists of L layers, and the function expression of the output of each layer is:

[0089] ,

[0090] In the above formula, and Respectively and The output of the layer, is the activation function; is a degree matrix whose diagonal elements By matrix Elements Sure: ,matrix pass To calculate, is the identity matrix, is the adjacency matrix, the adjacency matrix According to the input of the first layer calculate , the input of the first layer For the classifier The linear layer model parameters corresponding to the C categories in ; and the covariance of the fitted joint distribution is calculated according to the output of C graph convolutional networks according to the following formula :

[0091] ,

[0092] In the above formula, The covariance of the fitted joint distribution is The inverse matrix of represents the latent space features extracted by C graph convolutional networks, ~ are the d-dimensional latent space features extracted by the 1st to Cth graph convolutional networks, is a hyperparameter, is a unit matrix of the same dimension; and the gradient information is calculated according to the following formula :

[0093] ,

[0094] In the above formula, Represents dimension information, is the dimension of the model parameters, is the number of label categories, is the model parameter The gradient of is the crossover function, is the i-th participating center The classifier, and are batch normalization layer and activation layer functions respectively, is the i-th participating center The feature encoder Feature encoder based Model parameters right The extracted coding features, is the jth participating center Magnetic resonance imaging data or images of brain structures, is the jth participating center The brain structure magnetic resonance imaging data or image labels of , gradient information , model parameters Using encryption functions After encryption, it is passed to other participating centers for calculating the federated variational distillation loss;

[0095] S3.2, each participating center calculates its own material distillation loss based on the data transmitted by other participating centers:

[0096] ,

[0097] In the above formula, is the i-th participating center The material distillation loss, is the expected value, is the encryption function, is the covariance of the fitted joint distribution; is the i-th participating center The model parameters of the image classification model are, is the jth participating center The model parameters of the image classification model are, is the jth participating center Gradient information of

[0098] S3.3, each participating center calculates the federated variational distillation loss according to the following formula:

[0099] ,

[0100] In the above formula, is the i-th participating center The federated variational distillation loss, is the number of participating centers, is the i-th participating center of material distillation losses.

[0101] Encryption Function Random noise can be added as needed. For example, as an optional implementation, the encryption function in this embodiment To use Gaussian noise, the function expression is:

[0102] ,

[0103] In the above formula, Indicates the encryption function Information that needs to be shared The encryption result is represents a mean of 0 and a variance of Gaussian noise. In addition, the encryption function Other ways of adding random noise can be used as desired.

[0104] Since the optimization direction of the personalized federated model should be the local optimum for each participating center, it is necessary to further extract domain-specific knowledge to constrain the optimization of the model. By introducing the gradient matching loss, we can further extract federated information that is homogeneous with the domain-specific information of each participating center. The core of minimizing the gradient matching loss is to maximize the matching degree between the gradients of different input information based on the same model parameters, thereby maximizing the similarity between the optimization direction and the local data distribution. Specifically, in each center Building a gradient matching module , is a learnable embedding network model, whose model parameters are . Participation Center The model parameters of the gradient matching module By using the encryption function introduced in the above steps After encryption, securely share to other centers , calculate the feature encoder Encoding features The resulting gradient . Then, Shared safely back to the center , through the computing center Gradient The similarity between the gradients of other centers is used as the gradient matching loss to find the center Constrain personalized parameter optimization direction. Specifically, the calculation function expression of the gradient matching loss in step S3 of this embodiment is:

[0105] ,

[0106] ,

[0107] ,

[0108] ,

[0109] In the above formula, is the i-th participating center Gradient matching loss, is the number of participating centers, is the i-th participating center Gradient and the jth participating center Gradient similarities between; represents the expected value, is the encryption function, represents the transpose operation, and are the i-th participating center and the jth participating center Magnetic resonance imaging data or images of brain structures, and are the i-th participating center and the jth participating center brain structure magnetic resonance imaging data or image labels, and are the i-th participating center and the jth participating center The training set, is the i-th participating center Gradient matching module Model parameters The gradient of is the cross entropy loss, is the i-th participating center The feature encoder Feature encoder based Model parameters right The extracted coding features, is the jth participating center The feature encoder Feature encoder based Model parameters right The extracted coding features. Therefore, in each participating center The gradient matching loss calculated by the gradient matching module based on other participating centers in the federation can be obtained, thereby constraining the parameter update direction to the local optimum.

[0110] like Figure 2 As shown, in step S4 of this embodiment, the calculation function expression of the total loss formed by integrating the classification loss, the federated variational distillation loss and the gradient matching loss is:

[0111] ,

[0112] In the above formula, is the i-th participating center The total loss, is the i-th participating center The cross entropy loss is is the i-th participating center The federated variational distillation loss, is the i-th participating center The gradient matching loss is calculated. By calculating the loss, the calculated cross entropy loss is used as the main loss for updating the model parameters. The obtained federated variational distillation loss and gradient matching loss are used as regular terms of the objective function to achieve the update and optimization of the embedded network model parameters.

[0113] In step S4 of this embodiment, the function expression for optimizing and updating the model parameters of the image classification model of each participating center by using the total loss is:

[0114] ,

[0115] In the above formula, To optimize the updated i-th participating center The model parameters of the image classification model are as follows, " indicates optimized update operation. To optimize the i-th participating center before the update The model parameters of the image classification model are, is the learning rate, is the gradient of the model parameters, is the i-th participating center The total loss, model parameters Including feature encoder , Classifier , No knowledge distillation module and gradient matching module The model parameters.

[0116] When determining whether to continue training in step S5, the required conditions can be selected as needed, such as the number of iterative training times being equal to a preset threshold, or a certain indicator (such as accuracy) of the image classification model being greater than or equal to a preset threshold.

[0117] In order to verify the effectiveness and performance of the federated personalized learning method based on knowledge-free distillation and gradient matching, the model construction, feature encoding, and training optimization of the personalized federated model in this embodiment are all implemented based on Pytorch. In this embodiment, the specific parameters of model training are set as follows: the sample batch size of the training set is set to 16, the learning rate of the model is set to 1e-4, and the number of iterations is set to 80. In step S103 of this embodiment, the gradient descent method is used to implement the model parameter update, which relies on the Pytorch framework. In this embodiment, the training of the image classification model is completed using labeled multimodal magnetic resonance imaging data to achieve parameter update, which requires repeated iterations. When the maximum number of iterations is reached, the training process will stop, and an image classification model suitable for multimodal magnetic resonance imaging data will be obtained. In this embodiment, the following method is used for verification: I. In this embodiment, the labeled brain structure magnetic resonance data collected from medical institutions are first randomly divided into training sets and test sets according to the 5-fold cross-validation method, and a personalized federated image classification model for multi-center brain images is constructed to extract domain-invariant information between federated multi-centers and extract domain-specific information according to the unique data distribution of each center, so that each center can obtain global knowledge in the federation while maintaining the local optimal optimization direction. By calculating the target loss of the model, the gradient descent method is used to update the model weights, and the above model training strategy is repeated until the preset number of iterations is reached, and finally a federated personalized model that can be used for brain image classification can be obtained in each participating center. According to the selection of multi-fold cross-validation, the above process is performed 5 times in total. This embodiment is trained based on brain image data sets collected by five medical centers (AMU, FMMU, Shanxi, Xiangya_1, Xiangya_2), where Xiangya_1 and Xiangya_2 are from the same medical center, but are brain image data sets collected by different instruments in different batches, so they are trained as two independent data sets in the comparative experiment. Each participating center has a data set consisting of brain structural MRI data of patients with depression and normal controls for experimental verification. Each subject in the data set has a brain structural MRI image and a label provided by an experienced doctor according to an internationally recognized scale. The size of the brain structural MRI data is 121×145×121, and the labels are 0 and 1. Label 0 is set for a normal person and 1 is set for a patient with depression. The experimental comparison verification method is adopted. First, the SOTA method (the most advanced research method at present) in federated deep learning in recent years is selected to compare the classification performance with the proposed method. All experimental result indicators are the average results of three repeated experiments. Subsequently, the effectiveness of the personalized federated deep learning model training optimization step is further verified by performing an ablation experiment on the domain-invariant information and domain-specific information in the proposed method.

[0118] The method proposed in this embodiment is tested on the multi-center depression dataset of REST-META-MDD with the SOTA (currently most advanced research method) method. The SOTA methods involved include: Centralized: a centralized training method that directly merges the data of all participating centers as a baseline comparison method; FEDAVG, see the literature: HB McMahan, E. Moore, D. Ramage, S. Hampson and BAY Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. International Conference on Artificial Intelligence and Statistics, 54, 1273–1282, (2017); FEDProx, see the literature: T. Li, et al. Federated Optimization inHeterogeneous Networks. In Proceedings of Machine Learning and Systems, 2,429-450, (2020); FEDMoon, see the literature: Q. Li, B. He, and D. Song. Model-contrastivefederated learning. IEEE Conference on Computer Vision and PatternRecognition (CVPR), 10713–10722, (2021); SCAFFOLD, see: SP Karimireddy, S. Kale, M. Mohri, SJ Reddi, SU Stich, and AT Suresh. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. International Conference on Machine Learning, 5132-5143, (2020); FEDALA, see: J. Zhang, et al. FedALA: Adaptive Local Aggregation for Personalized Federated Learning.Proceedings of the AAAI Conference on Artificial Intelligence, 37(9), 11237-11244, (2023). Finally, the accuracy (ACC) experimental results of the image classification model of the method in this embodiment and the SOTA method are shown in Table 1. .

[0119] Table 1: Comparison of the accuracy of the method in this embodiment and other existing SOTA methods

[0120]

[0121] As shown in Table 1, the accuracy of the federated deep model obtained by the method in this embodiment is better than other existing SOTA methods in terms of the accuracy of each participating center. In addition, other performance indicators, including SPE, SEN, AUC and F1 score, were also used for experiments. The results are as follows: Figure 3 As shown. Among them, SPE is the specificity (%) of the classification result, which indicates the ability to correctly classify non-patients; SEN is the sensitivity (%) of the classification result, which indicates the ability to correctly classify patients; AUC is the area under the receiver operating characteristic curve (ROC), which is usually used to measure the quality of the model; F1 score is an indicator used to measure the accuracy of the binary classification model, which can take into account both the accuracy and recall rate of the model. Figure 3 It can be seen that the SPE, SEN, AUC and F1 scores of the federated deep model obtained by the method of this embodiment on each participating center are superior to other existing SOTA methods.

[0122] In addition, an ablation experiment was designed for the method proposed in the embodiment, and the ablation experiment results of the accuracy (ACC) are shown in Table 2. The results shown in the figure represent the optimal model performance obtained by the gradient matching personalized federated deep learning method without complete knowledge distillation, which proves the effectiveness of the method proposed in this embodiment.

[0123] Table 2: Ablation experiment results

[0124]

[0125] As shown in Table 2, the federated variational distillation loss And gradient matching loss The addition of improves the accuracy (ACC) on multiple data sets, indicating that the federated personalized learning method based on knowledge-free distillation and gradient matching in this embodiment includes the federated variational distillation loss. And gradient matching loss The model performance obtained by the loss function is optimal, which proves that the federated variational distillation loss in the method proposed in this example And gradient matching loss effectiveness.

[0126] In summary, the federated personalized learning method based on knowledge-free distillation and gradient matching in this embodiment includes constructing a specific feature encoding module for each federated participating center, and constructing a knowledge-free distillation module and a gradient matching module at the same time, both of which are embedding networks with learnable parameters; the private data set of each participating center is divided into a training set and a test set, and the classification cross entropy loss of the feature encoder module is calculated based on the training set; then, the domain-invariant information of all participating centers in the federation is extracted using the knowledge-free distillation module, and the federated variational distillation loss is calculated; and the personalized domain-specific information of each participating center is maintained using the gradient matching module, and the gradient matching loss is calculated; the federated variational distillation loss and the gradient matching loss are used as the regularization term of the objective function, and the weighted summation with the cross entropy loss is performed as the final loss to realize the parameter update of the embedded network model; the above steps are repeated until the training is completed. The method of this embodiment can help medical imaging centers with limited private data sets and difficult to achieve data sharing aggregation to better train personalized models for tasks such as classification of medical imaging data, while fully learning to extract domain-invariant information in the federation, retaining the personalized domain-specific information of each center, and further improving the classification performance of the personalized model in each federated participating center. Compared with the existing methods, the method of this embodiment can effectively extract domain-invariant information from heterogeneous data, and can also further extract domain-specific information based on the data distribution characteristics of each center to constrain the optimization direction of the model. In this way, each center participating in the federation can obtain effective global information from the federation and retain local personalized information, so that each center participating in the federation can obtain the local optimal federated personalized model and improve the local brain image classification performance.

[0127] In addition, this embodiment also provides a federated personalized learning system based on knowledge-free distillation and gradient matching, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the federated personalized learning method based on knowledge-free distillation and gradient matching.

[0128] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the federated personalized learning method based on knowledge-free distillation and gradient matching through a processor.

[0129] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the federated personalized learning method based on knowledge-free distillation and gradient matching through a processor.

[0130] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A federated personalized learning method based on knowledge-free distillation and gradient matching, characterized in that: The steps include: S1, constructing an image classification model for each participating center respectively, including a feature encoder, a classifier, a knowledge-free distillation module and a gradient matching module, wherein the feature encoder is used to extract encoding features for brain structure magnetic resonance imaging data or images, the classifier is used to classify the encoding features, the knowledge-free distillation module is used to extract common features of domain-invariant information in each center in the federation, and the gradient matching module is used to learn domain-specific information according to the data distribution unique to each center, and the feature encoder, classifier, knowledge-free distillation module and gradient matching module are all trainable embedding networks; S2, randomly divide the private data of each participating center into training set and test set; S3, use the private data of each participating center to train the image classification model and calculate the classification loss; for each participating center, use the model parameters of the local classifier as the input of the knowledge-free distillation module to build a covariance matrix proprietary to the participating center to capture the personalized distribution information of the participating center, reconstruct the global knowledge information of other participating centers in each participating center after secure sharing to calculate the federated variational distillation loss using mutual information, and use the gradient information of the model parameters of the local classifier as the input of the gradient matching module to calculate the gradient matching loss using the information reconstructed with other participating centers; S4, the classification loss, federated variational distillation loss and gradient matching loss are integrated to form the total loss, and the total loss is used to optimize and update the model parameters of the image classification model of each participating center; S5, if it is necessary to continue training, jump to step S2; otherwise, end the training of each participating center, save the model parameters to obtain a personalized federated model that integrates global information.

2. The federated personalized learning method based on knowledge-free distillation and gradient matching according to claim 1, characterized in that: The classification loss in step S3 is the cross entropy loss, and the calculation function expression is: ， In the above formula, For the i Participating Centers The cross entropy loss is is the expected value, For the i Participating Centers Magnetic resonance imaging data or images of brain structures, For the i Participating Centers brain structure magnetic resonance imaging data or image labels, is the cross entropy function, For the i Participating Centers The prediction results of the image classification model are: ， In the above formula, For the i Participating Centers The classifier, and are batch normalization layer and activation layer functions respectively, For the i Participating Centers The feature encoder Feature encoder based Model parameters right The extracted coding features, , They represent the length, width and height of the brain structure magnetic resonance imaging data or image respectively.

3. The federated personalized learning method based on knowledge-free distillation and gradient matching according to claim 1, characterized in that: In step S3, the model parameters of the local classifier are used as the input of the knowledge-free distillation module to construct a covariance matrix specific to the participating center to capture the personalized distribution information of the participating center. After secure sharing, the global knowledge information of other participating centers is reconstructed in each participating center to calculate the federated variational distillation loss using mutual information, including: S3.1, for each participating center, the model parameters of the local classifier are used as the input of the knowledge-free distillation module to construct a covariance matrix specific to the participating center to capture the personalized distribution information of the participating center. The knowledge-free distillation module is a graph convolution embedding network model consisting of C graph convolutional networks. , graph convolutional embedding network model The model parameters are , each graph convolutional network consists of L layers, and the function expression of the output of each layer is: ， In the above formula, and Respectively and The output of the layer, is the activation function; is a degree matrix whose diagonal elements By matrix Elements Sure: ,matrix pass To calculate, is the identity matrix, is the adjacency matrix, the adjacency matrix According to the input of the first layer calculate , the input of the first layer For the classifier The linear layer model parameters corresponding to the C categories in ; and the covariance of the fitted joint distribution is calculated according to the output of C graph convolutional networks according to the following formula : ， In the above formula, The covariance of the fitted joint distribution is The inverse matrix of represents the latent space features extracted by C graph convolutional networks, ~ They are extracted by the 1st to Cth graph convolutional networks respectively. d dimensional latent space features, is a hyperparameter, is a unit matrix of the same dimension; and the gradient information is calculated according to the following formula : ， In the above formula, Represents dimension information, is the dimension of the model parameters, is the number of label categories, is the model parameter The gradient of is the crossover function, For the i Participating Centers The classifier, and are batch normalization layer and activation layer functions respectively, For the i Participating Centers The feature encoder Feature encoder based Model parameters right The extracted coding features, For the j Participating Centers Magnetic resonance imaging data or images of brain structures, For the j Participating Centers The brain structure magnetic resonance imaging data or image labels of , gradient information , model parameters Using encryption functions After encryption, it is passed to other participating centers for calculating the federated variational distillation loss; S3.2, each participating center calculates its own material distillation loss based on the data transmitted by other participating centers: ， In the above formula, For the i Participating Centers The material distillation loss, is the expected value, is the encryption function, is the covariance of the fitted joint distribution; For the i Participating Centers The model parameters of the image classification model are: For the j Participating Centers The model parameters of the image classification model are: For the j Participating Centers Gradient information of S3.3, each participating center calculates the federated variational distillation loss according to the following formula: ， In the above formula, For the i Participating Centers The federated variational distillation loss, is the number of participating centers, For the i Participating Centers of material distillation losses.

4. The federated personalized learning method based on knowledge-free distillation and gradient matching according to claim 3, characterized in that: The encryption function To use Gaussian noise, the function expression is: ， In the above formula, Indicates the encryption function Information that needs to be shared The encryption result is represents a mean of 0 and a variance of Gaussian noise.

5. The method for federated personalized learning based on knowledge-free distillation and gradient matching according to claim 4, characterized in that: The calculation function expression of the gradient matching loss in step S3 is: ， ， ， ， In the above formula, For the i Participating Centers Gradient matching loss, is the number of participating centers, For the i Participating Centers Gradient and j Participating Centers Gradient similarities between; represents the expected value, is the encryption function, represents the transpose operation, and Respectively i Participating Centers and j Participating Centers Magnetic resonance imaging data or images of brain structures, and Respectively i Participating Centers and j Participating Centers brain structure magnetic resonance imaging data or image labels, and Respectively i Participating Centers and j Participating Centers The training set, is the i-th participating center Gradient matching module Model parameters The gradient of is the cross entropy loss, For the i Participating Centers The feature encoder Feature encoder based Model parameters right The extracted coding features, For the j Participating Centers The feature encoder Feature encoder based Model parameters right Extracted encoding features.

6. The federated personalized learning method based on knowledge-free distillation and gradient matching according to claim 1, characterized in that: In step S4, the calculation function expression of the total loss formed by integrating the classification loss, federated variational distillation loss and gradient matching loss is: ， In the above formula, For the i Participating Centers The total loss, For the i Participating Centers The cross entropy loss is For the i Participating Centers The federated variational distillation loss, For the i Participating Centers Gradient matching loss.

7. The federated personalized learning method based on knowledge-free distillation and gradient matching according to claim 1, characterized in that: In step S4, the function expression for optimizing and updating the model parameters of the image classification model of each participating center using the total loss is: ， In the above formula, To optimize the updated i Participating Centers Model parameters of the image classification model, " " indicates optimized update operation. To optimize the update before i Participating Centers The model parameters of the image classification model are, is the learning rate, is the gradient of the model parameters, For the i Participating Centers The total loss, model parameters Including feature encoder , Classifier , No knowledge distillation module and gradient matching module The model parameters.

8. A federated personalized learning system based on knowledge-free distillation and gradient matching, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the federated personalized learning method based on knowledge-free distillation and gradient matching as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the federated personalized learning method based on knowledge-free distillation and gradient matching as described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the federated personalized learning method based on knowledge-free distillation and gradient matching as described in any one of claims 1 to 7 through a processor.

Images