Multi-center image recognition method and medium based on data-free knowledge distillation federated learning

Abstract

The application discloses a multi-center image recognition method and medium based on data-free knowledge distillation federated learning, each imaging center corresponds to a client, and all the clients share a server end; each client utilizes a trained local autism recognition model to output an autism recognition result according to a feature vector corresponding to an image of the local imaging center; wherein the learning and training method of the local autism recognition model of each client is that, on one hand, the local autism recognition model is learned and trained by using a local image database, and on the other hand, the local autism recognition model is learned and trained by using a data-free knowledge distillation method, autism classification knowledge is learned from other clients through the shared server end, and is migrated to the local autism recognition model, while the individual knowledge learned by using the local image database in the local autism recognition model is reserved. The application can eliminate the dependence of the federated distillation method on a real data set, and effectively improve the accuracy of multi-center autism classification.

Description

Technical Field

[0001] This invention belongs to the field of intelligent healthcare, specifically relating to a multi-center image recognition method and medium based on data-free knowledge distillation federated learning. Background Technology

[0002] Autism is a common and complex neurodevelopmental disorder that accompanies individuals throughout their lives. It typically manifests before the age of three in children. Clinically, autism mainly presents with the following symptoms: impairments in social communication or interaction, verbal or nonverbal communication difficulties, restricted interests or repetitive and stereotyped behaviors, mood swings, and motor impairments. Autism causes significant harm to the physical and mental well-being of patients, and also places considerable emotional and financial burdens on their parents. Early identification of autism is a crucial step in its rehabilitation and treatment. Currently, because there is no clear etiology or pathological mechanism, diagnosis of autism primarily relies on doctors assessing the patient's behavioral habits—a time-consuming and highly subjective process.

[0003] With the rapid development of neuroimaging, many studies have utilized machine learning methods to analyze magnetic resonance imaging data and propose more efficient and objective methods for autism identification. However, these methods face significant limitations due to growing concerns about privacy and security. Therefore, it is essential to develop a reliable, objective method for autism identification that also protects basic data privacy.

[0004] Combining machine learning methods with autism identification can address the inefficiencies and biases in clinical diagnosis. However, traditional machine learning methods require the centralized collection of large amounts of data for model training, which may involve data collection from multiple data centers. Furthermore, the collection and sharing of medical data between different hospitals raises privacy concerns, and training deep learning models using large-scale, high-quality medical data becomes increasingly difficult under strict user privacy policies. Traditional machine learning methods face significant limitations under these stringent data privacy and security requirements. Some studies have proposed combining federated learning with knowledge distillation, but these methods often require a distillation dataset, which is typically difficult to obtain. Therefore, this invention urgently seeks to develop a high-performance autism identification method that does not require the centralized collection of multi-center autism data. Summary of the Invention

[0005] This invention provides a multi-center image recognition method and medium based on data-free knowledge distillation federated learning, which solves the problem of distillation dataset dependence in the knowledge distillation process, strengthens the privacy protection of client data, and achieves better classification performance of each imaging center in scenarios where centralized data collection is not required.

[0006] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0007] A multi-center image recognition method based on data-free knowledge distillation federated learning is proposed. Each imaging center corresponds to a client, and all clients share a single server. Each client uses a pre-trained local autism recognition model to output autism recognition results based on the feature vectors corresponding to the local imaging center images. The training method for the pre-trained local autism recognition model of each client is as follows: on the one hand, the autism recognition model is trained using a local image database; on the other hand, autism classification knowledge is learned from other clients through data-free knowledge distillation via the shared server and transferred to the local autism recognition model, while retaining the individual knowledge learned from the local image database in the local autism recognition model.

[0008] Furthermore, the image at the imaging center refers to the feature vector of functional magnetic resonance imaging (fMRI) data, which is input into the autism recognition model. The method for extracting the feature vector from the fMRI data is as follows:

[0009] Different brain regions were extracted based on the AAL map. Pearson correlation coefficient between ;

[0010] Then, the Pearson correlation coefficient was analyzed. Perform Fisher transform to obtain the enhanced correlation coefficient. :

[0011]

[0012] The matrix composed of enhancement correlation coefficients from all different brain regions is used to extract the upper triangular matrix and concatenate them into a vector, which is the extracted feature vector. .

[0013] Furthermore, the method of using data-free knowledge distillation involves learning autism classification knowledge from other clients through a shared server and transferring it to the local autism recognition model, while retaining the individual knowledge learned from the local image database in the local autism recognition model. Specifically:

[0014] Step 1: Each client uploads the parameters and labels of the local autism recognition model that has been trained in the current iteration to the server.

[0015] Step 2: The server trains a generator model based on the distribution information obtained from the label statistics uploaded by all clients, and uses the generator model to generate a feature vector dataset, which is called the augmented dataset. Then, based on the recognition model parameters of each client, autism recognition is performed on the augmented dataset, and the average logistic value is calculated for all recognition results.

[0016] Step 3: Each client downloads the average logical value and generator model from the server.

[0017] Step 4: On the augmented dataset, each client performs knowledge distillation based on the average logistic value obtained from the server and the logistic value output by the local model, and optimizes the parameters of the local autism recognition model based on the distillation loss.

[0018] Step 5: Each client uses the feature vector set corresponding to its local image database to further optimize the autism recognition model parameters obtained in Step 4, and then uploads the model parameters to the server.

[0019] Step 6: The server performs a weighted average of all model parameters uploaded by clients and updates the parameters in the autism recognition model on the server side, except for the normalization layer.

[0020] Step 7: Each client downloads the autism recognition model from the server to its local machine and updates the parameters in the local autism recognition model except for the normalization layer.

[0021] Step 8: Update the parameters of the server-side model.

[0022] Furthermore, the server-side optimization objective function for training the generator model based on the distribution information obtained from the label statistics uploaded by all clients is as follows:

[0023]

[0024] in, Represents a generator model. These are the parameters of the generator model. The augmented representation of the feature vector in the latent feature space is denoted as the augmented data sample; J represents the target label of the generator model, and J is the optimization objective of the generator model. Indicates target label The prior distribution; Indicates data sample labels obey distributed, Indicates augmented data samples obey distributed; This indicates that the generator model outputs augmented data samples. The loss; It is the output of the layer preceding the activation function layer of the recognition model. It is an activation function; This represents the parameters of the k-th client-side autism identification model; for any target label The objective function uses the label statistics uploaded by the client and the predictor model parameters of the k-th client model. The calculation is completed by gradient descent on the augmented data samples; K represents the number of imaging centers, i.e., the number of clients or client autism recognition models, and k is the index of the client.

[0025] Furthermore, the optimization objective for the distillation loss during knowledge distillation by each client in step 4 is:

[0026]

[0027] in, It is an activation function. It is the average logical value calculated by the server. It is a feature extractor for each client. Model parameters, It is an augmented dataset generated by a generator model. It is an augmented dataset Data in the middle; These are the client predictors. The model parameters map the latent feature space to the label space; This indicates the calculation of KL divergence.

[0028] Furthermore, the server-side method for weighting the model parameters in step 6 is as follows:

[0029]

[0030] in, It's a global round. For the first Model parameters obtained by round weighted average It is the first In the global iteration of the client Model parameters, It is the number of clients. It is a client The number of samples, This represents the total number of samples across all clients.

[0031] Furthermore, the parameter update method for the client model is as follows:

[0032]

[0033] in, It is an augmented representation of the feature space on the latent space. It is a loss function. It is the output of the prediction model; It can be expressed by the following formula:

[0034]

[0035] in, It is a client The feature vector set corresponding to the local image dataset. express The samples in; This represents the mapping relationship between sample data and labels; the specific loss function. A binary classification cross-entropy loss function can be used, and the calculation formula is as follows:

[0036]

[0037] in For the sample The true label; For the sample Predicted labels.

[0038] Furthermore, the autism identification models on both the client and server sides all employ the same neural network model structure, including parameters. Determined feature extractor and parameters A defined predictor; the feature extractor maps the input feature vectors to a latent feature space, and the predictor predicts the category based on the feature vectors in the latent feature space.

[0039] Furthermore, the neural network model structure used in the autism identification model includes an input layer, a normalization layer, a random dropout layer, and an activation function layer.

[0040] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0041] Beneficial effects

[0042] 1. This invention introduces data-free knowledge distillation to learn the global distribution information of client data while retaining the normalized layer parameters of the client prediction model, thus enabling the client to have a personalized layer. This improves the model performance of federated learning technology in multi-center autism identification problems;

[0043] 2. By utilizing augmented samples generated by a generator trained on the server side, the problem of dataset dependency in federated distillation during multi-center autism identification is solved, expanding the applicability of the method and further strengthening the privacy protection of client data. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the method described in an embodiment of the present invention;

[0045] Figure 2 This is a method framework diagram of the method described in the embodiments of the present invention. Detailed Implementation

[0046] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.

[0047] like Figure 1 The diagram shown is a flowchart of the multi-center image recognition method based on data-free knowledge distillation federated learning in this embodiment, which includes the following steps:

[0048] S1. Functional magnetic resonance imaging (fMRI) data from various sites were obtained from existing multi-site autism datasets, and clients were segmented according to site data. The dataset used was specifically CPAC-preprocessed time-series fMRI data provided by the ABIDE database. CPAC preprocessing is a commonly used preprocessing method, including data format conversion, temporal correction, head motion correction, spatial normalization, and filtering. Four imaging centers (NYU, USM, UM, and UCLA) with more than 60 samples were selected from the ABIDE database. Because some time series data were missing in the original data, these data were removed to ensure experimental quality and to better conduct the experiment. Alternatively, missing data could be imputed, for example, by using the average time series value of the brain region as the imputation value. A simpler method was used here. Therefore, this embodiment contains a total of 439 samples, including 206 autistic patients and 243 normal individuals. The data from four selected imaging centers (NYU, USM, UM, and UCLA) were divided into four clients, with 160, 132, 67, and 90 data samples respectively. The time series lengths of the samples in each client were 176, 236, 296, and 116 respectively.

[0049] S2, each client preprocesses the site data obtained in step S1, extracting Pearson correlation coefficients for different brain regions using AAL maps. Specifically, this includes the following steps:

[0050] Each client calculates the Pearson correlation coefficients for different brain regions using AAL brain mapping based on local time-series data, obtaining the functional connectivity matrix. The formula for calculating the Pearson correlation coefficient is:

[0051]

[0052] and They represent the first The brain regions and the first Time series of brain regions It represents the mathematical expectation. The value ranges from -1 to 1. The closer the value is to 1, the more correlated the time series is; the closer it is to -1, the less correlated the time series is.

[0053] B. Enhance correlation through Fisher transform; the calculation formula is as follows:

[0054]

[0055] C. Extract the upper triangular matrix from the matrix obtained in step B, and concatenate these triangular matrices into a vector to obtain the features used for classification. . The number of features is determined by Confirmed, among which The region of interest (ROI) is the region of interest. The AAL map has 116 ROIs, so the feature dimension is 6670.

[0056] S3, each client divides the dataset obtained in step S2 into training and test sets according to a set ratio; specifically, the training and test sets are divided in an 8:2 ratio, and a five-fold cross-validation experiment is conducted.

[0057] S4: Each client uses the data from S3 and its local neural network to train its own model parameters, which are then uploaded to the server. Simultaneously, the label statistics for each client's data are also uploaded to the server. Specifically, this includes the following steps:

[0058] A. Each client uses local data and the same neural network model for local training. After training for E iterations, the client model parameters are... Uploaded to the server, the neural network model consists of two layers. The first layer has 64 neurons, followed by a normalization layer and a dropout layer to prevent overfitting. The last layer is a softmax layer. The parameters of this classification model are... It consists of two parts, the first part being the feature extractor. Its parameters are determined by The first step is to map the input features to the latent feature space. The second part is the predictor. Its parameters are determined by Sure;

[0059] B. Statistical tag information for each client This data is then uploaded to a server for the generator to learn the overall distribution of the client data. This distribution can be described by the following formula:

[0060]

[0061] in and These are the true prior and posterior distributions of the target labels.

[0062] S5: The server trains a generator model based on the label statistics uploaded by all clients, and uses the generator model to generate a feature vector dataset. Then, based on the recognition model parameters of each client, it performs autism recognition on the generated feature vector dataset and calculates the average logical value of all recognition results.

[0063] In the generator, the client model is placed in the generator. The generated augmented dataset The logical values ​​output are averaged to obtain This guides the client-side model training. The calculation formula is:

[0064]

[0065] in It is the number of clients. It is a client-side local model, and its output is the output of the prediction layer. Knowledge sharing between models is achieved through ensemble distillation, with each participant... By augmenting the dataset Classification scores calculated above This represents the knowledge learned by the client-side model. These classification scores are uploaded to the server, where the server calculates the average of this knowledge. , This can be viewed as the consensus knowledge learned by these client models, which can guide the learning of the client models.

[0066] S6, each client downloads the average logical value and generator model obtained from the server in step S5 to its local machine.

[0067] S7, each client performs knowledge distillation on the dataset generated by the generator using the average logistic value and the logistic value output by the local model, and calculates the distillation loss; specifically, the optimization objective is as follows:

[0068]

[0069] in, It is an activation function. , These are client-side model parameters. It is a feature extractor The model parameters map the input features to the latent feature space. It is a predictor The model parameters map the latent feature space to the label space. Calculate the KL divergence.

[0070] S8. Each client fine-tunes the model using the data obtained in step S3, and then uploads the model parameters to the server. Specifically, each client trains the model locally for 5-10 epochs on the data obtained in S3 before uploading the model parameters.

[0071] S9, the server performs a weighted average of all model parameters uploaded by clients. In this step, only the parameters other than the normalized network layer are updated; specifically, the model parameters of the network layers other than the normalized layer are obtained by the following formula during the weighted average:

[0072]

[0073] in, It's a global round. For the first The weighted average model parameters obtained in turn, It is the first In the global iteration of the client Model weights, It is the number of clients. It is the number of client-side samples. This is the total number of samples from all clients. (The sentence is incomplete and requires more context to translate accurately.) The model parameters of each client are weighted and averaged to obtain the average weight. The weight is the ratio of the number of samples of the current client to the total number of samples.

[0074] S10, each client downloads the weighted average model obtained in step S9 to its local machine and replaces the network layer model parameters other than the normalization layer in the local model with the weighted average model.

[0075] S11, Update server-side model parameters.

[0076] The client-side model parameter update method specifically optimizes the following formula:

[0077]

[0078] in, It is an augmented representation of the feature space on the latent space. It is a user-defined loss function. It is the output of the prediction model. It can be expressed by the following formula:

[0079]

[0080] in It is a client Private dataset, This indicates the mapping relationship between sample data and labels. Indicates client Private dataset The samples in the dataset. The specific loss function can be the binary classification cross-entropy loss function, calculated as follows:

[0081]

[0082] in For each sample, For the sample The true label; For the sample Predicted labels

[0083] S12, each client uses the prediction model obtained in step S10 to classify and predict autism. The evaluation metric is accuracy. The calculation formula is:

[0084]

[0085] TN and TP represent the number of samples correctly predicted as normal individuals and the number of samples correctly predicted as autistic individuals, respectively. FN and FP represent the number of autistic individuals who were incorrectly predicted as normal individuals and the number of normal individuals who were incorrectly predicted as autistic individuals, respectively. A higher value for this evaluation index indicates better performance.

[0086] The method of the present invention will be further described below with reference to an embodiment:

[0087] This method is based on the PyTorch deep learning framework, and model training is accelerated using an NVIDIA RTX 2080Ti graphics card. The final hyperparameter values ​​selected for the model are shown in Table 1.

[0088]

[0089] The invention was validated on the ABIDE multicenter autism dataset, with accuracy as the evaluation metric; a higher accuracy indicates better method performance. To evaluate the effectiveness of the invention, it was compared with the following methods:

[0090] JDA: It uses a transformation matrix to make the conditional and marginal distributions of the transformed source and target domain data more similar. After learning the new feature representation, it uses the K-nearest neighbor algorithm for prediction.

[0091] TCA: Maps data from the source and target domains to a reproducible high-dimensional Hilbert space, minimizing the data distance between the source and target domains. After learning the feature representations of the source and target domains, prediction is performed using the K-nearest neighbor algorithm.

[0092] maLRR (maLRR) learns a common low-rank representation for data from multiple centers to reduce the differences in data distribution across different centers. One site is designated as the target domain, and the remaining sites as source domains. Data from these domains is mapped to a common low-rank representation space, ensuring that each sample from the source domain can be linearly represented by samples from the target domain. A K-nearest neighbor classifier is then trained using this new data representation from multiple centers for prediction.

[0093] FedAvg is a classic federated learning algorithm whose main purpose is to obtain a better-performing global model by weighting the client model parameters according to the number of data samples for each client.

[0094] FedBN: An improved algorithm based on the FedAvg algorithm, which keeps the normalization layer parameters of all client models locally and does not participate in global model updates. This method is effective when there are feature shifts in the client data.

[0095] FedDistill is also an improved algorithm based on the FedAvg algorithm, which improves the communication efficiency of the FedAvg algorithm by replacing the uploaded model parameters in the FedAvg algorithm with uploaded logical values.

[0096] The present invention and the comparative method were evaluated on the same test set using a five-fold crossover experiment. The experimental results are shown in Table 2.

[0097]

[0098] As can be seen from Table 2, compared with the three domain adaptive methods and the three federated learning methods, the method proposed in this invention has significant advantages, indicating that the federated learning image recognition method without data knowledge distillation proposed in this invention is effective for autism diagnosis.

[0099] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.

Claims

1. A multi-center image recognition method based on data-free knowledge distillation federated learning, characterized in that, Each imaging center corresponds to one client, and all clients share one server. Each client uses a pre-trained local autism recognition model to output autism recognition results based on the feature vectors corresponding to the images in the local imaging center. The training method for the local autism recognition model trained on each client is as follows: on the one hand, the autism recognition model is trained using the local image database; on the other hand, the autism classification knowledge is learned from other clients through a shared server using a data-free knowledge distillation method, and then transferred to the local autism recognition model, while retaining the individual knowledge learned from the local image database in the local autism recognition model. The method of using data-free knowledge distillation involves learning autism classification knowledge from other clients through a shared server, transferring it to the local autism recognition model, and retaining the individual knowledge learned from the local image database within the local autism recognition model. Specifically: Step 1: Each client uploads the parameters and labels of the local autism recognition model that has been trained in the current iteration to the server. Step 2: The server trains a generator model based on the distribution information obtained from the label statistics uploaded by all clients, and uses the generator model to generate a feature vector dataset, which is called the augmented dataset. Then, based on the recognition model parameters of each client, autism recognition is performed on the augmented dataset, and the average logistic value is calculated for all recognition results. Step 3: Each client downloads the average logical value and generator model from the server. Step 4: On the augmented dataset, each client performs knowledge distillation based on the average logistic value obtained from the server and the logistic value output by the local model, and optimizes the parameters of the local autism recognition model based on the distillation loss. Step 5: Each client uses the feature vector set corresponding to its local image database to further optimize the autism recognition model parameters obtained in Step 4, and then uploads the model parameters to the server. Step 6: The server performs a weighted average of all model parameters uploaded by clients and updates the parameters in the autism recognition model on the server side, except for the normalization layer. Step 7: Each client downloads the autism recognition model from the server to its local machine and updates the parameters in the local autism recognition model except for the normalization layer. Step 8: Update the parameters of the server-side model.

2. The multi-center image recognition method according to claim 1, characterized in that, The image at the imaging center refers to functional magnetic resonance imaging (fMRI) data. The feature vectors input into the autism recognition model are extracted from the fMRI data using the following method: Different brain regions were extracted based on the AAL map. Pearson correlation coefficient between ; Then, the Pearson correlation coefficient was analyzed. Perform Fisher transform to obtain the enhanced correlation coefficient. : ； The matrix composed of enhancement correlation coefficients from all different brain regions is used to extract the upper triangular matrix and concatenate them into a vector, which is the extracted feature vector. .

3. The multi-center image recognition method according to claim 1, characterized in that, The server-side optimization objective function for training the generator model based on the distribution information obtained from the statistical data of tags uploaded by all clients is as follows: ； in, Represents a generator model. These are the parameters of the generator model. The augmented representation of the feature vector in the latent feature space is denoted as the augmented data sample; This represents the target label of the generator model. ; Indicates target label The prior distribution; Indicates data sample labels obey distributed, Indicates augmented data samples obey distributed; This indicates that the generator model outputs augmented data samples. The loss; It is the output of the layer preceding the activation function layer of the recognition model. It is an activation function; This represents the parameters of the k-th client-side autism identification model; for any target label The objective function uses the label statistics uploaded by the client and the predictor model parameters of the k-th client model. The calculation is completed by gradient descent on the augmented data samples; K represents the number of imaging centers, i.e., the number of clients or client autism recognition models, and k is the index of the client.

4. The multi-center image recognition method according to claim 1, characterized in that, The optimization objective for the distillation loss during knowledge distillation by each client in step 4 is: ； in, It is an activation function. It is the average logical value calculated by the server. It is a feature extractor for each client. Model parameters, It is an augmented dataset generated by a generator model. It is an augmented dataset Data in the middle; These are the client predictors. The model parameters map the latent feature space to the label space; This indicates the calculation of KL divergence.

5. The multi-center image recognition method according to claim 1, characterized in that, In step 6, the server-side method for averaging and weighting the model parameters is as follows: ； in, It's a global round. For the first Model parameters obtained by round weighted average It is the first In the global iteration of the client Model parameters, It is the number of clients. It is a client The number of samples, This represents the total number of samples across all clients.

6. The multi-center self-image recognition method according to claim 3, characterized in that, The parameter update method for the client model is as follows: ； in, It is an augmented representation of the feature space on the latent space. It is a loss function. It is the output of the prediction model; It can be expressed by the following formula: ； in, It is a client The feature vector set corresponding to the local image dataset. express The samples in; This represents the mapping relationship between sample data and labels; the specific loss function. The binary classification cross-entropy loss function is used, and the calculation formula is as follows: ； in For the sample The true label; For the sample Predicted labels.

7. The multi-center image recognition method according to claim 1, characterized in that, All client-side and server-side autism identification models use the same neural network model structure, including parameters. Determined feature extractor and parameters A defined predictor; the feature extractor maps the input feature vectors to a latent feature space, and the predictor predicts the category based on the feature vectors in the latent feature space.

8. The multi-center image recognition method according to claim 7, characterized in that, The neural network model used in the autism identification model includes an input layer, a normalization layer, a random dropout layer, and an activation function layer.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images