A Medical Image Classification Method Based on Knowledge Distillation Federated Learning under Neural Architecture Search
By employing neural architecture search and knowledge distillation, this study addresses the aggregation problem of heterogeneous models in federated learning, reduces communication and computational costs, improves image classification accuracy, enhances data privacy protection, and alleviates system heterogeneity.
Patent Information
- Application Number
- CN202410101924.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2024-01-23
- Filing Date
- 2024-01-25
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-01-25
AI Technical Summary
In federated learning, the aggregation of heterogeneous models results in high communication and computational costs, high privacy risks, and unresolved model heterogeneity issues, leading to a decline in image classification performance and accuracy.
By employing neural architecture search and knowledge distillation methods, and through model aggregation of ensemble distillation and local distillation, we reduce communication and computation costs, share only the model output layer parameters, enhance privacy protection, and alleviate system heterogeneity.
It achieves model aggregation with low communication and computational costs, improves the performance and accuracy of image classification, enhances data privacy protection, and prevents gradient depth leakage.
Smart Images

Figure CN118279639B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and specifically relates to a medical image classification method that can be used to classify and detect medical images of different diseases in medical institutions. Background Technology
[0002] With increasing concerns about user data privacy, federated learning has become a hot research topic. Especially in the medical field, medical image data is highly sensitive to privacy because it may contain detailed personal information about patients. Due to relevant privacy policies and laws and regulations, its circulation is strictly limited, making it difficult to aggregate large amounts of data from various medical institutions to leverage big data advantages and train a deep learning model. Therefore, federated learning has been proposed to fully utilize this privacy-sensitive data. However, in traditional federated learning algorithms, all clients' local model architectures must be consistent for model aggregation on a central server. This ignores the system heterogeneity of participating clients, i.e., the computing power and bandwidth of each client's local device vary, and frequent model transfers generate high communication costs. Furthermore, in practical applications of federated learning, each participant may want to use its own unique model architecture. For example, when multiple medical institutions collaborate without sharing local data, they may need to carefully design their own models to meet different specifications, and for privacy and intellectual property considerations, they may be unwilling to share detailed information about their local models. How to perform federated learning when each participant's local model has a different architecture and is an invisible black box to any other client and central server is the core problem that this image classification aims to solve.
[0003] In his 2017 paper "Communication-efficient learning of deep networks from decentralized data," McMahan B proposed a method for updating model parameters in training global network models for image classification, namely the classic FedAvg method. This method directly weights and averages model parameters from all clients. Due to parameter sharing, data privacy is difficult to protect, and gradient depth leakage is easily observed. Furthermore, it cannot address the issue of client-side model heterogeneity during training, requiring that local client models be homogeneous; otherwise, model aggregation cannot be performed.
[0004] In 2019, Daliang Li proposed the FedMD method in the paper "Heterogeneous Federated Learning via Model Distillation". It uses a translator to translate heterogeneous models of image classification into a standard structure, and then performs distillation learning. Although this method solves the problem of heterogeneous model training, the amount of communication and computation is huge, and the translator needs to be optimized and improved. Otherwise, the performance and accuracy of image classification will decrease due to the loss of image information and model bias. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of the prior art by proposing a medical image classification method based on knowledge distillation and personalized federated learning under neural architecture search. This method aims to aggregate heterogeneous models from different clients, reduce the communication and computational load between the client and the server, avoid gradient depth leakage, ensure data privacy and security, and improve the performance and accuracy of image classification.
[0006] The technical approach to achieving the objectives of this invention is to aggregate heterogeneous models from various clients using an integrated distillation-based model aggregation method. Model updates via neural architecture search and local distillation significantly reduce communication and computational loads, improving model performance and accuracy. By sharing only the model's output layer, data privacy is enhanced, preventing privacy attacks against federated learning, such as gradient depth leakage.
[0007] Based on the above ideas, the technical solution of the present invention includes the following steps:
[0008] (1) Obtain image classification datasets and network models from N clients:
[0009] (1a) Obtain relevant disease datasets from N medical institutions or public datasets, and process them according to the degree of non-independent and identically distributed distribution based on the Dirichlet distribution to obtain image classification datasets for N clients, where N>=2;
[0010] (1b) Extract images of the same number of samples from various medical institutions’ datasets or public datasets to form a public dataset on the server side;
[0011] (1c) Divide the above image classification dataset and public dataset into training data and test data in a ratio of 8:2, 7:3 or 9:1 respectively;
[0012] (1d) Select network models from the existing network models as the teacher network T on the server side and the student network S on the client side;
[0013] (2) By using the neural network architecture search method, select the optimal model structure S1 for each client student network S structure;
[0014] (3) Model aggregation for server-side integrated distillation using a weighted average of model parameters:
[0015] (3a) Pre-train the server-side teacher network model T and the optimal student network S1 model of each client to obtain the pre-trained teacher network T1 model and student network S2 model. Then, perform secondary training on the pre-trained student network S2 to obtain the secondary trained student network S3 model.
[0016] (3b) The output layer of the S3 model of each client student network after the second training is weighted and averaged to obtain the output layer Θ(x) of the model;
[0017] (3c) Input the test data from the public dataset into the server-side teacher network T1 model for prediction, and obtain the prediction result T-RESULT for image classification;
[0018] (3d) Utilize the output layer Θ(x) of the client-side model and the output layer θ of the server-side teacher network T1 model. g The prediction result T-RESULT and the true values of the test data in the public dataset are used to perform ensemble distillation on the pre-trained server-side teacher network model T1 to obtain the ensemble-distilled server-side model T2.
[0019] (4) Use the integrated distilled server-side teacher network model T2 to guide the client in updating the local distilled model:
[0020] (4a) Each client inputs the test data of its local image classification dataset into the server's T2 model for testing, and saves the output layer parameters θ after each test. g ;
[0021] (4b) The client inputs the test data from its respective image classification dataset into the corresponding client-trained student network model S3 for testing, and obtains the prediction results S-RESULT for its respective image classification.
[0022] (4c) Utilizing the output layer parameters θ of the integrated distillation post-server teacher network T2 model g The output layer θ of the student network S3 model trained on the client side. k The prediction result S-RESULT and the true values of the test data in each image classification dataset are used to perform local distillation on the student model S3 trained on each client to obtain the locally distilled client student network model S4.
[0023] (5) Repeat steps (3) to (4) until the teacher network T2 model trained on the server converges and completes the local distillation of the model update for each client, and each client obtains the final student network S4 model.
[0024] (6) The client inputs the test data from its respective image classification dataset into the final student network S4 model to obtain the classification results.
[0025] Compared with the prior art, the present invention has the following advantages:
[0026] 1. Good data privacy protection.
[0027] Existing federated learning models require sharing the gradient depth information of participants, which is crucial for reconstructing pixel-level accurate images and tag-level matching text. This insecurity can lead to data privacy breaches. In contrast, this invention only shares the client's output layer parameters, eliminating the need to upload the local model and thus ensuring data privacy and security.
[0028] 2. Significantly reduces communication and computational load.
[0029] This invention allows each client student model to learn by uploading only a vector result of its local model length equal to the dataset category C, significantly reducing the communication burden compared to directly uploading the local model. Furthermore, during local distillation, each client trains at most one local model, with the server-side teacher network playing only an auxiliary role, thus greatly reducing the computational burden.
[0030] 3. Alleviate system heterogeneity and improve the performance and accuracy of local models.
[0031] The local model is selected by each client based on its own task requirements and neural architecture network search method, choosing the client network model with the optimal structure. This helps alleviate the system heterogeneity problem in federated learning. Furthermore, since the local distillation in this invention is performed on the client's local private dataset, it can train a personalized model that is more adaptable to non-independent and identically distributed local data, thus improving model accuracy. Attached Figure Description
[0032] Figure 1 This is a flowchart illustrating the implementation of the present invention.
[0033] Figure 2 This is a classification image of the PathMNIST colon pathology dataset obtained in this invention;
[0034] Figure 3 This is a visualization matrix of the local data distribution of the first 10 clients in this invention;
[0035] Figure 4This is a schematic diagram simulating binary single-point crossover and bit-flip mutation in this invention. Detailed Implementation
[0036] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should all fall within the protection scope of the present invention.
[0037] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.
[0038] Reference Figure 1 The implementation steps of this embodiment include the following:
[0039] Step 1: Obtain the colon pathology dataset.
[0040] Colon pathology datasets can be obtained from multiple medical institutions or from publicly available datasets.
[0041] like Figure 2 As shown, this embodiment uses the PathMNIST colonic pathology dataset obtained from the MedMNIST V2 public dataset, which contains 107,180 colonic pathology images used to predict the survival rate of colorectal cancer histological sections. It includes 9 categories: adipose tissue (ADI), background (BACK), debris (DEB), lymphocytes (LYM), mucus tissue (MUC), smooth muscle tissue (MUS), normal colonic mucosa (NORM), tumor-associated stromal tissue (STR), and colorectal adenocarcinoma epithelium (TUM).
[0042] Step 2: Divide the acquired disease dataset.
[0043] 2.1) Obtain relevant disease datasets from N medical institutions or public datasets, and process them according to the Dirichlet distribution to reduce their degree of non-independence and identical distribution, resulting in N client image classification datasets. In this embodiment, the PathMNIST colon pathology dataset is processed into an image classification dataset with a concentration parameter of 0.1 according to the Dirichlet distribution. Figure 3 As shown;
[0044] 2.2) Images of the same number of samples for each type are extracted from datasets of various medical institutions or public datasets to form a public dataset. In this embodiment, images of the same number of samples for each type are extracted from the PathMNIST colon pathology dataset to form a public dataset.
[0045] 2.3) Divide the samples in the above public dataset and each client image classification dataset into training data and test data in a ratio of 8:2, 7:3 or 9:1 respectively. In this embodiment, the training data and test data are divided in a ratio of 8:2, that is, the training data in the public dataset accounts for 80% and the test data accounts for 20%, and the training data in each client image classification dataset accounts for 80% and the test data accounts for 20%.
[0046] Step 3: Obtain the network model.
[0047] Select network models from the existing network models to serve as the teacher network T on the server side and the student network S on the client side. Specifically, randomly select one of the existing Resnet18 network model, Resnet34 network model, CNN convolutional neural network model, MobilenetV2 network model, ShufflenetV2 network model, and Densenet121 network model as their respective network models.
[0048] In this embodiment, there are a total of 50 clients, of which:
[0049] The ResNet18 network model was selected as the client student network model S, numbered 0 to 9.
[0050] The ResNet34 network model was selected as the client student network model S, numbered 10 to 19.
[0051] The MobilenetV2 network model was selected as the client student network model S, numbered 20 to 29.
[0052] ShufflenetV2 network models were selected as client student network models S, numbered 30 to 39.
[0053] The Densenet121 network model was selected as the client student network model S numbered 40 to 49;
[0054] ShufflenetV2 was selected as the teacher network model T on the server side.
[0055] Step 4: Select the optimal model structure for each client student network S structure using a neural network architecture search method.
[0056] 4.1) Initialize the student network model S for each client, and generate N subnet codes C for each student network model structure. p (0), according to C p (0) Generate N S sub-network models;
[0057] 4.2) Train N S sub-network models using training data from each client's image classification dataset;
[0058] 4.2.1) Set the model learning rate to 0.01, the number of training epochs to 5, and the number of images in a batch to 16;
[0059] 4.2.2) Using the training data in the public dataset, calculate the cross-entropy loss function through the backpropagation algorithm, and use the SGD optimizer to update the parameters;
[0060] 4.2.3) Repeat (4.2.2) until the preset number of rounds to obtain the trained N client student sub-network S models;
[0061] 4.3) Aggregate the N client student sub-network S models after training, and use the aggregation result to update the student network S model:
[0062] 4.3.1) The client student network S model structure is filled with nodes in accordance with N trained client student sub-network S models to ensure that the reconstructed client student network S model structure can contain the structure of N trained client student sub-network S models.
[0063] 4.3.2) Use the parameter averaging method to aggregate the client student subnetwork S model after filling N nodes, and use the aggregation result to update the student network S model, that is, update the aggregated model parameters to the student network model S.
[0064] 4.4) Using the N trained student network S sub-network models as parent individuals, mutate them using binary single-point crossover and binary bit-flip mutation algorithms:
[0065] 4.4.1) Randomly select 2 or 1 sub-network models of S from the N trained student network S sub-network models;
[0066] 4.4.2) Mutate the selected subnetwork model to generate N new subnetwork codes C q (t), according to C q (t) Generate N new subnetworks:
[0067] like Figure 4 As shown in (a), when selecting a sub-network model with 2 S, the binary single-point crossover mutation method is adopted, that is, a crossover point is randomly selected from two binary codes of the same length, and the codes on both sides of the point are swapped to form two new individuals. In this embodiment, the probability of the sub-network codes crossing is 0.9.
[0068] like Figure 4(b) shows that when selecting a sub-network model of 1 S, the binary bit flip mutation method is adopted, that is, the binary code is flipped with a predetermined probability to form a new individual. In this embodiment, the mutation probability is 0.1.
[0069] 4.5) Train N new sub-network models using training data from each client's image classification dataset, and aggregate these N new sub-network models using an aggregation strategy. Use the aggregation result to perform a secondary update on the student network model S:
[0070] 4.5.1) Set the model learning rate to 0.01, the number of training epochs to 5, and the number of images in a batch to 16;
[0071] 4.5.2) Using the training data in the public dataset, calculate the cross-entropy loss function through the backpropagation algorithm, and use the SGD optimizer to update the parameters;
[0072] 4.5.3) Repeat step (4.5.2) until the preset number of rounds are reached to obtain N new client sub-network models after training;
[0073] 4.5.4) Use the parameter averaging method to aggregate the N new client sub-network models after training, and use the aggregation result to update the student network model S a second time, that is, update the aggregated model parameters to the student network model S.
[0074] 4.6) Using test data from the client's local image classification dataset, evaluate the classification accuracy of the S student network model, N S sub-network models, and N new sub-network models, and obtain the accuracy test result Acc for each model. avg ;
[0075] 4.7) Based on the complexity F of the S student network model, N S sub-network models, and N new sub-network models, combined with the accuracy test results Acc avg The performance metric p for each model is calculated using the following formula:
[0076] p=α(1-F)+(1-α)Acc avg
[0077] Where α∈[0,1] is a constant, and in this embodiment α is taken as 0.25, F is the model complexity, and Acc is the model complexity. avg This represents the accuracy of the model in classifying test data on the client's local image classification dataset.
[0078] 4.8) Retain the top N sub-network models ranked by performance index p as the sub-network codes C for the next generation of N S sub-network models. p (t+1), according to Cp (t+1) Generate the next generation of N S sub-network models;
[0079] 4.9) Repeat steps 4.2) to 4.8) until the model with the largest performance index p is found among the S student network model, N S sub-network models, and N new sub-network models. This model is then selected as the optimal student network model S1 for this client.
[0080] Step 5: Perform server-side integrated distillation model aggregation by weighted averaging of model parameters.
[0081] 5.1) Pre-train the server-side teacher network model T and each client-side student network model S1 to obtain the pre-trained teacher network model T1 and student network model S2:
[0082] 5.1.1) Set the training learning rate of the server-side teacher network model T to 0.01, the number of training rounds to 5, and the number of images in a batch to 16;
[0083] 5.1.2) Using the training data in the public dataset, the cross-entropy loss function of the server-side teacher network model T and the cross-entropy loss function of the optimal student network model S1 are calculated by the backpropagation algorithm, and the parameters of the server-side teacher network model T and the optimal student network model S1 of each client are updated by the optimizer SGD.
[0084] 5.1.3) Repeat 5.1.2) until the preset training rounds are completed to obtain the pre-trained server-side teacher network T1 model and the pre-trained client-side student network S2 model.
[0085] 5.2) The pre-trained student network S2 is trained a second time to obtain the trained student network model S3:
[0086] 5.2.1) The server randomly samples M clients to participate in the secondary training, where M≤N, and in this embodiment, M=12;
[0087] 5.2.2) Set the model learning rate to 0.01, the training epochs to 5, and the batch size to 16 images;
[0088] 5.2.3) Using the training data from the image classification datasets of each client, the cross-entropy loss function is calculated through the backpropagation algorithm, and the parameters are updated using the SGD optimizer;
[0089] 5.2.4) Repeat 5.2.3) until the preset number of rounds is reached to obtain the client network model S3 after secondary training;
[0090] 5.3) The server distributes a public dataset to M clients. The clients input the test data from the public dataset into their respective network models S3 for testing, and output the parameters θ of the S3 network. k Upload to the server;
[0091] 5.4) The server will send the S3 network output layer parameters θ uploaded by M clients. k We perform a weighted average to obtain the weighted average output layer Θ(x) of the client-side model:
[0092] Θ(x)=Ensemble Avg ({θ k (x)} k∈K )
[0093] The average logits strategy is used here, θ k (x) represents the output layer parameters of the k-th client, Ensemble Avg This means averaging the output layers of all client models to obtain a new integrated vector.
[0094] 5.5) Input the test data from the public dataset into the server-side teacher network T1 model for prediction, and obtain the image classification prediction result T-RESULT;
[0095] 5.6) Calculate the cross-entropy loss L between the prediction result T-RESULT of the server-side teacher network T1 model and the true value of the test data in the public dataset. CE :
[0096]
[0097] Where y T σ is the actual value of the test data in the public dataset on the server side, B is the batch size, and σ is the softmax function.
[0098] 5.7) Calculate θ from the output layer of the teacher network T1 g KL divergence D of the ensemble result to the client model output layer Θ(x) KL (Θ||θ g ):
[0099]
[0100] 5.8) Based on the cross-entropy loss L CE and KL divergence D KL (Θ||θ g Calculate the final loss function L:
[0101] L = L CE +DKL (Θ||θ g )
[0102] 5.9) Using the training data in the public dataset on the server, the pre-trained server-side teacher network model T1 is iteratively ensembled and distilled using the stochastic gradient descent (SGD) method until the loss function L is minimized, thus obtaining the ensembled and distilled server-side teacher network model T2.
[0103] Step 6: Use the integrated distillation server-side teacher network model T2 to guide the client to update the local distillation model.
[0104] 6.1) Each client inputs its local image classification dataset test data into the server-side T2 model for testing, and saves its own output layer parameters θ after testing. g ;
[0105] 6.2) The client inputs the test data from its respective image classification dataset into the corresponding client-trained student network model S3 for testing, and obtains the prediction results S-RESULT for their respective image classification.
[0106] 6.3) Calculate the cross-entropy loss L between the prediction result S-RESULT of the client student network S3 and the true values of the test data in the client image classification dataset. CE :
[0107]
[0108] Where y T σ is the true value of the test data in the client's image classification dataset, B is the batch size, and σ is the softmax function.
[0109] 6.4) Calculate θ from the output layer of the client student network S3. k To the output layer θ of the teacher network T2 g KL divergence D of the integrated results KL (θ g ||θ k ):
[0110]
[0111] 6.5) Based on the cross-entropy loss L CE and KL divergence D KL (θ g ||θ k The final loss function L is obtained. k :
[0112] L k =L CE +DKL (θ g ||θ k )
[0113] 6.6) Using the training data from each client's image classification dataset, iteratively distill the client student network S3 using the stochastic gradient descent (SGD) method until the loss function L... k Minimize to obtain the locally distilled client student network model S4.
[0114] Step 7: Complete the final update of the student network S4 model and complete the classification and detection tasks for different medical images.
[0115] 7.1) Repeat steps 5 to 6 until the server-side teacher network T2 model converges, and obtain the final student network S4 model for each client.
[0116] 7.2) The client inputs the test data from its respective image classification dataset into the final student network S4 model to obtain the classification results and complete the classification and detection tasks of different medical images.
[0117] The effects of this invention can be further illustrated by the following simulation results.
[0118] I. Experimental Environment and Data
[0119] The experimental environment was Ubuntu 18.04 and Python 3.7, with an Intel Core i7-8700K CPU, an Nvidia RTX 2080Ti GPU, and 32GB of RAM. The deep learning framework used was PyTorch 1.4.0.
[0120] The dataset used in the experiment was the PathMNIST colonic pathology data from the MedMNIST V2 dataset, such as... Figure 2 As shown.
[0121] II. Experiment Content
[0122] Experiment: The accuracy and communication volume of the pFedKD method of this invention and existing image classification methods FedAvg, FedMD, FML and FedKEMF were compared and simulated on the PathMNIST dataset, as shown in Table 1.
[0123] Table 1: Comparison of Experimental Results for Different Algorithms
[0124] method Accuracy (%) Traffic volume (GB) FedAvg 74.7 11.42 FedMD 91.4 / FML 91.4 4.88 FedKEMF 91.0 4.36 This invention 93.0 2.51
[0125] As shown in Table 1, the present invention achieves the best performance in terms of accuracy and communication volume (GB). The accuracy is significantly improved by 18.3 percentage points compared with the traditional method FedAvg, and the communication volume is significantly reduced by 8.91 GB.
Claims
1. A medical image classification method based on knowledge distillation and federated learning under neural architecture search, characterized in that, include: (1) Obtain image classification datasets and network models from N clients: (1a) Obtain relevant disease datasets from N medical institutions or public datasets, and process them according to the degree of non-independent and identically distributed distribution based on the Dirichlet distribution to obtain image classification datasets for N clients, where N>=2; (1b) Extract images of the same number of samples from various medical institutions’ datasets or public datasets to form a public dataset on the server side; (1c) Divide the above image classification dataset and public dataset into training data and test data in a ratio of 8:2, 7:3 or 9:1 respectively; (1d) Select network models from the existing network models as the teacher network T on the server side and the student network S on the client side; (2) By using the neural network architecture search method, select the optimal model structure S1 for each client student network S structure; (3) Model aggregation for server-side integration distillation using a weighted average of model parameters: (3a) Pre-train the server-side teacher network model T and the optimal student network S1 model of each client to obtain the pre-trained teacher network T1 model and student network S2 model. Then, perform secondary training on the pre-trained student network S2 to obtain the secondary trained student network S3 model. (3b) The output layer of the S3 model of each client student network after the second training is weighted and averaged to obtain the output layer Θ(x) of the model; (3c) Input the test data from the public dataset into the server-side teacher network T1 model for prediction, and obtain the prediction result T-RESULT for image classification; (3d) Utilize the output layer Θ(x) of the client-side model and the output layer θ of the server-side teacher network T1 model. g The prediction result T-RESULT and the true values of the test data in the public dataset are used to perform ensemble distillation on the pre-trained server-side teacher network model T1 to obtain the ensemble-distilled server-side model T2. (4) Use the integrated distilled server-side teacher network model T2 to guide the client to perform local distillation model updates: (4a) Each client inputs the test data of its local image classification dataset into the server's T2 model for testing, and saves the output layer parameters θ after each test. g ; (4b) The client inputs the test data from its respective image classification dataset into the corresponding client-trained student network model S3 for testing, and obtains the prediction results S-RESULT for its respective image classification. (4c) Utilizing the output layer parameters θ of the integrated distillation post-server teacher network T2 model g The output layer θ of the student network S3 model trained on the client side. k The prediction result S-RESULT and the true values of the test data in each image classification dataset are used to perform local distillation on the student model S3 trained on each client to obtain the locally distilled client student network model S4. (5) Repeat steps (3) to (4) until the teacher network T2 model trained on the server converges and completes the local distillation of the model update for each client, and each client obtains the final student network S4 model. (6) The client inputs the test data from its respective image classification dataset into the final student network S4 model to obtain the classification results.
2. The method according to claim 1, characterized in that, Step (1a) processes the non-independent and identically distributed distribution of the image classification dataset according to the Dirichlet distribution. This is done by setting the main sample quantity concentration parameter in the image classification dataset to 0.1, based on the principle that the total number of samples is the same but the sample classification is different.
3. The method according to claim 1, characterized in that, Step (1d) Select network models from the existing network models to serve as the teacher network T on the server side and the student network S on the client side. Each network model is randomly selected from the existing Resnet18 network model, Resnet34 network model, CNN convolutional neural network model, MobilenetV2 network model, ShufflenetV2 network model, and Densenet121 network model.
4. The method according to claim 1, characterized in that, Step (2) uses a neural network architecture search method to select the optimal model structure for the S-structure of each client student network. The steps are as follows: (2a) Initialize the student network S for each client, and generate N subnet codes C according to the network structure of S. p (0), according to C p (0) Generate N S sub-network models; (2b) Train the N sub-network models of S using the training data of each client image classification dataset, and aggregate these N heterogeneous models after training, and use the aggregation result to update the student network S model. (2c) Using the N trained S sub-network models as parent individuals, mutate them using binary single-point crossover and binary bit-flipping mutation algorithms to generate N new sub-network codes C. q (t), according to C q (t) Generate N new subnetworks; (2d) Train N new sub-network models using training data from each client image classification dataset, and aggregate these N new sub-network models using an aggregation strategy. Use the aggregation result to update the student network S model a second time. (2e) Use the test data of the local image classification dataset on the client to evaluate the classification accuracy of the S student network model, N S sub-network models, and N new sub-network models, and obtain the individual accuracy test results of each model. (2f) Based on the FLOPs of the complexity of the S student network model, the N S sub-network models, and the N new sub-network models, and combined with the accuracy test results, calculate the performance index p of each model, and retain the top N sub-network models in terms of performance as the subnet codes C of the next generation of N S sub-network models. p (t+1); (2g) Repeat steps (2c) to (2f) until the model with the largest performance index p is found among the S student network model, N S sub-network models, and N new sub-network models. This model is then selected as the optimal student network model S1 for this client.
5. The method according to claim 4, characterized in that, After training in step (2b), the N heterogeneous models are aggregated using an aggregation strategy. First, the client student network S model structure is filled with nodes in accordance with the N heterogeneous models to ensure that the reconstructed client student network S model can contain the structure of the N heterogeneous models. Then, the parameter averaging method is used to aggregate the N node-filled heterogeneous models.
6. The method according to claim 4, characterized in that, Step (2c) uses binary single-point crossover and binary bit-flip mutation algorithms to generate N new subnet codes C. q (t), implemented as follows: (2c1) Randomly select 2 or 1 S sub-network models from N S sub-network models; (2c2) Mutate the selected sub-network model: When selecting a sub-network model with two S's, the binary single-point crossover mutation method is adopted, that is, a crossover point is randomly selected from two binary codes of the same length, and the codes on both sides of the point are swapped to form two new individuals; When selecting a sub-network model with 1S, the binary bit flip mutation method is adopted, that is, flipping a certain number of bits in the binary code with a predetermined probability to form a new individual; (2c3) Repeat (2c1) to (2c2) until all N S subnetwork models have undergone mutation using binary single-point crossover and binary bit-flipping mutation algorithms, generating N new subnetwork codes C. q (t).
7. The method according to claim 4, characterized in that, In step (2f), the performance metric p for each model is calculated using the following formula: p=α(1-F)+(1-α)Acc avg Where α∈[0,1] is a constant, F is the complexity of the calculated model, and Acc is the constant. avg This indicates the accuracy test results of the calculated model.
8. The method according to claim 1, characterized in that, In step (3a), the server-side teacher network model T and the optimal student network model S1 for each client are pre-trained. The steps are as follows: (3a1) Set the training learning rate of the server-side teacher network model T to 0.01, the number of training rounds to 5, and the number of images in a batch to 16; (3a2) Using the training data in the public dataset, the cross-entropy loss function of the server-side teacher network model T and the cross-entropy loss function of the optimal student network model S1 are calculated by the backpropagation algorithm, and the parameters of the server-side teacher network model T and the optimal student network model S1 of each client are updated by the optimizer SGD. (3a3) Repeat (3a2) until the preset training rounds are completed to obtain the pre-trained server-side teacher network T1 model and the pre-trained client-side student network S2 model.
9. The method according to claim 1, characterized in that, Step (3a) involves a second training of the pre-trained student network S2 to obtain the trained student network model S3. The steps are as follows: (3a4) The server randomly samples M clients to participate in this round of training, where M≤N; (3a5) Set the model learning rate to 0.01, the training epochs to 5, and the batch size to 16 images; (3a6) Using the training data of the image classification datasets of each client, calculate the cross-entropy loss function through the backpropagation algorithm, and use the optimizer SGD to update the parameters; (3a7) Repeat (3a6) until the preset number of rounds to obtain the client network model S3 after secondary training.
10. The method according to claim 1, characterized in that, Step (3b) involves performing a weighted average of the output layers of the trained client student network S3 model to obtain the output layer Θ(x). The steps are as follows: (3b1) The server distributes a public dataset to M clients. The clients input the test data from the public dataset into their respective network models S3 for testing, and then input the output layer parameters θ of the S3 network. k Upload to the server; (3b2) The server uses an average logits strategy to evaluate the S3 network output layer parameters θ uploaded by M clients. k After processing, the processed client model output layer Θ(x) is obtained: Θ(x)=Ensemble Avg ({θ k (x)} k∈K ) Where θ k (x) represents the output layer parameters of the k-th client, Ensemble Avg This represents the new vector obtained by averaging the output layers of all client models.
11. The method according to claim 1, characterized in that, In step (3d), the output layer Θ(x) of the client model and the output layer θ of the server-side teacher network T1 model are utilized. g The pre-trained server-side teacher network model T1 is ensembled and distilled using the predicted results T-RESULT and the true values of the test data in the public dataset. The steps include the following: (3d1) Calculate the cross-entropy loss L between the prediction result T-RESULT of the server-side teacher network T1 model and the true value of the test data in the public dataset. CE : Where y T σ is the actual value of the test data in the public dataset on the server side, B is the batch size, and σ is the softmax function. (3d2) Calculate θ from the output layer of the teacher network T1 g KL divergence D of the ensemble result to the client model output layer Θ(x) KL (Θ||θ g ): (3d3) Based on the cross-entropy loss L CE and KL divergence D KL (Θ||θ g Calculate the final loss function L: L=L CE +D KL (Θ||Θ g ) (3d4) Using the training data in the public dataset on the server, the pre-trained server-side teacher network model T1 is iteratively integrated and distilled using the stochastic gradient descent (SGD) method until the loss function L is minimized, thus obtaining the integrated and distilled server-side teacher network model T2.
12. The method according to claim 1, characterized in that, Step (4c) utilizes the output layer parameters θ of the integrated distillation server-side teacher network T2 model. g The output layer θ of the student network S3 model trained on the client side. k The predicted results S-RESULT and the true values of the test data in each image classification dataset are used to perform local distillation on the student model S3 trained on each client. The steps include the following: (4c1) Calculate the cross-entropy loss L between the prediction result S-RESULT of the client student network S3 and the true values of the test data in the client image classification dataset. CE : Where y T σ is the true value of the test data in the client image classification dataset, B is the batch size, and σ is the softmax function. (4c2) Calculate θ from the output layer of the client student network S3. k After integrated distillation, the teacher network T2 output layer θ g KL divergence D of the integrated results KL (θ g ||θ k ): (4c3) Based on the cross-entropy loss L CE and KL divergence D KL (θ g ||θ k Calculate the final loss function L k : L k =L CE +D KL (i g ||θ k ) (4c4) Using the training data from each client's image classification dataset, the client student network S3 is iteratively distilled locally using the stochastic gradient descent (SGD) method until the loss function L... k Minimize to obtain the locally distilled client student network model S4.
