Data-free federated distillation method and system based on zero-order gradient estimation

Abstract

The invention discloses a data-free federated distillation method and system based on zero-order gradient estimation, and the method comprises the steps: initializing a global model and a synthetic image generator through a central server, generating a synthetic image sample in each communication round, transmitting the synthetic image sample to a client, and enabling the client to update a local model through local privacy data, and uploading a prediction result. And the central server calculates the gradient of the generator by using a zero-order gradient estimation technology, and updates generator parameters and global model parameters. And by introducing fidelity loss, adversarial loss, diversity loss and negative information entropy loss, the performance of the generator is optimized. And the gradient of the generator is calculated through zero-order gradient estimation, so that the requirement of accessing a local model of a client is avoided, and privacy is effectively protected. The global model is updated by minimizing knowledge distillation loss, and at the same time, the client further optimizes the local model by receiving an integrated prediction result. The method has the advantage that the communication bandwidth requirement is reduced by reducing the access to the private data of the client.

Description

Technical Field

[0001] The present invention relates to the technical fields of federated learning and generative adversarial networks, and particularly relates to a data-free federated distillation method and system based on zero-gradient estimation. Background Art

[0002] Data sharing is an important driving force for the development of artificial intelligence technology. Built on a vast amount of user data, machine learning models have shone in various fields, providing great convenience for human production and life. However, in the context of increasingly strict data supervision, data users and various data holders cannot directly share data, forming isolated "data silos", which greatly limits the applicability of traditional centralized machine learning paradigms. To alleviate the growing contradiction between data sharing and privacy protection, federated learning has emerged as a new decentralized machine learning paradigm. Federated learning allows multiple participants to collaborate across devices or institutions without directly sharing local data, jointly training machine learning models, and promoting the interconnection of distributed data sources.

[0003] However, the way of updating the global model by aggregating client model parameters in federated learning has the following problems:

[0004] Excessive communication overhead: Frequent transmission of model parameters leads to high communication costs, especially when the number of clients is large or the model complexity is high.

[0005] Risk of privacy attacks: Transmission of model parameters may disclose the privacy information of clients, such as through model inversion attacks or membership inference attacks.

[0006] Limited personalized training: The unified update of the global model cannot meet the personalized needs of clients, resulting in poor performance of the model on local data.

[0007] To solve the above problems, Federated Distillation combines Knowledge Distillation with federated learning, and transfers knowledge by aggregating the prediction results of client models, thus avoiding the need to directly transmit model parameters. The advantages of Federated Distillation are as follows:

[0008] Reduces communication overhead because the amount of transmission of model prediction results is usually much smaller than that of model parameters.

[0009] Enhances privacy protection because clients do not need to share model parameters.

[0010] Supports personalized model design for clients, and each client can use a different local model structure.

[0011] However, federated distillation relies on a key assumption: there exists a publicly available external image migration dataset for use as the data source in the knowledge distillation process. The following limitations exist in the practical application of this assumption:

[0012] Unavailability of external data: In many privacy-sensitive scenarios, it is extremely challenging or even impossible to collect unlabeled cross-domain data related to the task.

[0013] Data distribution mismatch: Even if external data exists, its distribution may not be consistent with the distribution of the client's private data, resulting in poor knowledge transfer effects.

[0014] Privacy risk: Using external data may introduce additional privacy leakage risks, especially when the data source is unknown or unverified.

[0015] Therefore, researching a federated distillation method that does not rely on external data is of great significance for broadening the applicability and universality of federated distillation technology. The present invention proposes a data-free federated distillation method based on zero-gradient estimation. By generating synthetic image samples similar to the distribution of the client's private data, it achieves efficient knowledge transfer in scenarios without external data while retaining the privacy protection advantages of federated learning. Summary of the Invention

[0016] Aiming at the defects of the prior art, the present invention provides a data-free federated distillation method and system based on zero-gradient estimation. By generating synthetic image samples similar to the distribution of the client's private data, it achieves efficient knowledge transfer in scenarios without external data while retaining the privacy protection advantages of federated learning.

[0017] To achieve the above invention objectives, the technical solutions adopted by the present invention are as follows:

[0018] A data-free federated distillation method based on zero-gradient estimation, comprising the following steps:

[0019] S1: The central server initializes the global model F(·|Θ) and the synthetic image generator G(·|Θ G );

[0020] S2: In each communication round, the central server generates synthetic image samples through the generator G(·|Θ G ) and sends the synthetic image samples and their perturbed samples to the clients;

[0021] S3: The clients update the local model using the local private data, perform forward prediction on the received synthetic image samples and their perturbed samples, and upload the prediction results to the central server;

[0022] S4: The central server aggregates the prediction results of the clients, calculates the gradients of the generator through the zero-gradient estimation technique, and updates the generator parameters; meanwhile, the global model parameters are updated using the synthetic image samples;

[0023] S5: The central server feeds back the global knowledge to the clients, and the clients update their local models based on the feedback;

[0024] Among them, the optimization objectives of the generator G(·|Θ G ) include: fidelity loss, adversarial loss, diversity loss, and negative information entropy loss, and the gradients of the generator are calculated through zero-gradient estimation in a black-box manner without accessing the client local model parameters.

[0025] Furthermore, the fidelity loss of the generator G(·|Θ G ) is defined as: Defined as:

[0026]

[0027] Among them, CE is the cross-entropy loss; Represents the synthetic image sample, generated by the generator G; Is the randomly sampled class label; Is the integrated prediction of the client local model; The uniform distribution indicates random sampling from class labels 1 to C; The synthetic image generator G, with the input being the noise z and the class label The output is the generated synthetic image Represents the parameters of the generator; ρ k Represents the weighted coefficient of client k.

[0028] Furthermore, the adversarial loss of the generator G(·|Θ G ) is defined as: Defined as:

[0029]

[0030] Among them, KL is the Kullback-Leibler divergence; σ(·; τ) is the softmax function with the temperature parameter τ, Represents the output of the client k local model for the synthetic image ; Represents the output of the global model for the synthetic image.

[0031] Furthermore, the diversity loss of the generator G(·|Θ G ) is defined as: Defined as:

[0032]

[0033] Among them, B is the batch size, and z i and z j are potential noise vectors; represent two different instances of the synthetic image samples.

[0034] Furthermore, the negative information entropy loss of the generator G(·|Θ G ) is calculated through the following steps: By the following steps:

[0035] S11: Calculate the class frequency distribution p of the synthetic image samples under the integrated prediction of the client local model;

[0036] S12: Among them, p c is the frequency of class c. Minimizing this loss makes the class distribution balanced.

[0037] Furthermore, the zero-gradient estimation includes:

[0038] S21: The central server sends the synthetic image samples and their perturbed samples to the clients, where u i is a random direction vector and ε is a smoothing parameter;

[0039] S22: The clients return the prediction results of the local model for the samples;

[0040] S23: The central server estimates the gradient through forward difference based on the prediction differences:

[0041]

[0042] Among them, u i represents the random direction vector, which is used to introduce diversity in the perturbation. q represents the number of perturbed samples.

[0043] Furthermore, the update gradient of the generator parameters is calculated by the chain rule as:

[0044]

[0045] Among them, is obtained by accessing the generator parameters through white-box.

[0046] Furthermore, the update of the global model is achieved by minimizing the knowledge distillation loss L KD , and the formula is as follows:

[0047]

[0048] Among them, σ(·; τ) represents the softmax function with temperature parameter τ; represents the output of the local model of client k for the synthetic image at time step t; represents the local model parameters of client k at time step t. represents the output of the global model at time step t - 1; represents the set of clients at the current moment.

[0049] Furthermore, after receiving the integrated prediction from the central server, the client further updates its local model, and its loss function is:

[0050]

[0051] represents the prediction of the local model of client k for the synthetic image sample.

[0052] Furthermore, the synthetic image generator G(·|Θ G ) is a conditional generative adversarial network, with the input being Gaussian noise z and uniformly sampled class labels and the output being synthetic image samples similar to the distribution of the client's private data.

[0053] The present invention also discloses a data - free federated distillation system based on zero - order gradient estimation, which can be used to implement the above - mentioned data - free federated distillation method based on zero - order gradient estimation. Specifically, it includes:

[0054] A central server, which is used to manage the training of the global model and send synthetic image samples and offset samples to the client;

[0055] A generator, which is used to generate synthetic image samples according to random noise and random labels, and optimize their fidelity, transferability, diversity, and class distribution balance;

[0056] A client, which is used to calculate the predictions of its local model for the synthetic image samples and their offset samples and upload them to the central server for gradient aggregation;

[0057] A zero - order gradient estimation module, which is used to estimate the approximate gradient of the generator, avoid white - box access to the local model of the client, and promote the training process of data - free federated distillation.

[0058] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the above - mentioned data - free federated distillation method based on zero - order gradient estimation.

[0059] The present invention also discloses a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-described data-free federated distillation method based on zero-gradient estimation is implemented.

[0060] Compared with the prior art, the advantages of the present invention are as follows:

[0061] 1. By adopting the data-free federated distillation method, the present invention can perform model training without accessing the local data of the client, effectively avoiding the risk of privacy leakage.

[0062] 2. By combining the generative adversarial network (GAN) and the zero-gradient estimation technology, high-quality synthetic image samples can be generated, thereby improving the training effects of the global model and the client local model and enhancing the prediction performance.

[0063] 3. Since the present invention does not need to frequently exchange large-scale raw data or local model parameters, only synthetic images and their perturbed samples need to be exchanged, greatly reducing the requirement for communication bandwidth.

[0064] 4. Through the design of the diversity loss and the adversarial loss, synthetic data with high diversity can be generated, enhancing the model's adaptability to different data distributions and noises and improving the generalization ability of the model.

[0065] 5. By adopting the zero-gradient estimation technology, the central server can perform gradient calculation and model update without accessing the local model parameters of the client, reducing the complexity and privacy risk of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is the system framework diagram of the data-free federated distillation method based on zero-gradient estimation according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0067] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the following further describes the present invention in detail with reference to the drawings and by way of examples.

[0068] A data-free federated distillation method based on zero-gradient estimation includes the following steps:

[0069] S1: The central server initializes the global model F(·|Θ) and the synthetic image generator G(·|Θ G );

[0070] S2: In each communication round, the central server generates synthetic image samples G through the generator G(·|Θ and sends the synthetic image samples and their perturbed samples to the client;

[0071] S3: The client updates the local model using local private data, makes forward predictions on the received synthetic image samples and their perturbed samples, and uploads the prediction results to the central server;

[0072] S4: The central server aggregates the prediction results of the clients, calculates the gradients of the generator through the zero-gradient estimation technique, and updates the generator parameters; meanwhile, updates the global model parameters using the synthetic image samples;

[0073] S5: The central server feeds back the global knowledge to the client, and the client updates the local model based on the feedback;

[0074] Among them, the optimization objectives of the generator G(·|Θ G ) include: fidelity loss, adversarial loss, diversity loss, and negative information entropy loss, and the gradients of the generator are calculated through zero-gradient estimation in a black-box manner without accessing the client's local model parameters.

[0075] In practical application scenarios, data is usually scattered across K different clients. Each client k ∈ {1, 2,..., K} holds an independent private dataset where represents the sample input, represents the sample label, and N k represents the number of samples. Federated learning aims to perform distributed collaborative training based on the datasets distributed across various clients to jointly train a machine learning model whose overall training objective function is where is the local training objective function of the k-th client, is the contribution weight of the k-th client, which is usually represented by the private data of each client the data volume N k of the private data of all clients and the ratio of the data volume N. To solve the above optimization objective, federated learning decentralizes the model training process to the clients. Before the training starts, the central server needs to share the model architecture it uses with all clients so as to allow the clients to use local models with the same structure to replace the global model to complete the training process on the private data to prevent the leakage of private data.

[0076] Different from federated learning, federated distillation transfers the knowledge of the client's local model to the global model of the central server with the help of an unlabeled public transfer dataset, enabling each client to independently design its own local model which can have different model structures and sizes. However, such an unlabeled public transfer dataset is not always available in practice.

[0077] For the scenario of knowledge transfer without publicly available external image transfer data sources, the present invention integrates the knowledge distillation technology without data into federated learning, and uses the client local model as a discriminator to train the global model in an adversarial learning manner on the central server and a synthetic image generator based on a generative adversarial network to approximate the training space of the client local model, thereby generating synthetic image samples similar to the distribution of the client's private data, and further providing an effective data source for the subsequent knowledge transfer stage. Specifically, the conditional generator takes a d -dimensional random noise vector sampled from a standard Gaussian distribution z and a random class label sampled from a uniform distribution as input, and a random class label sampled from a uniform distribution as conditional context, and outputs a synthetic image sample where represents the number of classes. However, directly using this generator in the federated distillation process will generate task-irrelevant or even completely random synthetic image samples, and the knowledge of the client local model cannot be effectively transferred to the global model. Therefore, it is necessary to design a suitable training objective for the generator so that it can capture the training space of the client local model to generate synthetic image samples similar to the distribution of the client's private data.

[0078] To this end, the present invention constructs a fidelity loss for the generator to ensure that the generated synthetic image samples fall into a data distribution similar to the client's private data, as shown in Equation (1):

[0079]

[0080] The above formula promotes the fidelity of the synthetic image sample under the class label to the client's private data distribution by minimizing the cross-entropy loss CE(·) between the integrated prediction of the client local model on the generated synthetic image sample and the specified class label . As a result, the synthetic image sample is classified into the specified class label with a high probability without accessing the client's private data, thereby promoting the fidelity of the synthetic image sample to the client's private data distribution under the class label Knowledge distillation is performed as a migration data source to transfer the integrated knowledge of the client's local model to the global model, as shown in Equation (2):

[0081]

[0082] where KL(·||·) is the Kullback-Leibler divergence loss function, which is used to measure the difference between two distributions, and σ(·; τ) is a softmax function with a temperature parameter τ added, which is used to adjust the smoothness of the predicted probability distribution. When τ > 1, a smoother predicted probability distribution can be obtained, which provides more additional information about the similarity between classes and helps the target model better learn the detailed information of the source model. In each communication round, the global model and the generator are alternately trained on the central server to continuously improve their respective capabilities.

[0083] However, empirical experience shows that if only the fidelity loss is used to train the generator as the training progresses, it will approach 0, making it easy for the generator to generate synthetic image samples that can be correctly classified by the client's local model. Therefore, these simple synthetic image samples will gradually fall into the space far from the decision boundary of the client's local model, making it difficult for the client's local model to transfer its knowledge near the decision boundary to the global model on the central server through these samples. To guide the generator to generate samples in the space near the decision boundary of the client's local model, the present invention constructs an adversarial loss for the generator to improve the transferability of the generated synthetic image samples, as shown in Equation (3):

[0084]

[0085] The above formula maximizes the difference between the predicted distributions of the global model and the client's local model on the generated synthetic image samples to form an adversarial learning objective as shown below with Equation (2):

[0086]

[0087] By optimizing the generator By conducting continuous adversarial exploration, the global model can be exposed to more difficult synthetic image samples, thereby facilitating the transfer of knowledge from the client's local model near the decision boundary space to the global model on the central server.

[0088] In addition, the conditional generator tends to encounter the mode collapse problem during training because it tends to learn from the conditional context that determines the main content while ignoring the potential noise z that induces different outputs, resulting in the generated synthetic image samples being too similar to each other and lacking diversity. Correspondingly, since the generated synthetic image samples cannot cover all patterns in the client's private data distribution, the subsequent knowledge transfer process can only extract limited knowledge from the client's local model. To make the synthetic image samples generated by the generator diverse, the present invention introduces a diversity loss for the generator as shown in Equation (5):

[0089]

[0090] where B represents the batch size. The above formula increases the chance of generating data from different distribution patterns by explicitly maximizing the distance between the synthetic image samples generated in each training batch and the corresponding latent noise, thereby ensuring the generation of diverse synthetic image samples.

[0091] Although the above considerations take into account the fidelity, transferability, and diversity of the conditional generator , recent research progress in data-free knowledge distillation shows that simply uniformly sampling the class labels does not guarantee that the generated synthetic image samples are class-balanced, because the integrated predictions of the synthetic image samples generated by the client's local model in each training batch generally tend to have a higher probability of falling into certain classes and a lower probability of falling into other classes. To ensure that the synthetic image samples generated in each training batch are evenly distributed across each class, the present invention introduces a negative information entropy loss into the training process of the generator First, calculate the frequency distribution of its falling into each class based on the integrated predictions of the synthetic image samples generated by the client's local model in each training batch which represents the probability of generating synthetic image samples in each training batch for each class, as shown in Equation (6):

[0092]

[0093] where σ(·) is the standard softmax function. Then, the corresponding negative information entropy loss is calculated on the frequency distribution p, as shown in Equation (7):

[0094]

[0095] where H(·) is the information entropy loss, which is used to measure the information entropy of a probability distribution. By minimizing Equation (7), the corresponding information entropy loss will be maximized. When the information entropy loss reaches the maximum value, each element p in the frequency distribution p c will be equal to This means that the generator can generate data from each category with roughly the same probability, thereby generating a batch of synthetic image samples with a balanced class distribution.

[0096] In summary, by combining the fidelity loss the adversarial loss the diversity loss and the negative information entropy loss the final training objective function of the generator is shown in Equation (8):

[0097]

[0098] where β1, β2, and β3 are adjustable scaling factors used to balance the contributions of the newly introduced different losses.

[0099] The global model training loss in Equation (2) and the generator training loss in Equation (8) are the two training objective functions that need to be minimized during the federated distillation process. Among them, training the global model requires using the gradient descent method to calculate the gradient of the loss function with respect to the d-dimensional optimization variable to complete the backpropagation update of the parameters Θ of the global model, as shown in Equation (9): where η is the learning rate of the global model. The above equation can be naturally completed on the central server because the central server can access the parameters Θ of the global model to calculate the corresponding gradient. While training the generator

[0100]

[0101] requires using the gradient descent method to calculate the gradient of the loss function with respect to the d'-dimensional optimization variable to complete the backpropagation update of the parameters Θ' of the generator, as shown in Equation (10): the gradient To complete the backpropagation update of the parameters of the generator, as shown in Equation (10):

[0102]

[0103] where is the learning rate of the generator. However, according to the chain rule, the calculation of the generator gradient requires accessing the parameters Θ of the local models of each client k , as shown in Equation (11):

[0104]

[0105] To solve Equation (11), each client must share its local model with the central server for white-box access to its parameters, so as to perform first-order gradient-based optimization to complete the training of the generator. However, this approach runs counter to the original design of federated distillation and greatly undermines the many benefits brought about by federated distillation due to avoiding model parameter sharing. To be able to train the generator without sharing the local models of the clients, the present invention uses zero-gradient estimation technology to estimate the gradient after flowing through the local models of the clients in a black-box optimization manner to complete the training of the generator, promoting the applicability of federated distillation in scenarios lacking publicly available external image migration data sources.

[0106] Specifically, through zero-gradient estimation, a biased estimate of the gradient of the loss function with respect to the d-dimensional optimization variable can be obtained as an approximate gradient. However, since the generator usually contains millions of model parameters, directly performing gradient estimation on the high-dimensional model parameters not only requires high resource requirements but is also extremely inefficient, because the training accuracy and computational efficiency of zero-gradient estimation will decrease as the dimension of the target to be estimated increases. To overcome this curse of dimensionality, the present invention chooses to estimate the gradient of the loss function with respect to the d-dimensional synthetic image samples generated by the generator because it has a significantly lower dimension (i.e., ) and is independent of the model scale of the generator. The corresponding chain rule is as shown in Equation (12):

[0107]

[0108] where the second term in Equation (12) can be obtained by accessing the generator​ parameters to calculate the corresponding gradients, as shown in Equation (13):

[0109]

[0110] However, the first term in Equation (12) still needs to access the local model parameters Θ k to calculate the corresponding gradients, as shown in Equation (14):

[0111]

[0112] To solve Equation (14), the present invention uses the zero-order gradient estimation technique. First, the central server sends synthetic image samples to each client along with the samples offset by a set of q random direction variables sampled from the standard Gaussian distribution

[0113] where ∈>0 is called the smoothing parameter and is used to control the offset size of the offset samples. Second, each client uses the local model obtained after training on its local private data to calculate the local predictions corresponding to the synthetic image samples and their offset samples and and uploads them to the central server. Then, the central server aggregates the local predictions collected from the clients to obtain the integrated predictions and and calculates the forward difference to obtain an approximate solution of Equation (14), as shown in Equation (15): and Based on the above equation, a biased estimate

[0114]

[0115] of the true gradient can be obtained as the approximate gradient, as shown in Equation (16):

[0116]

[0117] This approximate gradient can then be used to complete the backpropagation update of the generator in Equation (10), as shown in Equation (17):

[0118]

[0119] Through the above process, the central server only needs to send the synthetic image samples generated by the generator in each training batch and their offsets to each client, and the client only needs to calculate the corresponding local model predictions and upload them to the central server, so that the training of the global model and the generator can be completed simultaneously. At the same time, the client only needs to perform the forward propagation of the local model on the received data during this process, so it will not cause too much computational burden. Compared with previous studies, the present invention eliminates the need to share the local model parameters of the client, retains many benefits brought by federated distillation, and realizes a federated distillation method that does not rely on publicly available external image migration data sources.

[0120] As Figure 1 shown, in each communication round t, the specific training process of the present invention is as follows:

[0121] Step 1: The central server randomly samples a subset of m clients from K clients to participate in federated learning to simulate the situation where not all clients are always online or available in the real scenario, where m = max(εK, 1) and the sampling rate ε ∈ [0, 1].

[0122] Step 2: The central server samples a batch of random noise and random labels to generate a batch of synthetic image samples

[0123] Step 3: The central server samples a batch of random direction variables and constructs offset samples of the synthetic image samples Then it sends the synthetic image samples and their offset samples to each client.

[0124] Step 4: Each client participating in collaborative training iteratively trains E k times on its respective private data k to update its local model The update formula is where, is usually a cross-entropy loss function CE(·), which aims to measure the difference between the predictions of the local model on the private data X k and the true labels Y k The specific form is

[0125] Step 5: Each client participating in collaborative training Compute the local predictions of the updated local model on the synthetic image samples received from the central server and the offset samples in parallel and upload them to the central server.

[0126] Step 6: The central server collects the local predictions uploaded by the clients and computes the ensemble prediction and estimates the gradient through formula (16) to update the parameters of the generator whose update formula is where is the generator loss proposed by the present invention, and its specific form is shown in formula (8).

[0127] Step 7: The central server uses the ensemble prediction to update the parameters of the global modeler on the synthetic image samples whose update formula is where is the global distillation loss function, which aims to measure the difference between the predictions of the global model on the synthetic image samples and the ensemble prediction, and its specific form is

[0128] Step 8: The central server broadcasts the ensemble prediction to each client.

[0129] Step 9: Each client participating in collaborative training parallelly uses the ensemble prediction to update the parameters of the local model on the synthetic image samples whose update formula is where is the local distillation loss function of the k-th client, and its specific form is

[0130] The above steps are iteratively executed for T communication rounds or until a specific condition is met.

[0131] In another embodiment of the present invention, a data-free federated distillation system based on zero-gradient estimation is provided, which can be used to implement the above data-free federated distillation method based on zero-gradient estimation. Specifically, it includes:

[0132] A central server for managing the training of the global model and sending synthetic image samples and offset samples to the clients;

[0133] A generator for generating synthetic image samples based on random noise and random labels and optimizing their fidelity, transferability, diversity, and balanced class distribution;

[0134] A client for calculating the predictions of its local model on the synthetic image samples and their offset samples and uploading them to the central server for gradient aggregation;

[0135] A zero-gradient estimation module for estimating the approximate gradient of the generator, avoiding white-box access to the client's local model, and facilitating the training process of data-free federated distillation.

[0136] In another embodiment of the present invention, a terminal device is provided. The terminal device includes a processor and a memory. The memory is used to store a computer program. The computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function. The processor described in the embodiment of the present invention may be used for the operation of the data-free federated distillation method based on zero-gradient estimation, including the following steps:

[0137] S1: The central server initializes the global model F(·|Θ) and the synthetic image generator G(·|Θ G );

[0138] S2: In each communication round, the central server generates synthetic image samples through the generator G(·|Θ G ) and sends the synthetic image samples and their perturbed samples to the client; and sends the synthetic image samples and their perturbed samples to the client;

[0139] S3: The client updates the local model using local private data, makes forward predictions on the received synthetic image samples and their perturbed samples, and uploads the prediction results to the central server;

[0140] S4: The central server aggregates the prediction results of the clients, calculates the gradients of the generator through the zero-gradient estimation technique, and updates the generator parameters; meanwhile, updates the global model parameters by using the synthetic image samples;

[0141] S5: The central server feeds back the global knowledge to the clients, and the clients update their local models based on the feedback;

[0142] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is the memory device in the terminal device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides storage space, and this storage space stores the operating system of the terminal. And, one or more instructions suitable for being loaded and executed by the processor are also stored in this storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0143] One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the above-mentioned embodiment of the data-free federated distillation method based on zero-gradient estimation; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps:

[0144] S1: The central server initializes the global model F(·|Θ) and the synthetic image generator G(·|Θ G )

[0145] S2: In each communication round, the central server generates synthetic image samples through the generator G(·|Θ G ) and sends the synthetic image samples and their perturbation samples to the clients;

[0146] S3: The clients update their local models using the local private data, perform forward prediction on the received synthetic image samples and their perturbation samples, and upload the prediction results to the central server;

[0147] S4: The central server aggregates the prediction results of the clients, calculates the gradients of the generator through the zero-gradient estimation technique, and updates the generator parameters; meanwhile, updates the global model parameters by using the synthetic image samples;

[0148] S5: The central server feeds back the global knowledge to the client, and the client updates the local model based on the feedback.

[0149] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0151] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0153] Those of ordinary skill in the art will realize that the embodiments described herein are provided to assist the reader in understanding the implementation methods of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on these technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention.

Claims

1. A data-free federated distillation method based on zero-order gradient estimation, characterized in that: The following steps are involved: S1: The central server initializes the global model F(·|Θ) and the synthetic image generator G(·|Θ G ); S2: In each communication round, the central server generates G ) Generate synthetic image samples And send the synthetic image samples and their perturbation samples to the client; S3: The client updates the local model using the local private data, performs forward prediction on the received synthetic image samples and their perturbation samples, and uploads the prediction results to the central server; S4: The central server aggregates the prediction results of the clients, calculates the gradient of the generator through the zero-order gradient estimation technique, and updates the generator parameters. At the same time, the global model parameters are updated using the synthetic image samples. S5: The central server feeds back the global knowledge to the client, and the client updates the local model based on the feedback; Wherein, the generator G(·|Θ G )’s optimization objectives include: fidelity loss, adversarial loss, diversity loss, and negative information entropy loss, and the gradient of the generator is calculated through zero-order gradient estimation in a black-box manner without accessing the client’s local model parameters.

2. The data-free federated distillation method based on zero-order gradient estimation according to claim 1, characterized in that: The generator G(·|Θ G ) Defined as: Among them, CE is the cross entropy loss; Represents a synthetic image sample, generated by the generator G; is the randomly sampled class label; Provides ensemble predictions for client-side local models; Uniform distribution, which means random sampling from category labels 1 to C; Synthetic image generator G, input is noise z and category label The output is the generated composite image represents the parameters of the generator; ρ k represents the weighting coefficient of client k.

3. The data-free federated distillation method based on zero-order gradient estimation according to claim 2, characterized in that: The generator G(·|Θ G )’s adversarial loss Defined as: Where KL is the Kullback-Leibler divergence; v(·; τ) is the softmax function with temperature parameter τ, Represents the client k local models for synthesized images Output: represents the output of the global model for the synthesized image.

4. The data-free federated distillation method based on zero-order gradient estimation according to claim 3, characterized in that: The generator G(·|Θ G ) Defined as: Where B is the batch size, z i 、z j is the potential noise vector; Represents two different instances of synthetic image samples.

5. The data-free federated distillation method based on zero-order gradient estimation according to claim 4, characterized in that: The generator G(·|Θ G )’s negative information entropy loss Calculate by following the steps below: S11: Calculate the category frequency distribution p of the synthetic image sample under the client local model ensemble prediction; S12: Among them, p c is the frequency of category c, minimizing this loss makes the category distribution balanced.

6. The data-free federated distillation method based on zero-order gradient estimation according to claim 5, characterized in that: The zero-order gradient estimation includes: S21: The central server sends a synthetic image sample to the client and its perturbation samples where u i is a random direction vector, ε is a smoothing parameter; S22: The client returns the prediction result of the local model for the sample; S23: The central server estimates the gradient through forward difference based on the predicted difference: Among them, u i represents a random direction vector used to introduce diversity in the perturbation; q represents the number of perturbation samples.

7. The data-free federated distillation method based on zero-order gradient estimation according to claim 6, characterized in that: The updated gradient of the generator parameters is calculated by the chain rule as: in, Obtained via white-box access to the generator parameters.

8. The data-free federated distillation method based on zero-order gradient estimation according to claim 7, characterized in that: The global model is updated by minimizing the knowledge distillation loss L KD To implement, the formula is as follows: Among them, σ(·; τ) represents the softmax function with temperature parameter τ; Represents the local model of client k at time step t to synthesize the image Output: represents the local model parameters of client k at time step t; represents the output of the global model at time step t-1; Represents the client collection at the current moment.

9. The data-free federated distillation method based on zero-order gradient estimation according to claim 8, characterized in that: After receiving the integrated prediction from the central server, the client further updates the local model, and its loss function is: represents the prediction of the local model of client k for the synthetic image sample.

10. The data-free federated distillation method based on zero-order gradient estimation according to claim 1, characterized in that: The synthetic image generator G(·|Θ G ) is a conditional generative adversarial network, with Gaussian noise z and uniformly sampled class labels as input The output is a synthetic image sample with a distribution close to the client's private data.

Images