A differential privacy federated learning method and system for heterogeneous data

Abstract

The application provides a differential privacy federated learning method and system for heterogeneous data. The method comprises the following steps: a client receives local model parameters issued by a server, calculates information quantity and performs hierarchical normalization to obtain normalized information quantity, and constructs a mask matrix with a preset hyperparameter, and calculates an element product with the local model parameters to screen a parameter set with low information quantity; a regularization term is introduced to suppress the update amplitude of the parameters in the parameter set to construct a loss function, the local model parameters are iteratively updated to obtain intermediate local model parameters, which are decomposed into a low-dimensional vector and a redundant vector, and after being cropped and adding noise, the target model parameters are restored and uploaded to the federated model for aggregation according to the set aggregation weight; the server identifies the global data according to the local discriminator, and the closer the distribution of the local data set is to the distribution of the global data, the higher the aggregation weight allocated by the client; the method can improve the model prediction accuracy in the heterogeneous data while meeting the strict differential privacy proof.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model training, in particular to a differential privacy federated learning method and system for heterogeneous data. BACKGROUND

[0002] With the continuous acceleration of global digital transformation process, various industries have generated a large amount of data resources in daily life and work. In actual application scenarios, these data often show a highly dispersed characteristic and contain a large amount of sensitive information. Direct data publishing will cause serious social problems due to privacy leakage. Therefore, the existing research mainly focuses on how to protect user privacy while fully utilizing these dispersed data to realize the interconnection and intercommunication of multi-party data sources. Federated learning, as an innovative distributed deep learning paradigm, provides a new idea for solving this problem, allowing data to be stored locally without leaving the environment, and building a joint model through multi-party collaborative training, so as to realize the optimization and improvement of model performance while ensuring data privacy and security.

[0003] In the prior art, federated learning faces the problem that the data held by each party participating in federated learning shows high heterogeneity and federated learning still has privacy leakage. Therefore, federated learning algorithm needs to be combined with other advanced privacy protection strategies. Differential privacy is one of the most widely adopted privacy protection means in the field of federated learning. Carefully designed random noise is introduced into the original data or algorithm output to ensure that the addition or deletion of a single data record has little effect on the final output result. Existing research shows that when the federated learning model introduces the differential privacy mechanism in the heterogeneous data scenario, its utility will often suffer more serious losses, specifically manifested as model accuracy decline, generalization ability weakening, etc., which restricts the wide application of federated learning in real scenarios. The existing research ignores the characteristics of data heterogeneity and differential privacy mechanism, and also fails to explore the reasons why the introduction of differential privacy in heterogeneous data causes more serious loss of model utility in combination with the internal relationship between the two. This makes the algorithm proposed by the present application only a simple superposition of the existing data heterogeneity optimization method and differential privacy optimization method, thereby causing problems such as poor portability, limited improvement effect, and dependence on public data sets. SUMMARY

[0004] In view of this, the present application provides a differential privacy federated learning method and system for heterogeneous data to eliminate or improve one or more defects in the prior art, and solves the problem that the prediction accuracy of the differential privacy federated learning training model for heterogeneous data cannot be improved while protecting privacy in the prior art.

[0005] One aspect of the present invention provides a differential privacy federated learning method for heterogeneous data, the method comprising the following steps:

[0006] Receive local model parameters sent by the server, calculate the amount of information used to represent the correlation between the local model parameters and the client's local dataset, and perform hierarchical normalization processing according to the level where the local model parameters are located to obtain normalized information.

[0007] A mask matrix is ​​constructed based on the comparison between the normalized information content and the preset hyperparameters. The element-wise product of the mask matrix and the local model parameters is calculated to filter out local model parameters whose normalized information content is lower than the preset hyperparameters and construct them as a parameter set.

[0008] A regularization term is introduced to suppress the update magnitude of the parameters in the parameter set to construct a loss function. The local model parameters are iteratively updated until a preset iteration termination condition is reached to obtain intermediate local model parameters. The intermediate local model parameters are projected onto the feature subspace to decompose them into low-dimensional vectors and redundant vectors. The low-dimensional vectors and redundant vectors are pruned and Gaussian noise is added to restore them to the original dimensions to obtain the target model parameters. The feature subspace is obtained by the server using a pre-trained generator to generate a global dataset, calculate the gradient matrix of the federated model, and then extract a predetermined number of principal component feature vectors using the power method.

[0009] The target model parameters are uploaded to the federated model on the server for aggregation according to the set aggregation weights; wherein, the process of setting the aggregation weights includes: the server collects the target model parameters of each client and the parameters of a pre-trained local discriminator for recognizing local client data, loads the parameters of the local discriminator to recognize the global data, and assigns higher aggregation weights to clients whose local dataset distribution is closer to the global data distribution based on the recognition results.

[0010] In some embodiments, the expression for calculating the amount of information representing the correlation between the local model parameters and the client's local dataset is:

[0011]

[0012] The expression for the normalized information content is obtained by performing hierarchical normalization processing according to the level where the local model parameters are located:

[0013] ;

[0014] in, This represents the local model parameters sent by the server. Indicates client Local dataset, representing the client upper layer local model parameters, representing the client layer local model parameters.

[0015] In some embodiments, the expression of the mask matrix is:

[0016] ;

[0017] wherein, representing the preset hyperparameter, representing the client layer local model parameters.

[0018] In some embodiments, the pre-training process of the local discriminator includes:

[0019] marking the samples in the local data set as positive samples, and marking the samples in the adversarial data set obtained after data transformation or adversarial sample generation as negative samples;

[0020] inputting the samples in the local data set into the initial local discriminator to obtain a first result, and inputting the samples in the adversarial data set into the initial local discriminator to obtain a second result;

[0021] constructing a local discriminator loss function according to the difference between the first result and the second result, taking minimizing the local discriminator loss function as an objective, and performing parameter iterative update on the initial local discriminator to obtain the local discriminator; the expression of the local discriminator loss function is:

[0022] ;

[0023] wherein, representing the local data set, representing the adversarial data set, representing the i-th sample in the local data set, representing the i-th sample in the adversarial data set.

[0024] In some embodiments, the pre-training process of the generator includes:

[0025] inputting randomly sampled noise into an initial generator to output generated data;

[0026] ​​A generator loss function is constructed based on the generated data and the local dataset. The generator is obtained by iteratively updating the parameters of the initial generator with the objective of minimizing the generator loss function. The expression for the generator loss function is:

[0027]

[0028]

[0029] ;

[0030] ;

[0031] in, This refers to the generated data. The labels representing the generated data, This refers to the generator. and Indicates the weighting coefficient. Indicates the total number of clients. This indicates the category number to which the generated data belongs. Indicates the first Generated data Input client The parameters are The calculation results of the local model, Indicates the generator at the 1st... The mean of the feature maps on each batch normalized layer Indicates client In the local model, the first The mean of each batch of normalized layers Indicates the generator at the 1st... The variance of feature maps on each batch normalized layer Indicates client In the local model, the first The variance of each batch of normalized layers Indicates the first The input parameters for generating data are: The calculation results of the local discriminator.

[0032] In some embodiments, the feature subspace is obtained by the server using a pre-trained generator to calculate the gradient matrix of the federated model from global data, and then extracting a predetermined number of principal component feature vectors using the power method.

[0033] Global data is generated based on a pre-trained generator, and then a training dataset is constructed by assigning random labels to the global data.

[0034] computing gradients of the federated model on the training dataset and combining all the gradients into a gradient matrix;

[0035] generating a random matrix and computing a matrix composed of the first preset number of eigenvectors of the gradient matrix through the power method for multiple iterations, and combining the first preset number of eigenvectors into an eigenspace; the expression of the power method for multiple iterations is:

[0036] ;

[0037] ;

[0038] wherein, denotes the gradient matrix, denotes the random matrix.

[0039] In some embodiments, the intermediate local model parameters are projected on the eigenspace to be decomposed into a low-dimensional vector and a redundant vector; the expression of the low-dimensional vector and the redundant vector is:

[0040] ;

[0041] ;

[0042] After the low-dimensional vector and the redundant vector are clipped and added with Gaussian noise, the expression of the target model parameters is:

[0043] ;

[0044] ;

[0045] ;

[0046] wherein, denotes the intermediate local model parameters, denotes the random matrix, denotes a preset clipping coefficient, denotes a noise variance, denotes a low-dimensional vector dimension, denotes a redundant vector dimension, denotes a total number of clients.

[0047] In some embodiments, the method further comprises:

[0048] inputting the global data into the local model to output a plurality of client probability vectors representing categories to which the data belongs; and performing weighted summation on the plurality of client probability vectors to obtain a client weighted probability vector.

[0049] outputting a federated model probability vector representing a category to which the data belongs after inputting the global data into the federated model;

[0050] constructing a federated model loss function according to the client weighted probability vector and the federated model probability vector, and iteratively updating the federated model parameters; wherein the expression of the federated model loss function is:

[0051]

[0052]

[0053]

[0054] wherein, the client weighted probability vector is represented by, the federated model probability vector output after inputting the i-th global data into the federated model with the i-th parameter is represented by, the client probability vector output after inputting the i-th global data into the local model with the i-th parameter is represented by,

[0055] In another aspect, the present application also provides a differential privacy federated learning system for heterogeneous data, comprising a processor, a memory and computer programs / instructions stored on the memory, wherein the processor is configured to execute the computer programs / instructions, and the system implements the steps of the method according to any one of the above aspects when the computer programs / instructions are executed.

[0056] In another aspect, the present application also provides a computer readable storage medium having computer programs / instructions stored thereon, wherein the programs / instructions are executed by a processor to implement the steps of the method according to any one of the above aspects.

[0057] ​​​​​​​​​​​​​In the differential privacy federated learning method for heterogeneous data described in this invention, a regularization term is introduced to suppress the update magnitude of parameters in the parameter set to construct a loss function, effectively preserving key information of the client model; the intermediate local model parameters are projected onto the feature subspace to decompose them into low-dimensional vectors and redundant vectors, reducing the total amount of noise added during training; the parameters of the local discriminator are loaded to identify the global data, and based on the identification results, clients whose local dataset distribution is closer to the global data distribution are assigned higher aggregation weights to correct the update direction of the federated model; privacy leakage is avoided while improving the prediction accuracy of the differential privacy federated learning model under heterogeneous data through pruning and noise addition.

[0058] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0059] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0060] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0061] Figure 1 This is a flowchart illustrating the differential privacy federated learning method for heterogeneous data according to an embodiment of the present invention.

[0062] Figure 2 This is a schematic diagram of the differential privacy federated learning method for heterogeneous data according to an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0064] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0065] It should be emphasized that the term "comprises / comprising" when used in this text is taken to specify the presence of stated features, elements, steps or components but does not preclude the presence or addition of one or more other features, elements, steps or components.

[0066] It should also be noted here that the term "connected" is used herein to mean not only direct connection but also indirect connection via intervening items, unless specifically stated otherwise.

[0067] In the following, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, like reference numerals designate like or similar parts or like or similar steps.

[0068] In the prior art, when the federated learning model introduces a differential privacy mechanism in a heterogeneous data scenario, its utility often suffers more serious losses. Existing research work ignores the characteristics of data heterogeneity and differential privacy mechanism, and also fails to explore the reason why the introduction of differential privacy under heterogeneous data causes more serious loss of model utility in combination with the internal relationship between the two. This makes the algorithm proposed by the existing research work only a simple superposition of the existing data heterogeneity optimization method and the differential privacy optimization method, thereby causing a variety of problems such as poor portability, limited improvement effect, and dependence on public data sets. The present application proposes a differential privacy federated learning method and system for heterogeneous data, receives the local model parameters issued by the service end, calculates the information quantity for representing the correlation between the local model parameters and the client local data set, and performs hierarchical normalization processing to obtain the normalized information quantity. A mask matrix is constructed by comparing the size of the preset hyperparameter, and the element product of the mask matrix and the local model parameters is calculated to screen the local model parameters whose normalized information quantity is lower than the preset hyperparameter and construct a parameter set. A regularization term is introduced to suppress the update amplitude of the parameters in the parameter set to construct a loss function, and the local model parameters are iteratively updated to obtain intermediate local model parameters. Projection is performed on the feature subspace to decompose into low-dimensional vectors and redundant vectors, and after clipping and adding Gaussian noise, the original dimension is restored to obtain target model parameters. The feature subspace is obtained by multiplying the power method to extract the first predetermined number of principal component feature vectors after the service end calculates the gradient matrix of the federated model using the pre-trained generator to generate the global data set. The target model parameters are uploaded to the federated model of the service end to aggregate according to the set aggregation weight. Among them, the service end collects the target model parameters of each client and the parameters of the local discriminator pre-trained to identify the local data of the client, loads the parameters of the local discriminator to identify the global data, and according to the identification result, the client whose local data set distribution is closer to the global data distribution is allocated a higher aggregation weight.

[0069] Figure 1This is a flowchart illustrating a differential privacy federated learning method for heterogeneous data according to an embodiment of the present invention. Specifically, one aspect of the present invention provides a differential privacy federated learning method for heterogeneous data, which includes the following steps S101-S104:

[0070] Step S101: Receive the local model parameters sent by the server, calculate the amount of information used to represent the correlation between the local model parameters and the client's local dataset, and perform hierarchical normalization processing according to the level where the local model parameters are located to obtain the normalized information amount.

[0071] Step S102: Construct a mask matrix based on the comparison between the normalized information content and the preset hyperparameters, calculate the element-wise product of the mask matrix and the local model parameters, and filter out local model parameters whose normalized information content is lower than the preset hyperparameters and construct them as a parameter set.

[0072] Step S103: Introduce a regularization term to suppress the update magnitude of parameters in the parameter set to construct a loss function. Iterate and update the local model parameters until the preset iteration termination condition is reached to obtain intermediate local model parameters. Project the intermediate local model parameters onto the feature subspace to decompose them into low-dimensional vectors and redundant vectors. Prune the low-dimensional vectors and redundant vectors, add Gaussian noise, and restore them to the original dimensions to obtain the target model parameters. The feature subspace is obtained by the server using a pre-trained generator to generate a global dataset, calculate the gradient matrix of the federated model, and then extract a predetermined number of principal component feature vectors using the power method.

[0073] Step S104: Upload the target model parameters to the federated model on the server to aggregate according to the set aggregation weights; wherein, the process of setting aggregation weights includes: the server collects the target model parameters of each client and the parameters of the pre-trained local discriminator used to identify the local data of the client, loads the parameters of the local discriminator to identify the global data, and assigns higher aggregation weights to clients whose local dataset distribution is closer to the global data distribution according to the identification results.

[0074] In steps S101 and S102, federated learning includes a server and multiple clients. Each client contains a local discriminator and a local model, while the server contains a generator and a federated model. The client receives model parameters issued by the server in a certain round and uses them as local model parameters. The information content of the gradient information of the local dataset is quantified based on the local model parameters. The larger the information content of each local model parameter, the higher the sensitivity of the parameter to the local data and the stronger the correlation with the distribution of the local dataset. The local model adopts a neural network, which consists of multiple layers, including but not limited to input layer, hidden layer, output layer, activation layer, normalization layer and convolutional layer. Normalization is performed according to the layer in which the local model parameters are located.

[0075] In some embodiments, the expression for calculating the amount of information representing the correlation between local model parameters and the client's local dataset is:

[0076]

[0077] The expression for the normalized information content is obtained by performing hierarchical normalization according to the level where the local model parameters are located:

[0078] ;

[0079] in, This represents the local model parameters sent by the server. Indicates client Local dataset, Indicates client Upper One local model parameter, Indicates the client's first Layer The amount of information in each local model parameter.

[0080] Furthermore, the mask matrix and local model parameters Obtain the parameter set by performing element-wise product The expression is ,because The time mask matrix is ​​1. The time mask matrix is ​​0, therefore, local model parameters are retained when the normalized information content is less than or below the preset hyperparameters, and local model parameters are not retained when the normalized information content is higher than the preset hyperparameters; through the client... Layer Normalized information content and hyperparameters of each local model parameter By comparing, the client's first Layer A mask matrix for each local model parameter; in some embodiments, the expression for the mask matrix is:

[0081] ;

[0082] in, This indicates the preset hyperparameters. Indicates the client's first Layer The normalized information content of each local model parameter.

[0083] In step S103, a set of regularization constraint parameters is added to the regular cross-entropy loss function. The loss function for constructing the local model is expressed as follows:

[0084]

[0085] wherein, denotes the total number of local dataset samples, denotes the actual value of the th sample, denotes the predicted value of the th sample, denotes the parameter set updated after the model training.

[0086] Further, the preset iteration termination condition includes reaching a preset number of iterations and exhausting a privacy budget; the consumption of the privacy budget is tracked using Gaussian differential privacy, which is then converted into -DP; Gaussian differential privacy is a technique for achieving data privacy protection by adding Gaussian noise, and the privacy budget is an index used to quantify the degree of privacy protection in federated learning, including and the probability of privacy protection failure There is a certain privacy loss at each model update, and the cumulative privacy loss reaches the exhaustion of the privacy budget, at which point the local model parameter update is stopped.

[0087] In some embodiments, the feature subspace is a process obtained by multiplying the first predetermined number of principal component feature vectors using the power method after the server calculates the gradient matrix of the federated model using the pre-trained generator to generate the global data, which includes steps S1031-S1033:

[0088] Step S1031: generating global data according to the pre-trained generator, and constructing a training dataset after assigning random labels to the global data.

[0089] Step S1032: calculating the gradient of the federated model on the training dataset and combining all the gradients into a gradient matrix.

[0090] Step S1033: generating a random matrix and performing multiple iterations of the gradient matrix composed of the first predetermined number of feature vectors through the power method with the gradient matrix, and the first predetermined number of feature vectors form a feature subspace; the expression for multiple iterations of the power method is:

[0091] ;

[0092] ;

[0093] wherein, denotes the gradient matrix, denotes the random matrix.

[0094] Specifically, the server trains the generator to generate global data and assigns random labels to the global data to construct a training data set , the gradient of the federated model on the training data set is calculated, and the expression is: All gradients are combined into a gradient matrix , represents the dimension of the local model parameter, represents the number of gradients; first, according to calculate , then update the random matrix after , after multiple iterations, the matrix composed of the first eigenvectors is obtained, and a feature subspace is constructed therefrom.

[0095] Further, the intermediate local model parameters are projected on the feature subspace, the updated random matrix after multiple iterations is calculated according to the power method, and a low-dimensional vector with a dimension of and a redundant vector with a dimension of are obtained according to the rules of matrix multiplication. The low-dimensional vector and the redundant vector are clipped and added with noise to realize differential privacy protection, and then restored to the original dimension consistent with the intermediate local model parameters to obtain the target model parameters for subsequent federated aggregation. Clipping reduces the impact of a single update on the global model, thereby reducing the contribution of a single data point to model updates. Adding noise further confuses the data source by introducing randomness, making it difficult to infer any specific client's data from the model updates. In some embodiments, the intermediate local model parameters are projected on the feature subspace to decompose into a low-dimensional vector and a redundant vector. The expressions of the low-dimensional vector and the redundant vector are:

[0096] ;

[0097] ;

[0098] After clipping and adding Gaussian noise to the low-dimensional vector and the redundant vector, the expression of the target model parameters is:

[0099] ;

[0100] ;

[0101] ;

[0102] wherein, denotes the intermediate local model parameters, denotes the random matrix after multiple iterations of updates according to the power method, denotes a preset clipping coefficient, denotes a noise variance, denotes a low-dimensional vector dimension, denotes a redundant vector dimension, denotes a total number of clients.

[0103] In some embodiments, the pre-training process of the generator comprises steps S1-S2:

[0104] Step S1: input a randomly sampled noise into an initial generator, and output generated data.

[0105] Step S2: construct a generator loss function according to the generated data and the local data set, and perform parameter iterative updates on the initial generator to obtain a generator, with the objective of minimizing the generator loss function; the expression of the generator loss function is:

[0106]

[0107]

[0108] ;

[0109] ;

[0110] wherein, denotes generated data, denotes a label of the generated data, denotes a generator, and denotes a weight coefficient, denotes a total number of clients, denotes a class number to which the generated data belongs, denotes the th generated data input into a local model of a client with parameters , denotes a mean value of a feature map of the generator on the th batch normalization layer, denotes a mean value of the th batch normalization layer in the local model of the client , denotes a variance of a feature map of the generator on the th batch normalization layer, denotes a variance of the the variance of the th batch normalization layer in the local model, represents the calculation result of the th generation data input parameter for the local discriminator.

[0111] Specifically, to ensure that the data generated by the generator can have similar characteristics to the local data set, so that the local model of each client can predict a certain category with high probability; an additional term is introduced in the loss function to enhance the stability of the generator and better improve the class diversity and individual difference of the data in the generation process, by reducing the difference between the mean and variance of the feature map of the generator at the th batch normalization layer (BN) and the mean and variance of the local model batch normalization layer, the diversity of the generated data is improved; by reducing to improve the probability of the generated data being judged as true by the local discriminator.

[0112] In step S104, the server collects the target model parameters and the parameters of the local discriminator, and calculates the aggregation weight according to the following expression , which represents the data uniformly sampled from the global data , calculates the probability of belonging to the local data set of the client through the local discriminator, , represents the total number of clients, the higher the probability, the closer the global data is to the local data set, and the closer the client is to the higher aggregation weight; the target model parameters are weighted and summed according to the aggregation weight. In some embodiments, the pre-training process of the local discriminator includes steps S1041-S1043: Step S1041: Mark the samples in the local data set as positive samples, and mark the samples in the adversarial data set obtained after data transformation or adversarial sample generation as negative samples.

[0113] Step S1042: Input the samples in the local data set into the initial local discriminator to obtain a first result, and input the samples in the adversarial data set into the initial local discriminator to obtain a second result.

[0114] Step S1043: Construct a local discriminator loss function according to the difference between the first result and the second result, and update the parameters of the initial local discriminator to obtain the local discriminator; the expression of the local discriminator loss function is:

[0115]

[0116] ;

[0117] wherein, ​​denote a local dataset, denote an adversarial dataset, denote the i-th sample in the local dataset, denote the i-th sample in the adversarial dataset.

[0118] Further, in some embodiments, the method further comprises steps S100-S300:

[0119] Step S100: inputting the global data into the local model to output a plurality of client probability vectors representing the categories to which the data belongs; and performing weighted summation on the plurality of client probability vectors to obtain a client weighted probability vector.

[0120] Step S200: inputting the global data into the federated model to output a federated model probability vector representing the category to which the data belongs.

[0121] Step S300: constructing a federated model loss function according to the client weighted probability vector and the federated model probability vector, and iteratively updating the federated model parameters; wherein the expression of the federated model loss function is:

[0122] ;

[0123] ;

[0124] ;

[0125] wherein, denote the client weighted probability vector, denote the federated model probability vector output by the federated model when the i-th global data is input into the federated model with the parameter being denote the client probability vector output by the local model when the i-th global data is input into the local model with the parameter being denote the aggregation weight, denote the probability that the i-th global data belongs to the client local dataset calculated by the local discriminator, denote the total number of clients.

[0126] ​​​​​​​​​Specifically, first, the aggregation weight is calculated by the probability that the global data belongs to the local data set, when the global data is a picture and the federated model performs a picture classification task, a picture is input to the local model and the classification probability that the picture belongs to the corresponding category is output, the classification probabilities of the local models of multiple clients are weighted and summed according to the aggregation weight to obtain a client weighted probability vector, a picture is input to the federated model and a federated model probability vector that the picture belongs to the corresponding category is output; then the relative entropy divergence (KL divergence) of the client weighted probability vector and the federated model probability vector is calculated as a loss function to update the parameters of the federated model; in a new round of iteration update process, the federated model of the server sends parameters to the local model of the client.

[0127] In another aspect, the application also provides a differential privacy federated learning system for heterogeneous data, comprising a processor, a memory and computer programs / instructions stored on the memory, the processor being configured to execute the computer programs / instructions, and the system realizing the steps of the above method when the computer programs / instructions are executed.

[0128] In another aspect, the application also provides a computer readable storage medium having computer programs / instructions stored thereon, the programs / instructions being executed by the processor to realize the steps of the above method.

[0129] The application will be described below in conjunction with a specific embodiment:

[0130] Figure 2This is a schematic diagram of the differential privacy federated learning method for heterogeneous data according to an embodiment of the present invention. This application proposes a differential privacy federated learning method for heterogeneous data (UDPTH method). Addressing the issues of local model information loss due to pruning operations and the more severe loss of federated learning utility caused by using differential privacy mechanisms in heterogeneous data, the pruning and noise addition processes of differential privacy are optimized according to the characteristics of heterogeneous data, achieving differential privacy and improving the prediction accuracy of the federated model in heterogeneous data. This application designs an information-based local model constraint method, which dynamically constrains model parameters with weak correlation to local data by accurately calculating the correlation between local model parameters and the dataset, thereby reducing information loss caused by pruning operations. To address the issue of federated model iteration direction shift caused by uniform pruning, this application designs a data-generation-based federated aggregation method, which corrects the original federated model iteration direction by generating global data and redistributing the aggregation weights of each client model. To address the problem of random noise exacerbating heterogeneity among client models, this application designs a noise reduction method based on dimensionality transformation, which significantly reduces the amount of random noise added by decomposing the parameter update vector into low-dimensional and redundant vectors without losing local model information. This application consists of three parts: The first part filters model parameters using an information matrix before local model training begins, constructs a new loss function based on the filtering results, and trains the local model based on this loss function to reduce information loss caused by pruning operations; The second part constructs a loss function training generator on the server to generate a global dataset, calculates the contribution of each client using the global dataset and the local discriminator, and assigns corresponding aggregation weights accordingly to mitigate the shift in the update direction of the federated model; The third part calculates the feature subspace using the power method before adding random noise to the local model, projects the updated parameters onto this feature subspace to obtain low-dimensional vectors and corresponding redundant vectors, and prunes and adds noise to the low-dimensional vectors and redundant vectors respectively, thereby achieving the same level of privacy protection with less noise addition.

[0131] 1. The specific process of differential privacy-preserving federated learning for heterogeneous data:

[0132] 1) Each client Hold a local discriminator And a local model The server holds a generator. and federal model Client The local discriminator and local model are initialized with parameters, while the server initializes the generator and federated model with parameters.

[0133] 2) The client receives the server's data at the... Parameters of the federated model issued in round Each client calculates the parameters of each model in its local model relative to the local data. Gradient information is used to quantify the amount of information in the local model parameters. The local model employs a neural network; among which, Represents model parameters In local dataset The loss function value is obtained by analyzing the gradient information content. The size identifies parameters that are strongly correlated with local data: The larger the value, the higher the sensitivity of the parameter to local data and the stronger its correlation with the local data distribution; the expression for information content is:

[0134]

[0135] The calculated results are normalized according to the hierarchy of the model parameters. Layer The amount of information in each model parameter after hierarchical normalization The expression is:

[0136] ;

[0137] After completing the information content calculation, set a hyperparameter. And construct a mask matrix based on the amount of information. :

[0138]

[0139] Based on this mask matrix and model parameters Perform element-wise product This allows for the filtering of data with information content lower than the hyperparameter. parameter set This is the parameter set consisting of the parameter values ​​before the local model training began; a regularization term is added to the regular loss function to constrain the parameter set. The update scale is used as the loss function for the local model parameters, and the expression for the loss function is:

[0140] ;

[0141] in, This indicates the number of samples used for testing. This indicates the class number to which the sample belongs. For example, the MNIST dataset has 10 classes. , express This sample number The value of the class, representing the true label of the sample, can be set by human inspection, when the MNIST image is 1, then , other and are 0, Similarly, is a random number between 0 and 1 calculated by the model and used to represent the probability that the sample belongs to a certain class; represents the parameter set after the model training parameter value changes.

[0142] 3) Before uploading the parameter updates of each client model, the server will generate data using the trained generator and assign random labels to it to construct a new data set . Then calculate the federal model according to the following formula The gradient on the generated data set is , the expression is:

[0143] ;

[0144] Combine all the calculated gradients into the corresponding gradient matrix , where represents the dimension of the model parameters, represents the number of sample gradients.

[0145] Randomly initialize the matrix , and perform multiple iterations through the power method as follows to calculate the matrix composed of the first k eigenvectors of the gradient matrix , and the eigenvectors corresponding to the k largest eigenvalues are used as the basis of the new feature subspace, the expression is:

[0146] ;

[0147] ;

[0148] Then the local model parameters trained by the th client in the th round of updates are projected on the feature subspace according to the following formula, according to the rules of matrix multiplication, a low-dimensional vector with dimension and a redundant vector with dimension can be obtained, the expression is:

[0149] ;

[0150] ;

[0151] To achieve strict differential privacy protection, the low-dimensional vector and the corresponding redundant vector calculated above are clipped and added with Gaussian noise according to the following formula:

[0152] ;

[0153] ;

[0154] where the two-norm of the low-dimensional vector and the redundant vector is threshold clipped to constrain their sensitivity, and the expression is:

[0155] ;

[0156] ;

[0157] ;

[0158] The noise used for model update disturbance is the modified Gaussian noise. After using this noise for disturbance, differential privacy protection can be provided for the gradient, and the consumption of the privacy budget is tracked using Gaussian differential privacy, which is then converted to .

[0159] Restore the and after completing the differential privacy protection to the original dimension, which facilitates the subsequent federated aggregation process, and the expression is:

[0160] ;

[0161] 4) The server collects the local model adoption numbers and discriminator parameters uploaded by each client, and calculates the KL divergence between the client weighted probability vector and the federated model probability vector according to the aggregation method shown in the following formula to update the parameters of the federated model, and uses the global data and the local discriminator to redistribute the aggregation weights for each client model, ensuring that the total weight allocated to the client model that is closer to the global data distribution is larger, thereby improving the contribution of the client model with rich data categories and balanced distribution to the federated model, and correcting the update direction of the federated model. When performing the picture classification task, input a picture to each model and output a probability vector representing the category to which the corresponding model predicts the picture belongs; the client weighted probability vector is obtained by weighting and summing the multiple probability vectors output by the local model after inputting the global data; the federated model probability vector is the probability vector output by the federated model after inputting the global data ; the aggregation weight is obtained by the output of each discriminator on the global data , where the expression is:

[0162] ;

[0163] The client weighted probability vector is obtained by weighted sum of the plurality of probability vectors output by the local model according to the aggregation weights, and the expression is:

[0164]

[0165] The KL divergence between the client weighted probability vector and the federated model probability vector is calculated to update the parameters of the federated model, and the expression is:

[0166] ;

[0167] wherein, represents a data sample obtained by uniformly sampling from the global data, represents a discriminator maintained by each client, used to calculate the probability that the data sample belongs to the local data set of the current client, represents all client numbers.

[0168] 5) After the server aggregates the local model parameters according to the aggregation weights, the federated model parameters are issued to the server for the next round of iteration update until the specified iteration number is reached or the privacy budget is exhausted.

[0169] 2. Training process of local discriminator and generator.

[0170] To train the local discriminator model, the client constructs an adversarial data set by data transformation and adversarial sample generation , and marks the samples in the adversarial data set as negative samples, and marks the samples in the local data set as positive samples. The local data set is held by each client in the scene and can be directly accessed. A loss function is constructed to train the local discriminator , and the expression is:

[0171]

[0172] Then, the server constructs a loss function to train the generator according to the following formula, thereby optimizing the global generated data set. Wherein, , represents the generated data obtained after the randomly sampled noise passes through the generator .

[0173] ;

[0174] The item is used to ensure that the data generated by the generator can have similar characteristics to the local data set, so as to be predicted as a certain category by each client model with a high probability, The expression of the item is:

[0175] ;

[0176] Further, only relying on The item for training the generator model will cause the model to tend to generate data of the same category or generate exactly the same data for a certain category, limiting the diversity and generalization ability of the model. Therefore, the method introduces an additional The item in the loss function, aiming to enhance the stability of the generator model, so as to better improve the category diversity and individual difference of the data in the generation process, by reducing the difference between the mean And variance Of the feature map of the generator on the first Batch normalization layer (BN, Batch Normalization) and the mean And variance Of the batch normalization layer corresponding to the client model, The expression of the item is:

[0177] ;

[0178] The method of the application is the same as the traditional generative adversarial network model training method, which improves the probability that the data generated by the model is judged as true by the discriminators of each client model by reducing The expression of the item is:

[0179]

[0180] 3. When the heterogeneous data-oriented differential privacy federated learning method is applied in the medical field, the local data set of each client includes a plurality of medical image data, comprising:

[0181] Receiving the local model parameters issued by the server, calculating the information quantity for representing the correlation between the local model parameters and the medical image data set of the client, and performing hierarchical normalization processing according to the layer where the local model parameters are located to obtain normalized information quantity; the medical image data set includes but is not limited to computed tomography data, magnetic resonance imaging data, ultrasonic imaging data and thermal imaging data.

[0182] According to the size comparison relationship between the normalized information quantity and the preset hyperparameter, a mask matrix is constructed, and the element product of the mask matrix and the local model parameters is calculated to screen the local model parameters with normalized information quantity lower than the preset hyperparameter and construct a parameter set.​

[0183] The regularization term is introduced to suppress the update amplitude of the parameters in the parameter set to construct a loss function, and the local model parameters are iteratively updated until a preset iteration termination condition is reached to obtain intermediate local model parameters; the intermediate local model parameters are projected on the feature subspace to be decomposed into a low-dimensional vector and a redundant vector, and the low-dimensional vector and the redundant vector are clipped and added with Gaussian noise to restore to the original dimension to obtain target model parameters; the feature subspace is obtained by the server using the pre-trained generator to generate the gradient matrix of the global model, and then extracting the first preset number of principal component feature vectors by the power method.

[0184] The target model parameters are uploaded to the federated model of the server to be aggregated according to the set aggregation weight; wherein the process of setting the aggregation weight includes: the server collects the target model parameters of each client and the parameters of the pre-trained local discriminator used to identify the local data of the client, loads the parameters of the local discriminator to identify the global data, and according to the identification result, the closer the distribution of the medical image data set is to the distribution of the global data, the higher the aggregation weight of the client.

[0185] Further, when the local model and the federated model perform a classification task, the medical image data is input into the local model, and the class to which the disease corresponding to the medical image data belongs is output; the randomly sampled noise is input into the generator to output global medical image data, and the global medical image data is input into the federated model, and the class to which the disease corresponding to the medical image data belongs is output.

[0186] 4. Comparison of experimental effects of the present application and other methods.

[0187] In the comparison, widely used real data sets MNIST data set, Fashion-MNIST data set and CIFAR-10 data set are used, wherein the MNIST data set is from the National Institute of Standards and Technology, USA, and there are 70,000 picture data, of which 60,000 are used for training and 10,000 are used for testing. Each picture is a 28 by 28 pixel 0-9 handwritten number picture; the Fashion-MNIST data set is used instead of the classic MNIST data set, which contains 10 classes of clothing items in grayscale images, of which 60,000 pictures are used as a training set and 10,000 pictures are used as a test set; the CIFAR-10 data set is mainly used for universal object recognition, containing 60,000 32 by 32 RGB color pictures, a total of 10 categories, of which 50,000 are used for training and 10,000 are used for testing.

[0188] In the experimental parameter part, the total number of clients is set to 100 in the experiment in this section, the sampling probability Simple Rate = 0.1, the learning rate in the local model iteration process is searched in the range of {0.001, 0.003,..., 0.1, 0.3}, the batch size Batch Size = 50, the local update round Local Epoch = 10, and the global update round Global Epoch = 300. The hyperparameters required by the method and are searched in the range of {0.05, 0.1, 0.3, 0.5} by grid search; the parameters required by the method and are set to 1 and 0.5 respectively, the learning rate of the generator model is 0.01, the training round is 50, the federated model is trained by the way of SGD, the learning rate is 0.01, and the iteration number is fixed to 30; the clipping threshold in the method and are set to 2 and 0.5 respectively, the number of generated data set samples m = 2000, and the number of feature vectors k = 1000.

[0189] In the comparative method part, the method designed in this application is compared with three kinds of baseline methods; the differential privacy federated average algorithm (DP-FedAvg) directly combines the original federated average algorithm with the user-level differential privacy technology, which is one of the most commonly used baseline methods; the differential privacy federated learning algorithm enhanced by personalized sharpness perception minimization (DP-FedSAM) integrates the sharpness perception minimization (SAM) optimizer in the process of local model training to reduce the negative impact of differential privacy, so that the local model has a smoother loss landscape, thereby ensuring the robustness of the model to differential privacy noise; the BLUR+LUS algorithm mainly improves the utility of the federated learning algorithm under differential privacy in the following two ways: first, by adding a regularization term to the loss function of model training, the local model is updated smoothly, thereby reducing the utility loss caused by clipping; second, by sparsifying the parameters of the trained local model, the amount of Gaussian noise added is reduced.

[0190] The accuracy of the method of the present application and the existing method is compared below to illustrate the effect and performance of the method of the present application.

[0191] Table 1 is the accuracy comparison result of different methods under the condition of reducing the degree of data heterogeneity. Table 2 is the accuracy comparison result of different methods under the condition of increasing the privacy budget. On the one hand, as the privacy budget With the increase of privacy budget, the performance of all methods is improved, because the greater the privacy budget, the less the degree of noise introduction, which is more conducive to the convergence of the model; on the other hand, with the decrease of the degree of data heterogeneity, α gradually increases, and the utility of each method also has a significant improvement, which is consistent with the expectation of existing theoretical analysis; at the same time, the performance of the algorithm (UDPTH) proposed in the present application is better than that of the comparative algorithm in various cases, and it can also perform well when the privacy budget is small or the degree of data heterogeneity is high. The reasons for this include: (1) The personalized sensitivity perception minimization enhanced differential privacy federated learning algorithm only processes the influence of noise in the differential privacy mechanism, without considering the information loss caused by the deviation of the federated model update direction due to the differential privacy clipping operation under heterogeneous data, and without reducing the amount of random noise added in the model training process. Only the robustness of the model to noise is improved; (2) Although the BLUR+LUS method considers the influence of noise and clipping operation at the same time, on the one hand, its regularization constraint is too strict, which leads to that the client model cannot provide effective information about the local data to the server, on the other hand, the sparsification process also causes a certain degree of information loss of the local model, which ultimately leads to that the method has limited improvement compared with the original differential privacy federated average algorithm, and the algorithm proposed in the present application judges the relevance of the parameter and the local data when performing parameter constraint, so it can still ensure that the local model is fully trained, and when processing random noise, the redundant vector is retained as an unbiased estimate of the original model parameter, reducing the information loss of the local model. Therefore, compared with the original differential privacy federated average algorithm, it has a more obvious improvement effect.

[0192] Table 1 Accuracy comparison results of different methods under the condition of reducing the degree of data heterogeneity

[0193]

[0194] Table 2 Accuracy comparison results of different methods under the condition of increasing the privacy budget

[0195]

[0196] It can be seen from the experimental results that, compared with the original differential privacy federated average algorithm, the various methods designed by the application can improve the utility of the federated model to a certain extent. In addition, the noise reduction method based on dimension transformation has a more significant optimization effect when the privacy budget is small, but as the privacy budget increases, the optimization effect of the differential privacy federated average algorithm gradually decreases. The reasons include but are not limited to the fact that as the privacy budget increases, the amount of random noise introduced in the model training process gradually decreases, so the improvement effect of dimension transformation gradually decreases. As the privacy budget gradually increases, the improvement effect of the federated aggregation method based on data generation on the original differential privacy federated average algorithm gradually increases. The reason for this is that as the amount of noise decreases, the data features generated by the server are closer to the real data features, so using these generated data to train the federated model can better correct the model update direction deviation problem caused by the differential privacy clipping operation.

[0197] To sum up, the application provides a differential privacy federated learning method and system for heterogeneous data, receives the local model parameters issued by the server, calculates the information quantity representing the correlation between the local model parameters and the client local data set, and performs hierarchical normalization processing to obtain the normalized information quantity. A mask matrix is constructed by comparing the preset hyperparameters, and the element product of the mask matrix and the local model parameters is calculated to screen the local model parameters with normalized information quantity lower than the preset hyperparameters and construct a parameter set. A regularization term is introduced to suppress the update amplitude of the parameters in the parameter set to construct a loss function. The local model parameters are iteratively updated until the preset iteration termination condition is reached to obtain intermediate local model parameters. Projection is performed on the feature subspace to decompose into low-dimensional vectors and redundant vectors. After clipping and adding Gaussian noise, the original dimension is restored to obtain the target model parameters. The feature subspace is obtained by multiplying the power method to extract the first preset number of principal component feature vectors after the server calculates the gradient matrix of the federated model using the pre-trained generator to generate the global data set. The target model parameters are uploaded to the federated model of the server to aggregate according to the set aggregation weight. The process of setting the aggregation weight includes: the server collects the target model parameters of each client and the parameters of the pre-trained local discriminator used to identify the client local data, loads the parameters of the local discriminator to identify the global data, and according to the identification result, the client whose local data set distribution is closer to the global data distribution is allocated a higher aggregation weight.

[0198] The application also provides a computer device, which can include a processor, a memory,

[0199] The processor and the memory can be connected by a bus or other means.

[0200] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, or a combination thereof.

[0201] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the key shielding method of the vehicle display device in the embodiments of the present application. The processor executes various functions and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory.

[0202] The memory can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0203] The one or more modules are stored in the memory, and when executed by the processor, perform the method described in the embodiments.

[0204] Corresponding to the above method, the present application also provides a differential privacy federated learning system for heterogeneous data, which comprises a computer device, the computer device comprises a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the system realizes the steps of the method as described above.

[0205] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the edge computing server deployment method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0206] Those skilled in the art should understand that the exemplary components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software or a combination thereof. The decision to implement in hardware or software depends on the particular application and design constraints imposed on the technical solution. Skilled persons can use different methods to implement the described functions for each particular application, but such implementation should not be considered beyond the scope of the present application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link.

[0207] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and skilled persons can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present application.

[0208] In the present application, the features described and / or illustrated for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0209] The above description is only preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A differential privacy-preserving federated learning method for heterogeneous data, characterized in that, The method includes the following steps: Receive local model parameters sent by the server, calculate the amount of information used to represent the correlation between the local model parameters and the client's local dataset, and perform hierarchical normalization processing according to the level where the local model parameters are located to obtain normalized information. A mask matrix is ​​constructed based on the comparison between the normalized information content and the preset hyperparameters. The element-wise product of the mask matrix and the local model parameters is calculated to filter out local model parameters whose normalized information content is lower than the preset hyperparameters and construct them as a parameter set. A regularization term is introduced to suppress the update magnitude of the parameters in the parameter set to construct a loss function. The local model parameters are iteratively updated until a preset iteration termination condition is reached to obtain intermediate local model parameters. The intermediate local model parameters are projected onto the feature subspace to decompose them into low-dimensional vectors and redundant vectors. The low-dimensional vectors and redundant vectors are pruned and Gaussian noise is added to restore them to the original dimensions to obtain the target model parameters. The feature subspace is obtained by the server using a pre-trained generator to generate a global dataset, calculate the gradient matrix of the federated model, and then extract a predetermined number of principal component feature vectors using the power method. The target model parameters are uploaded to the federated model on the server for aggregation according to the set aggregation weights; wherein, the process of setting the aggregation weights includes: the server collects the target model parameters of each client and the parameters of a pre-trained local discriminator for recognizing local client data, loads the parameters of the local discriminator to recognize the global data, and assigns higher aggregation weights to clients whose local dataset distribution is closer to the global data distribution based on the recognition results; The method further includes: After inputting the global data into the local model, multiple client probability vectors representing the categories to which the data belongs are output; the multiple client probability vectors are weighted and summed to obtain a client weighted probability vector; After inputting the global data into the federated model, a federated model probability vector representing the category to which the data belongs is output. A federated model loss function is constructed based on the client-weighted probability vector and the federated model probability vector, and the federated model parameters are iteratively updated; wherein, the expression of the federated model loss function is: ； ； ； in, This represents the client-side weighted probability vector. Indicates the first Global data Input parameters are The federated model probability vector output after the federated model is obtained. Indicates the first Global data Input parameters are The client probability vector output after the local model. Indicates aggregate weight, Indicates the first Each global data point is used by a local discriminator to calculate the probability that it belongs to the client's local dataset. This indicates the total number of clients.

2. The differential privacy federated learning method for heterogeneous data according to claim 1, characterized in that, The expression for calculating the amount of information used to represent the correlation between the local model parameters and the client's local dataset is as follows: The expression for the normalized information content is obtained by performing hierarchical normalization processing according to the level where the local model parameters are located: ； in, This represents the local model parameters sent by the server. Indicates the client Local dataset, Indicates the client Upper One local model parameter, Indicates the client's first Layer The amount of information in each local model parameter.

3. The differential privacy federated learning method for heterogeneous data according to claim 1, characterized in that, The expression for the mask matrix is: ； in, This refers to the preset hyperparameters. Indicates the client's first Layer The amount of normalized information in each local model parameter.

4. The differential privacy federated learning method for heterogeneous data according to claim 1, characterized in that, The pre-training process of the local discriminator includes: The samples in the local dataset are labeled as positive samples, and the samples in the adversarial dataset obtained after data transformation or adversarial sample generation are labeled as negative samples. The samples in the local dataset are input into the initial local discriminator to obtain a first result, and the samples in the adversarial dataset are input into the initial local discriminator to obtain a second result; A local discriminator loss function is constructed based on the difference between the first result and the second result. The initial local discriminator is iteratively updated with parameters to minimize this loss function, resulting in the final local discriminator. The expression for the local discriminator loss function is: ； in, This refers to the local dataset. This refers to the adversarial dataset. Represents the first in the local dataset One sample, Represents the first in the adversarial dataset One sample.

5. The differential privacy federated learning method for heterogeneous data according to claim 1, characterized in that, The pre-training process of the generator includes: The randomly sampled noise is input into the initial generator, and the generated data is output. A generator loss function is constructed based on the generated data and the local dataset. The generator is obtained by iteratively updating the parameters of the initial generator with the objective of minimizing the generator loss function. The expression for the generator loss function is: ； ； in, This refers to the generated data. The labels representing the generated data, This refers to the generator. and Indicates the weighting coefficient. Indicates the total number of clients. This indicates the category number to which the generated data belongs. Indicates the first Generated data Input client The parameters are The calculation results of the local model, Indicates the generator at the 1st... The mean of the feature maps on each batch normalized layer Indicates the client In the local model, the first The mean of each batch of normalized layers Indicates the generator at the 1st... The variance of feature maps on each batch normalized layer Indicates the client In the local model, the first The variance of each batch of normalized layers Indicates the first The input parameters for generating data are: The calculation results of the local discriminator.

6. The differential privacy federated learning method for heterogeneous data according to claim 1, characterized in that, The feature subspace is obtained by the server using global data generated by a pre-trained generator to calculate the gradient matrix of the federated model, and then extracting a predetermined number of principal component feature vectors through the power method. The process includes: Global data is generated based on a pre-trained generator, and then a training dataset is constructed by assigning random labels to the global data. Calculate the gradient of the federated model on the training dataset and combine all gradients into a gradient matrix; A random matrix is ​​generated, and the gradient matrix is ​​iteratively multiplied multiple times using the power method to calculate the matrix composed of the first preset number of eigenvectors of the gradient matrix. These first preset number of eigenvectors then form an eigenspace. The expression for the multiple iterations using the power method is: ； ； in, The gradient matrix is ​​represented as follows. Let represent the random matrix.

7. The differential privacy federated learning method for heterogeneous data according to claim 6, characterized in that, The intermediate local model parameters are projected onto the feature subspace to decompose them into low-dimensional vectors and redundant vectors; the low-dimensional vectors and the redundant vector The expression is: ； ； The expression for pruning the low-dimensional vector and the redundant vector, adding Gaussian noise, and then merging them into the target model parameters is as follows: ； ； ； in, This represents the intermediate local model parameters. Denotes the random matrix, This indicates the preset clipping factor. Indicates the noise variance. Represents the dimension of a low-dimensional vector. Represents the dimension of the redundant vector. This indicates the total number of clients.

8. A differential privacy federated learning system for heterogeneous data, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

Images