A privacy protection heterogeneous federated learning method and system based on image translation

Abstract

The application discloses a privacy protection heterogeneous federated learning method and system based on image translation, which are corresponding solutions, in the solutions: a generated model with local user data semantic retention capability is optimized through an image translation server, a local user performs data enhancement on a local heterogeneous data set by using the generated model, so that the local data presents a state of uniform distribution of class labels, and a classification model of the local user is trained by using transformed data and enhanced data, so that the global model accuracy and the privacy protection capability are balanced. Moreover, there is no interaction between the image translation server and the data classification server, and the image translation server is in a trusted computing environment, so that the privacy protection capability is ensured.

Description

Technical Field

[0001] This invention relates to the field of privacy protection technology, and in particular to a privacy-preserving heterogeneous federated learning method and system based on image translation. Background Technology

[0002] Because the data collection environments of local users differ in federated learning, local user data is in a non-independent and identically distributed state. Performing heterogeneous federated learning under such a non-independent and identically distributed state will limit the accuracy of the global model.

[0003] Existing heterogeneous federated learning methods typically use knowledge distillation techniques to align the data feature spaces of different local users, but lack attention to data privacy protection capabilities.

[0004] In view of this, the present invention is hereby proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a privacy-preserving heterogeneous federated learning method and system based on image translation, which can balance the accuracy of the global model and the ability to protect privacy.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A privacy-preserving heterogeneous federated learning method based on image translation includes:

[0008] Step 11: The local user shares the network structure of the generative model and the discriminative model with the image translation server, and initializes the parameters of the generative model and the discriminative model;

[0009] Step 12: Local users train generative and discriminative models based on local heterogeneous datasets, and upload the parameters of the generative and discriminative models as shared parameters to the image translation server.

[0010] Step 13: The image translation server aggregates the parameters of the generative model and the discriminative model uploaded by local users and performs an image translation server update.

[0011] Step 14: The local user downloads the updated generative model and discriminative model from the image translation server. Steps 12 and 13 are repeated until the image translation server reaches the required number of updates, thus completing the training process of the generative model and discriminative model.

[0012] Step 15: Local users use the generative model to perform data augmentation on the local heterogeneous dataset, so that the local data presents a state of uniform distribution of class labels, and obtains a local uniformly distributed dataset.

[0013] Step 16: The local user shares the network structure of the classification model with the data classification server and initializes the parameters of the classification model;

[0014] Step 17: Local users train a classification model using a local uniformly distributed dataset and upload the parameters of the classification model as shared parameters to the data classification server.

[0015] Step 18: The data classification server aggregates the parameters of the classification models uploaded by local users and performs a data classification server update.

[0016] Step 19: The local user downloads the updated classification model from the data classification server. Steps 17 and 18 are repeated until the data classification server reaches the required number of updates, thus completing the training process of the data classification server.

[0017] A privacy-preserving heterogeneous federated learning system based on image translation includes: local users, an image translation server, and a data classification server, for executing the aforementioned method.

[0018] As can be seen from the technical solution provided by this invention, this invention optimizes a generative model with the ability to preserve the semantics of local user data through an image translation server. Local users use the generative model to perform data augmentation on local heterogeneous datasets, resulting in a uniform distribution of category labels in the local data. The transformed and augmented data are then used to train the local user's classification model, thereby balancing global model accuracy and privacy protection. Furthermore, there is no interaction between the image translation server and the data classification server, and the image translation server operates in a trusted computing environment, ensuring privacy protection. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a privacy-preserving heterogeneous federated learning method based on image translation, provided as an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the network structure of the generative model and the discriminative model provided in the embodiments of the present invention;

[0022] Figure 3 A schematic diagram of the network structure of the classification model provided in this embodiment of the invention;

[0023] Figure 4 This is a schematic diagram of the image translation results of the original data in the local heterogeneous dataset provided in an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of reconstructed data when resisting gradient-inverse-based data reconstruction attacks provided in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of a privacy-preserving heterogeneous federated learning system based on image translation, provided as an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0027] First, the following explanations are provided for the terms that may be used in this article:

[0028] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0029] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0030] The following provides a detailed description of a privacy-preserving heterogeneous federated learning method based on image translation provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0031] Example 1

[0032] This invention proposes a privacy-preserving heterogeneous federated learning method based on image translation, aiming to address the problem of local user data privacy leakage in heterogeneous federated learning. In this method, local users perform data augmentation on local heterogeneous datasets using a generative model, resulting in a uniform distribution of class labels in the local data. The transformed and augmented data are then used to train the local user's classification model, thereby balancing the accuracy of the global model with privacy protection capabilities. Figure 1 As shown, the method mainly includes the following steps:

[0033] Step 11: The local user shares the network structure of the generative model and the discriminative model with the image translation server and initializes the parameters of the generative model and the discriminative model.

[0034] In this embodiment of the invention, a total of K local users are defined. Participating in federated learning, each local user Having n k Original data x k,i and corresponding category label y k,i Local heterogeneous dataset x k ,1≤k≤K, and the total data size

[0035] Since the local user data exhibits a non-independent and identically distributed pattern, without loss of generality, for the k-th local user, n k (c i )≠n k (c j ), i≠j, where n k (c i ) for local heterogeneous dataset x k The category label is c i The number of data points whose true category is the i-th class, n k (c j ) for local heterogeneous dataset x k The category label is c j The number of data points.

[0036] Furthermore, each local user possesses two structurally identical generative models and two structurally identical discriminative models. For the k-th local user, the two structurally identical generative models they possess are denoted as G. A,k and G B,k Let D be the discriminant model that has the same structure. A,k and D B,k .

[0037] like Figure 2As shown, in both the generative and discriminative models, the input data is a color image of size 3×32×32; the output data is also a color image of size 3×32×32. `Conv(in_channels, out_channels, kernel_size, stride, padding)` is a 2D convolution operation with input channels `in_channels`, output channels `out_channels`, kernel size `kernel_size`, stride `stride`, and padding `padding`. `BN` is a batch normalization operation. `ConvTranspose2d(in_channels, out_channels, kernel_size, stride, padding)` is a convolution operation with input channels `in_channels`, `out_channels`, `kernel_size`, stride `padding`, and padding `out_channels`. The output is a 2D transposed convolution with output channels of `out_channels`, kernel size of `kernel_size`, stride of `stride`, and padding of `padding`. `Flatten` is the data feature flattening function. `Linear(in_features, out_features)` is a linear layer operation with input dimension `in_features` and output dimension `out_features`. ReLU, Tanh, LeakyReLU, and Sigmoid are activation functions in deep learning. The dimension of the discriminant signal is 1 (a higher value indicates a higher degree of confidence in the authenticity of the input data, and vice versa). The dimension of the logical output is 10 (each dimension corresponds to the probability of a class), and the example provided here has 10 classes.

[0038] Step 12: Local users train generative and discriminative models based on local heterogeneous datasets, and upload the parameters of generative and discriminative models as shared parameters to the image translation server.

[0039] The preferred implementation method for this step is as follows:

[0040] (1) The kth local user on the local heterogeneous dataset x k The original data in the range 1 ≤ k ≤ K is subjected to block partitioning and random block scrambling operations to obtain the target dataset.

[0041] (2) The adversarial loss defined by the k-th local user can be expressed as:

[0042]

[0043] in, For the mathematical expectation, x k,i Represents a local heterogeneous dataset x k The original data in Represents the target dataset The target data in the middle.

[0044] (3) The circular consistency loss defined by the kth local user can be expressed as:

[0045]

[0046] Among them, G A,k (x k,i ) represents transformed data, G B,k (G A,k (x k,i )) indicates reconstructing the original data. This represents the translation data of the target data. This indicates the target data to be reconstructed.

[0047] (4) The classification loss defined by the k-th local user can be expressed as:

[0048]

[0049] Where, n k Represents a local heterogeneous dataset x k The original number of data points in the dataset, where c represents the local heterogeneous dataset x. k The number of category labels in y k Represents a local heterogeneous dataset x k The set of true category labels in the original data. y represents the logical output obtained by transforming data through linear layer operations in the discriminant model. i (j) represents the transformed data G A,k (x k,i ) corresponds to the true category label of the j-th class. Represents the transformed data G A,k (x k,i This corresponds to the logical output of the j-th class.

[0050] (5) The translation loss of the k-th local user-defined image can be expressed as:

[0051]

[0052] (6) The k-th local user obtains the generative model parameters as shared parameters during the t-th round of image translation server updates by minimizing the image translation loss. and discriminant model parameters

[0053] Step 13: The image translation server aggregates the parameters of the generative model and the discriminative model uploaded by local users and performs an image translation server update.

[0054] The parameter aggregation process between the generative model and the discriminative model can be represented as follows:

[0055]

[0056] Where ← is the assignment symbol. To generate the model parameters when the image translation server completes its update in round t, The parameters of the model are determined when the image translation server finishes updating in round t.

[0057] Step 14: The local user downloads the updated generative and discriminative models from the image translation server. Steps 12 and 13 are repeated until the image translation server reaches the required number of updates, thus completing the training process of the generative and discriminative models.

[0058] The image translation server is set to update round number T1, the specific value of which can be set by those skilled in the art based on actual conditions or experience. The generated model after training is... The discriminant model after training is

[0059] Step 15: Local users use the generative model to perform data augmentation on the local heterogeneous dataset, so that the local data presents a state of uniform distribution of category labels, resulting in a local uniformly distributed dataset.

[0060] The preferred implementation method for this step is as follows:

[0061] (1) The kth local user calculates the category label with the largest number of entries in the local heterogeneous dataset. And calculate for category label c i Number of data points to be supplemented

[0062] (2) The k-th local user uses the trained generative model For local heterogeneous datasets x k Perform image translation to obtain transformed data. And use random flipping and data cropping for category label c i m k (c i ) transform data Perform data augmentation while preserving the category labels to supplement the category labels as c. i The data makes it consistent with the category label c i The corresponding number of data is m k (c i The calculation method for ) can be expressed as:

[0063]

[0064] Where min is the value of n k (c i )and The minimum value operator.

[0065] (3) The k-th local user is constructed from n k Transformed data and A local uniformly distributed dataset composed of augmented data points, presenting a uniform distribution of category labels.

[0066] The number of data points in the locally uniformly distributed dataset is and

[0067] Step 16: The local user shares the network structure of the classification model with the data classification server and initializes the parameters of the classification model.

[0068] The network structure of the classification model is as follows: Figure 3 As shown. In the classification model, the parameters of the classification model possessed by the k-th local user are... The input data is a 3×32×32 color image; the logical output dimension is 10; Conv(in_channels,out_channels,kernel_size,stride,padding) is a 2D convolution operation with in_channels input, out_channels output, kernel size of kernel_size, stride of stride, and padding; BN is batch normalization; Flatten is the data feature flattening function; Linear(in_features,out_features) is a linear layer operation with in_features input and out_features output; LeakyReLU is the activation function in deep learning.

[0069] Step 17: Local users train a classification model using their local uniformly distributed dataset and upload the model's parameters as shared parameters to the data classification server.

[0070] The preferred implementation method for this step is as follows:

[0071] (1) The kth local user uses classification model C k Calculate the category label y k,i Data Logical output (Probability distribution generated by the Softmax function), classification loss It can be represented as:

[0072]

[0073] Let be the mathematical expectation, where, Given a locally uniformly distributed dataset for the k-th local user, This represents the cross-entropy loss.

[0074] (2) The gradient of the classification model uploaded by the kth local user as a shared parameter to the global model server can be expressed as:

[0075]

[0076] Where ← is the assignment symbol. To classify model C when the q-th round of training is completed k The parameters, For the classification model C, when the training of the (q-1)th round is completed k The parameter, when q=1, The initialization parameters in step 16, For classification model C k The gradient of the parameters, where η is the learning rate.

[0077] Step 18: The data classification server aggregates the parameters of the classification models uploaded by local users and performs a data classification server update.

[0078] The parameter aggregation process for classification models uploaded by local users can be represented as follows:

[0079]

[0080] Where ← is the assignment symbol. Let the parameters of the classification model be the values ​​of the data classification server when the data classification server finishes updating in round t. For a locally uniformly distributed dataset The amount of data, n * Let K be the data volume of a locally uniformly distributed dataset with K local users.

[0081] Step 19: The local user downloads the updated classification model from the data classification server. Steps 17 and 18 are repeated until the data classification server reaches the required number of updates, thus completing the training process of the data classification server.

[0082] In this embodiment of the invention, the number of data classification server update rounds is set to T2, and its specific value can be set by those skilled in the art based on actual conditions or experience.

[0083] It should be noted that the above steps only illustrate the processing flow for a single local user. The processing flow for other local users is the same, so it will not be repeated here.

[0084] The solution provided in this invention optimizes a generative model with local user data semantic preservation capabilities through an image translation server. Local users use the generative model to perform data augmentation on local heterogeneous datasets, resulting in a uniform distribution of category labels in the local data. The transformed and augmented data are then used to train the local user's classification model, thereby balancing global model accuracy and privacy protection. Furthermore, there is no interaction between the image translation server and the data classification server, and the image translation server operates in a trusted computing environment, ensuring privacy protection.

[0085] To test the global model accuracy and privacy protection capability of the proposed privacy-preserving heterogeneous federated learning method based on image translation, this invention compares the proposed method with federated learning method A, where A is a heterogeneous federated learning method based on knowledge distillation.

[0086] To test the global model accuracy and privacy protection capabilities, the CIFAR-10 dataset was used, with stochastic gradient descent as the basic model optimization algorithm. The number of local users was set to 5, the update rounds for both the image translation server and the data classification server were set to 5, the local model training rounds were set to 10, the random sampling dataset size was set to 64, and the learning rate was set to 0.01. To test the global model accuracy, local users used a locally uniformly distributed dataset and the trained classification model to test classification accuracy. To test privacy protection capabilities, the data classification server used a gradient-inverse data reconstruction attack to steal local user data, and the peak signal-to-noise ratio (PSNR) of the reconstructed data, calculated using the original data as a reference, was used to measure the privacy protection capability of the corresponding method.

[0087] Table 1. Experimental results comparing the accuracy of the global model under different data heterogeneity factors.

[0088]

[0089] Table 1 shows the experimental results comparing the global model accuracy under different data heterogeneity factors. The data heterogeneity factor is a parameter in the Dirichlet function that controls the degree of non-independent and identically distributed (i.i.d.) distribution; a higher value indicates a more significant degree of non-independent and identically distributed distribution. As shown in Table 1, the privacy-preserving heterogeneous federated learning method based on image translation proposed in this invention achieves the highest global model accuracy.

[0090] Figure 4 This displays the image translation results from the original data in a local heterogeneous dataset. Figure 4It can be seen that the transformed data has a lower peak signal-to-noise ratio compared to the original data, thus possessing privacy protection capabilities. Figure 4 In the diagram, subgraphs (a), (b), and (c) correspond to the original data, the transformed data, and the reconstructed original data, respectively.

[0091] Figure 5 This demonstrates the reconstructed data when a privacy-preserving heterogeneous federated learning method based on image translation resists gradient-inverse data reconstruction attacks. Figure 5 It can be seen that the reconstructed data using the privacy-preserving heterogeneous federated learning method based on image translation proposed in this invention has a low peak signal-to-noise ratio and high privacy protection capability when resisting gradient-inverse data reconstruction attacks.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0093] Example 2

[0094] This invention also provides a privacy-preserving heterogeneous federated learning system based on image translation, such as... Figure 6 As shown, it mainly includes: a local user, an image translation server, and a data classification server, used to execute the methods provided in the aforementioned embodiments. Considering that the main technical details involved in this system have already been described in detail in the previous embodiments, they will not be repeated here.

[0095] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0096] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A privacy-preserving heterogeneous federated learning method based on image translation, characterized in that, include: Step 11: The local user shares the network structure of the generative model and the discriminative model with the image translation server, and initializes the parameters of the generative model and the discriminative model; Step 12: Local users train generative and discriminative models based on local heterogeneous datasets, and upload the parameters of the generative and discriminative models as shared parameters to the image translation server. Step 13: The image translation server aggregates the parameters of the generative model and the discriminative model uploaded by local users and performs an image translation server update. Step 14: The local user downloads the updated generative and discriminative models from the image translation server. Steps 12 and 13 are repeated until the image translation server reaches the required number of updates, thus completing the training process of the generative and discriminative models. Step 15: Local users perform data augmentation on the local heterogeneous dataset using a generative model, making the local data exhibit a uniform distribution of class labels, resulting in a locally uniformly distributed dataset, including: for the... Each local user will denote the trained generative model as... ;No. Local users calculate the most numerous category labels in local heterogeneous datasets. And calculate for category labels Number of data points to be supplemented ;No. A local user uses the trained generative model Local heterogeneous datasets Perform image translation to obtain transformed data. And use random flipping and data cropping to adjust the category labels. of Transformed data Perform data augmentation while preserving the category labels to supplement the category labels. The data makes it possible to correlate with category labels. The corresponding number of data is , The calculation method is expressed as follows: ;in, To obtain and The minimum value operator; the first A local user is constructed by Transformed data and A local uniformly distributed dataset composed of augmented data points, presenting a uniform distribution of category labels. The number of data points in the locally uniformly distributed dataset is [number missing]. ,and ; Step 16: The local user shares the network structure of the classification model with the data classification server and initializes the parameters of the classification model; Step 17: Local users train a classification model using a local uniformly distributed dataset and upload the parameters of the classification model as shared parameters to the data classification server. Step 18: The data classification server aggregates the parameters of the classification models uploaded by local users and performs a data classification server update. Step 19: The local user downloads the updated classification model from the data classification server. Steps 17 and 18 are repeated until the data classification server reaches the required number of updates, thus completing the training process of the data classification server.

2. The privacy-preserving heterogeneous federated learning method based on image translation according to claim 1, characterized in that, Also includes: Define common Local users Participating in federated learning, each local user Possessing Original data and corresponding category tags Local heterogeneous dataset And total data volume For the first There are local users ,in, For local heterogeneous datasets Category labels are The number of data, For local heterogeneous datasets Category labels are The number of data; Each local user has two structurally identical generative models and two structurally identical discriminative models. For the first... Let each local user have two generative models with identical structures, denoted as _____. and Let the two discriminant models with the same structure be denoted as and .

3. The privacy-preserving heterogeneous federated learning method based on image translation according to claim 2, characterized in that, The local users train generative and discriminative models based on local heterogeneous datasets, including: For the Local users will use local heterogeneous datasets. The original data is subjected to block partitioning and random block scrambling operations to obtain the target dataset. ; By generating models and Discriminant model and Combining local heterogeneous datasets With the target dataset Perform adversarial training and construct adversarial loss; utilize generative models. and Construct the cycle consistency loss; calculate the classification loss using a discriminative model; The image translation loss is calculated by combining the adversarial loss, cycle consistency loss, and classification loss. The image translation loss is used to train the generative model and the discriminative model.

4. The privacy-preserving heterogeneous federated learning method based on image translation according to claim 3, characterized in that, The adversarial loss is expressed as: (1) in, For mathematical expectation, Representing local heterogeneous datasets The original data in Represents the target dataset The target data in; Cyclic consistency loss Represented as: (2) in, Indicates transformed data, This indicates that the original data has been reconstructed. This represents the translation data of the target data. This indicates the target data to be reconstructed; Classification loss Represented as: (3) in, Representing local heterogeneous datasets The number of original data in the data. Representing local heterogeneous datasets The number of category labels in Representing local heterogeneous datasets The set of true category labels in the original data. This represents the logical output obtained by transforming data through linear layer operations in the discriminant model. Represents transformed data Corresponding to the The actual category label of the class, Represents transformed data Corresponding to the The logical output of the class; The image translation loss is calculated by combining the adversarial loss, cycle consistency loss, and classification loss. , is represented as: (4) No. The local user obtains the result by minimizing the image translation loss at the th . Generative model parameters used as shared parameters when the image translation server updates. and discriminant model parameters .

5. A privacy-preserving heterogeneous federated learning method based on image translation according to claim 4, characterized in that, The parameters of the image translation server, which aggregates the generative and discriminative models uploaded by local users, are represented as follows: (5) in, For assignment operator, In the first The parameters of the model are generated when the image translation server finishes updating. In the first The parameters of the discriminant model are determined when the image translation server finishes updating.

6. A privacy-preserving heterogeneous federated learning method based on image translation according to claim 1 or 2, characterized in that, The local user trains the classification model using a local uniformly distributed dataset, including: No. Local users use classification models Calculate category labels Data Logical output That is, the probability distribution is generated by the Softmax function, and the classification loss is... Represented as: (6) in, Let be the mathematical expectation, where, For the first Locally uniformly distributed dataset of local users Cross-entropy loss; No. Each local user uploads the gradient of the classification model as a shared parameter to the global model server, represented as: (7) in, For assignment operator, In the first When the training round is completed, the classification model The parameters, In the first When the training round is completed, the classification model The parameters, when hour, The initialization parameters in step 16, For classification models gradient of parameters This is the learning rate.

7. A privacy-preserving heterogeneous federated learning method based on image translation according to claim 1 or 2, characterized in that, The parameters of the classification models uploaded by local users and aggregated by the data classification server are represented as follows: (8) in, For assignment operator, In the first When the data classification server finishes updating the parameters of the classification model, For a locally uniformly distributed dataset The amount of data, for The amount of data in a local uniformly distributed dataset for each local user.

8. A privacy-preserving heterogeneous federated learning system based on image translation, characterized in that, include: A local user, an image translation server, and a data classification server are used to perform the method described in any one of claims 1 to 7.

Images