Federal learning privacy image data reconstruction method based on generative model

Abstract

The invention discloses a federated learning privacy image data reconstruction method based on a generative model, and relates to the technical field of federated learning privacy security. Comprising the following steps: firstly, adopting a pre-training generative adversarial network as a basic model, and optimizing a potential space and an intermediate feature domain of the basic model to generate virtual data matched with a target gradient; secondly, in the generation process, gradient matching loss and image fidelity loss are calculated and minimized, and it is ensured that a generation result not only conforms to gradient information, but also keeps visual authenticity; and finally, screening the image with the optimal gradient matching degree from the plurality of candidate images as a final reconstruction result. According to the method, a dual optimization mechanism is adopted, the method can adapt to distributed data, and the reconstruction quality can be effectively improved even for new data which do not appear in a training set. The method has wide application value in the scenes of privacy protection, data security analysis and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning privacy security, and specifically relates to a method for reconstructing private image data in federated learning based on a generative model. By improving the core parameter space (latent space) of the generative model and the intermediate stage (intermediate feature domain) of the data generation process, the method significantly improves the quality of reconstructing the original private image data through the system update information (gradient). Background Art

[0002] Federated learning is a new type of distributed artificial intelligence training method that allows multiple devices to jointly train an intelligent model without sharing the original data. Its basic principle is that each device first trains the model on its local private image data and only uploads the direction of model parameter adjustment (gradient) to the central server for aggregation and update. This method has important value in fields such as medical diagnosis and financial risk control that require strict privacy protection. However, the latest research has found that these uploaded gradients may be maliciously exploited to infer the original private image data. The existing methods for reconstructing private image data mainly have two major technical bottlenecks:

[0003] (1) Limited generation quality: The existing technologies mainly adjust in the initial parameter space of the generative model, resulting in insufficient clarity of the restored private image data and unable to accurately present the details of the original image.

[0004] (2) Insufficient adaptability: The existing methods require the training data of the generative model to be exactly matched with the target private image data, and the effect significantly decreases when encountering new types of data. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for reconstructing private image data in federated learning based on a generative adversarial network and a privacy protection strategy. The generative adversarial network (GAN) is a machine learning model composed of a generator and a discriminator, and the two learn the data distribution through an adversarial training method (analogous to the game process between a "forger" and an "anti-counterfeiting expert"). By optimizing the latent space of the generative model (i.e., the initial parameter space that controls the generation result, which can be understood as the "gene encoding" of the image) and the intermediate feature domain (the data features in the intermediate stage of the generation process, equivalent to the "semi-finished product processing link" of image generation), the method generates high-fidelity reconstructed private image data and simultaneously improves the adaptability to out-of-distribution data (new types of data with significant differences in features from the training data). To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] 1) Use a pre-trained generative adversarial network model (a benchmark model that has learned the characteristics of public data) as prior knowledge. By jointly optimizing the latent space and the intermediate feature domain of the generative model, gradually generate a reconstructed gradient (simulated generated parameter adjustment direction) that matches the target gradient (the direction of model parameter adjustment uploaded by the client).

[0007] 2) In the intermediate feature domain of the generative model, by minimizing the gradient matching distance (a measure of the difference between the reconstructed gradient and the target gradient, the smaller the value, the higher the matching degree), gradually optimize the intermediate features to generate higher-fidelity private image data.

[0008] 3) From the output of the intermediate layer of the generative model, select the private image data with the smallest gradient matching distance as the final reconstruction result.

[0009] The system includes three core modules: Generative model optimization module: By adjusting the latent vector (the initial parameter that controls the generation result, which can be regarded as the "design blueprint" of the image) and the intermediate feature domain parameters, generate virtual private image data that matches the target gradient; Gradient matching module: Calculate the difference between the reconstructed gradient and the target gradient, and optimize the generation process through the backpropagation algorithm;

[0010] Reconstruction result selection module: Screen the private image with the optimal matching degree from multiple groups of generation results.

[0011] The key technical points of the present invention are as follows:

[0012] 1. Dual optimization mechanism: Traditional gradient-based data reconstruction methods (techniques for reverse-inferring original data by analyzing model parameters) only optimize the latent space, resulting in loss of details. The present invention innovatively optimizes simultaneously in the latent space (initial parameters) and the intermediate feature domain (generation process): First, perform parameter fine-tuning in the latent space to retain the global features of the data (such as the image contour); Second, optimize local details (such as texture and color transition) in the intermediate feature domain, and improve the reconstruction accuracy through multi-level gradient matching (layer-by-layer optimization from abstract to specific).

[0013] 2. The present invention aims at the problem of poor reconstruction effect for out-of-distribution data, that is, the data that the generator has not seen usually has a poor reconstruction effect. When the generative model processes cross-domain data with different distributions, it is prone to overfitting or inability to generalize effectively. The present invention can effectively improve the performance of the generative model on diverse datasets, overcoming the training difficulties and insufficient generalization ability faced in traditional methods. By optimizing between the latent space and the intermediate feature domain, the present invention can more accurately retain the high-order features and detailed information in the data, thus significantly improving the fidelity of the reconstruction result.

[0014] This method significantly improves the quality of reconstructing the original private image data through the system update information (gradient) by improving the core parameter space (latent space) of the generative model and the intermediate stage (intermediate feature domain) of the private image data generation process. Brief Description of the Drawings

[0015] Figure 1 It is a system architecture diagram of the private image data reconstruction method based on the generative model.

[0016] Figure 2 It is a schematic diagram of the location of the attacker in the process of the private image data reconstruction method based on the generative model. Detailed Implementation Manner

[0017] This implementation method combines the optimization techniques of the generative model and the gradient inversion attack, and proposes a federated learning private image data reconstruction method based on the generative model. The purpose of this method is to improve the expression ability and generalization ability of the generative model. By optimizing the latent space and intermediate feature domain of the generative model, it restores the private image data of the client and solves the privacy leakage problem in the federated learning system.

[0018] The main goal of the generative model optimization module is to make the gradient of the generated image as close as possible to the gradient of the target private image by optimizing the latent space (usually a random vector) of the generative model. This is the core module of the whole system. It uses an optimization algorithm to gradually adjust the input latent vector so as to generate an image similar to the gradient of the target private image. First, a latent space vector z needs to be initialized. This vector is the input of the generative model G and is usually a random high-dimensional vector. After multiple layers of non-linear transformation, the feature domain is gradually generated: h1 = G0(z), h2 = G1(h1),..., h K = G N-1 (h K-1 ), x = G N (h K ). Among them, G0, G1,..., G N represent different layers of the generative model, h K is the feature representation of the Kth layer, and x is the finally generated reconstructed image. In addition, this module also calculates two loss functions, the gradient matching loss L match and the image fidelity loss L fidelity , and optimizes through the weighted sum of the two total loss function L total . Its goal is to minimize the difference between the gradient of the generated private image and the target gradient. The gradient matching loss is used to measure the similarity between the gradient of the generated private image and the gradient of the target private image, and the calculation formula is as follows: Among them, x * is the target private image, x is the reconstructed private image, is the gradient uploaded by the user, is the gradient for reconstructing the private image, represents the square of the Euclidean distance. The private image fidelity loss evaluates the similarity between the reconstructed private image and the target private image in the pixel space, ensuring that the generated private image not only matches the target data in the gradient space but also is as close as possible to the real data in the visual effect. The calculation formula is as follows: To comprehensively consider the balance between gradient matching and private image fidelity, a total loss function is introduced, and its calculation formula is as follows: L total = λ1L match + λ2L fidelity . Where λ1 and λ2 are weight parameters used to control the influence of gradient matching and fidelity on the final optimization process. By adjusting λ1 and λ2, more attention can be paid to gradient information matching or private image quality during the reconstruction process. In each optimization iteration, the latent vector z is updated by calculating the total loss function: where η is the learning rate to gradually approach the real data. The optimization process stops until the total loss function converges or reaches the specified number of iterations.

[0019] 2. The role of the gradient matching module is to calculate the difference between the gradients of the generated private image and the target private image. This module helps evaluate whether the reconstructed private image is close to the target private image, thus determining whether further optimization is needed. For each reconstructed private image x, calculate its corresponding gradient This can be achieved by applying a gradient calculation method (such as backpropagation in a convolutional neural network) to each pixel of the private image. The target gradient is the gradient uploaded by the user, usually from the gradient information of other participants in the federated learning system. Then, the similarity between them is measured by calculating the Euclidean distance between the gradient of the reconstructed private image and the target gradient: The smaller the gradient matching loss, the higher the matching degree between the reconstructed private image and the target private image. Finally, by minimizing the gradient matching loss function, the generated private image is optimized. The optimization process will gradually adjust the generated private image through multiple iterations to make its corresponding gradient closer to the target private image gradient.

[0020] 3. The goal of the inversion result selection module is to select the best result from multiple generated private images. By evaluating the matching degree of each generated private image with the target, the closest private image is selected as the final result. For each candidate generated private image x i , calculate its total loss L total with the target. Then, by comparing the loss values of each private image, the private image with the minimum total loss function is selected as the reconstructed image: where, x optIndicates the selected optimal generated privacy image, Indicates selecting x that minimizes the total loss i As the optimal solution x opt .

[0021] The federated learning data reconstruction method of the generation model proposed in this implementation method effectively overcomes the limitations of the expressive power and generalization ability in traditional methods by optimizing the latent space and intermediate feature domain of the generation model. Through layer-by-layer optimization, it can ensure that the restored privacy image has high fidelity. In addition, by extending to out-of-distribution data and the adaptability of defense strategies, it demonstrates strong performance and broad application prospects in more complex application scenarios.

Claims

1. A method for reconstructing private image data in federated learning based on a generative model, characterized in that, Reconstructing private image training data using the gradient information exchanged in federated learning includes the following steps: Step 1), using a pre-trained generative model, i.e., a generative adversarial network (GAN), as prior knowledge, gradually generate initial private image data that matches the target gradient by optimizing the latent space and intermediate feature domain of the generative model; Step 2), based on the initial private image data generated in Step 1), in the intermediate feature domain of the generative model, gradually optimize the intermediate features by minimizing the matching loss between the reconstructed private image data and the target gradient to generate high-fidelity reconstructed private image data; Step 3), during the optimization process in Step 2), limit the search range of the intermediate features to avoid generating unrealistic private images; Step 4), based on the optimization results of Step 2) and Step 3), select the output with the minimum gradient matching loss from the intermediate features of different layers of the generative model as the final reconstructed private image.

2. The method for reconstructing privacy image data in federated learning based on a generative model according to claim 1, wherein, The specific operation method for optimizing the latent space and intermediate feature domain of the generative model in Step 1) is: 1-1) Randomly initialize the latent vector of the generative model and optimize the latent vector by the gradient descent method to obtain the optimal latent vector; 1-2) Decompose the generative model into multiple intermediate layers and gradually optimize the features of each intermediate layer to generate reconstructed private image data; 1-3) When optimizing the features of each intermediate layer, ensure that the generated private images are authentic.

3. The method for reconstructing private image data in federated learning based on a generative model according to claim 1, characterized in that The specific operation method for minimizing the matching loss between the reconstructed private image data and the target private image data in Step 2) is: 2-1) Use part of the reconstructed private image data of the generative model and calculate the gradient of the reconstructed private image data; 2-2) Optimize the intermediate features by minimizing the matching loss between the gradient of the reconstructed private image data and the target gradient.

4. A method for reconstructing private image data in federated learning based on a generative model according to claim 1, characterized in that The specific operation method for selecting the output private image with the minimum gradient matching loss in Step 3) is: 4-1) After optimizing the features of each intermediate layer, calculate the matching loss between the reconstructed private image data generated by this layer and the target gradient; 4-2) Select the private image generated by the intermediate layer with the minimum matching loss as the final reconstruction result.

5. A method for reconstructing private image data in federated learning based on a generative model according to claim 1, characterized in that The goal of the generative model optimization module is to optimize the latent space of the generative model so that the gradient of the generated image matches the gradient of the target private image as much as possible. First, a latent space vector needs to be initialized. Space vector Is the input of the generative model G, which is a random high-dimensional vector; after multiple layers of nonlinear transformation, the feature domain is gradually generated: h2=G1(h1),...,h K =G N-1 (h K-1 ), Among them, G0, G1, ..., G N Represents different layers of the generative model, h K is the feature representation of the Kth layer, and x is the final reconstructed image; in addition, this module also calculates the gradient matching loss L match and image fidelity loss L fidelity Two loss functions, and the weighted sum of the two loss functions L total The optimization goal is to minimize the difference between the gradient of the generated private image and the target gradient. The gradient matching loss is used to measure the similarity between the gradient of the generated private image and the gradient of the target private image. The calculation formula is as follows: in, is the target private image, x is the reconstructed private image, Gradients uploaded by users, To reconstruct the gradient of the private image, represents the square of the Euclidean distance; the privacy image fidelity loss evaluates the similarity between the reconstructed privacy image and the target privacy image in the pixel space, ensuring that the generated privacy image not only matches the target data in the gradient space, but also is visually close to the real data; the calculation formula is as follows: In order to comprehensively consider the balance between gradient matching and privacy image fidelity, the total loss function is introduced, and its calculation formula is as follows: L total =λ1L match +λ2L fidelity ; λ1, λ2 are weight parameters used to control the impact of gradient matching and fidelity on the final optimization process; by adjusting λ1, λ2, more emphasis is placed on gradient information matching or privacy image quality during the reconstruction process; in each optimization iteration, the potential vector is updated by calculating the total loss function Where η is the learning rate; the optimization process stops until the total loss function converges or the specified number of iterations is reached.

6. A method for reconstructing private image data in federated learning based on a generative model according to claim 5, characterized in that, The role of the gradient matching module is to calculate the difference between the gradients of the generated privacy image and the target privacy image; for each reconstructed privacy image calculate the gradient corresponding to its image This can be achieved by applying a gradient calculation method to each pixel of the privacy image; the target gradient is the gradient uploaded by the user, usually from the gradient information of other participants in the federated learning system; then the similarity between them is measured by calculating the Euclidean distance between the gradient of the reconstructed privacy image and the target gradient: The smaller the gradient matching loss, the higher the matching degree between the reconstructed privacy image and the target privacy image; finally, by minimizing the gradient matching loss function, the generated privacy image is optimized; the optimization process will gradually adjust the generated privacy image through multiple iterations to make its corresponding gradient closer to the target privacy image gradient.

7. A method for reconstructing private image data in federated learning based on a generative model according to claim 6, characterized in that The goal of the inversion result selection module is to select an optimal result from multiple generated privacy images; by evaluating the matching degree of each generated privacy image with the target, the closest privacy image is selected as the final result; for each candidate generated privacy image calculate its total loss L with the target total ; then, by comparing the loss values of each privacy image, select the privacy image with the minimum total loss function as the reconstructed image: wherein, represents the selected optimal generated privacy image, represents the selection that makes the total loss minimum as the optimal solution

Images