Gradient inversion attack method based on image correction penalty

Abstract

The invention discloses a gradient inversion attack method based on image correction penalty, which adopts a self-adaptive over-parameterization network as an image generator, and enhances the capacity of capturing complex data distribution by the network by dynamically adjusting the number of channels of the generator. The method comprises the following steps of: generating noise input from different random seeds, calculating a registration average image of all candidate images, and introducing an image correction penalty term in each optimization process so as to reduce the visual deviation between a false image and a real image and ensure that the model has enough expression capability to process large-batch and high-resolution image data. A full connection layer is used as a label generator, and the two generators are jointly optimized, so that a generated false image and a false label can approach a real gradient, and then data are stably recovered. Through the image correction penalty term and over-parameter design, complex data distribution can be captured, the optimization stability is improved, the quality of the reconstructed image is improved, and more privacy information is disclosed.

Description

Technical Field

[0001] The present invention belongs to the field of federated learning and data security, and specifically relates to a gradient inversion attack method based on image correction penalty. Background Art

[0002] In today's digital age, data silos are ubiquitous. Various institutions or individuals hold data but it is difficult to integrate them. At the same time, data privacy and security issues are highly valued. Federated learning, as an innovative distributed collaborative training solution, aims to resolve the contradiction between data privacy protection and data fusion needs. Its core principle is that multiple data holders (such as various smart devices, financial and medical institutions) can collaboratively train models without sharing original data. In the federated learning architecture, there are multiple clients and a server. The client uses local data to calculate the model gradient and share it to the server. The server is responsible for aggregating these gradients to update the global model.

[0003] However, while federated learning protects the privacy of local data on the client, the shared gradient information has become a new risk point. During the model training process, the gradient information uploaded by the client contains the changes in the model parameters on the local data. Although these gradient information do not directly expose the original data, they contain rich data characteristics and distribution information. Attackers use complex mathematical analysis and optimization techniques to mine clues related to the original data from these gradient information, and then try to recover the client's private data. This type of attack is called a gradient inversion attack. For example, by analyzing the direction and size of the gradient and the gradient relationship between different parameters, combined with specific algorithms and model assumptions, attackers can construct optimization goals for data recovery, and continuously iterate the optimization to approach the original data, thereby leading to the risk of data leakage.

[0004] Existing gradient inversion attack methods have achieved good results in small batch low-resolution image scenarios, but they have exposed many limitations when facing large batch and high-resolution images. First, as the batch size increases, the feature information of a single image is diluted in the shared gradient, resulting in a sharp increase in the difficulty of recovery and a significant decrease in the effect. Secondly, the model stability problem is serious, and gradient vanishing and explosion phenomena occur frequently, which greatly affects the training efficiency and recovery effect. For example, in the optimization process based on distance metrics, improper metric selection or parameter setting may cause these unstable factors. Finally, high-resolution image restoration places higher requirements on the capacity and expression ability of the model. Existing methods find it difficult to maintain the fidelity and clarity of the image while ensuring computational efficiency. The generated images often cannot maintain the natural structure, which greatly limits their applicability in practical applications.

[0005] CN117935015A discloses a gradient inversion attack method based on a diffusion model, which obtains the parameters related to the federated learning model and the client to be attacked and the gradient of the last fully connected layer, restores the image sample label based on the gradient, and after multiple preliminary image reconstructions, selects the image with the smallest gradient loss as the registration image. When the image is finally reconstructed, a priori image is also generated and then iteratively optimized, and the prior knowledge of the diffusion model and improved regularization are used to improve the quality of the reconstructed image and reveal privacy information. However, the invention lacks direct constraints on the visual consistency of the generated image, and there may be limitations in relying on the diffusion model and a variety of network structure designs are not considered. There are deficiencies such as the single type of experimental data set that cannot fully verify the universality, the evaluation indicators are not rich and comprehensive enough to make it difficult to deeply analyze the performance, and the image may not be accurately restored due to reliance on the prior model when facing complex data distribution.

[0006] CN115527081A discloses a method for reconstructing input data in federated learning. First, multiple groups of pseudo samples and related hyperparameters are randomly initialized, and suitable hyperparameters are obtained by automatic hyperparameter exploration. The distributed gradient inversion algorithm is run, and then the initial gradient inversion is performed on multiple groups of pseudo samples respectively, and then the minimum loss combination optimization is performed to construct a group consistent regularization term and select the final sample. Finally, this sample is gradient inverted to obtain the reconstructed target sample. However, the invention lacks adaptive structural adjustment. When processing data of different complexity and scale, the fixed network structure may not be able to effectively capture data features, which limits the adaptability of the model to diversified data. Summary of the invention

[0007] The present invention discloses a gradient inversion attack method based on image correction penalty, which aims to effectively solve the problems faced by gradient inversion attack in federated learning. Due to the ambiguity of gradient information and model stability problems, the recovery effect is significantly reduced, and the existing methods often face gradient explosion and disappearance problems, which affect the training efficiency and recovery effect. The present invention is deployed on the server and can access the neural network structure and parameters of the global model. The only information exchanged between the server and the client is the gradient calculated based on the current global model. The present invention converts the data recovery task into a data generation problem guided by gradient information, and realizes data recovery through a carefully designed architecture. Specifically, an adaptive over-parameterized network is used as an image data generator, which can dynamically adjust the number of channels of the generator, expand the network capacity, and thus enhance the network's ability to capture complex data distribution. In order to improve the convergence of the real labeled image, noise input is generated from different random seeds, and the registered average image of all candidate images is calculated. An image correction penalty term is introduced in each optimization process to reduce the visual deviation between the false image and the real image, ensuring that the model has sufficient expressive power to process large batches and high-resolution image data. At the same time, the fully connected layer is used as the label data generator. The two generators are jointly optimized so that the fake images and fake labels they generate can approach the true gradient, thereby stably restoring the data.

[0008] To achieve the above object, the technical solution adopted by the present invention is: a gradient inversion attack method based on image correction penalty, including a false image generation branch and a false label generation branch;

[0009] The fake image generation branch generates realistic virtual images similar to real images by inputting random noise into the adaptive over-parameterized network. The generator converts these input information into images through the forward propagation of the neural network, and optimizes its generation ability through adversarial learning with the discriminator during the training process. The purpose is to make the generated image gradually approach the feature distribution of the real image, so that it is visually impossible to easily distinguish between real and fake images. The generator can not only generate diverse samples and expand the data set, but also be used for data enhancement to improve the generalization ability of the model.

[0010] The false label generation branch generates false label information based on the features or potential representations of the generated images, usually relying on a neural network or classifier to infer the labels. These labels are paired with the generated images in adversarial training, with the goal of deceiving the discriminator so that it cannot recognize the true relationship between the image and the label. The false label generation branch helps improve the quality and diversity of generated images, ensures the consistency between images and labels, and enhances the robustness of the model under diverse inputs and noisy data.

[0011] The two branches are mutually optimized through adversarial training to ensure that the generated image is highly consistent with the label, thereby improving the quality of the generated image.

[0012] A gradient inversion attack method based on image correction penalty includes the following steps:

[0013] Step 1: Initialize the latent space random vector input and set the iteration number threshold.

[0014] Step 2: Initialize the network parameters of the generator.

[0015] Step 3: Get the client's true gradient.

[0016] Step 4: Generate fake images and fake labels from the noise vector and conditional information through an adaptive over-parameterized network.

[0017] Step 5: Calculate the fake gradient based on the fake image and fake label.

[0018] Step 6: Calculate the loss value between the false gradient and the true gradient.

[0019] Step 7: Update the network parameters of the global model.

[0020] Step 8: Repeat steps 4 to 7 until the preset iteration number threshold is reached.

[0021] Step 9: Output the generated fake labels and fake images.

[0022] Preferably, in step 3, the present invention adopts an image classification model based on a convolutional neural network (CNN). At iteration t, the i-th client uses its local training data Calculate CNN model parameters θ t and local gradient F(·),θ t and L(·) are the global learning model, the network parameters at iteration t, and the loss function, respectively. To obtain the true gradient of the client, specifically:

[0023]

[0024] Preferably, in step 4, the present invention uses an adaptive over-parameterized network to generate fake images and fake labels, aiming to dynamically adjust the number of channels of the generator so that the number of parameters of the generator exceeds a specific threshold, so as to ensure that the model has sufficient capacity to learn complex data distributions and enhance its expressiveness. Specifically:

[0025] Step 4.1, over-parameterize the network. Given a latent vector z0 and a convolutional model G, define its generative model space as:

[0026]

[0027] Among them, Θ is the network parameter set, Generated images for the convolutional model.

[0028] Similarly, the optimal set of generated spaces can be defined as:

[0029]

[0030] A necessary condition for the success of the generative method is that the intersection of these two spaces is non-empty. Given a latent vector z0 and its corresponding generative model space, its intersection with the gradient constraint is defined as

[0031]

[0032] A necessary condition for successful gradient inversion is to In addition, Given a sufficiently large value, this non-emptiness condition always holds.

[0033] Step 4.2, adaptively update network parameters, update the model parameters W (including the weights of the convolutional layer and the linear layer) at specific moments in the model training process to ensure that the model does not overfit random noise, but is able to learn the key features in the data. Specifically, key parameter adjustments are made to the model when the training progress reaches 20%, 40%, and 60% to optimize model performance and promote its accurate learning of the real data distribution:

[0034] W pruned =W i -threshold(W i -W mean )

[0035]

[0036] Among them, threshold(x) is a threshold function that sets some smaller values ​​in the weights to zero, λ is a hyperparameter that determines the threshold of the pruned weights, and W mean is the weight matrix W i The mean of .

[0037] Step 4.3, based on the adaptive over-parameterized network obtained in the above steps 4.1 and 4.2, a fake image is output.

[0038] Step 4.4 uses a fully connected layer and a softmax layer for classification. It randomly extracts a vector from the latent space as input and outputs its corresponding false label.

[0039] Preferably, in step 5, the false gradient is calculated mainly based on the false image and the false label, specifically:

[0040] Step 5.1, use multiple random seeds to repeatedly generate multiple sets of images and calculate their average value, which is represented as the average image x mean ;

[0041]

[0042] Among them, G out Represents a single generated image.

[0043] Step 5.2, construct the image correction penalty term, which is defined by calculating the Euclidean distance (L2 norm) between a single generated image and the average image of a set of generated images, denoted as L IC :

[0044] L IC =α group ·||G out (i)-x mean ||2

[0045] where α group is the regularization coefficient, which is used to control the strength of the regularization term.

[0046] Step 5.3, calculate the loss function of fake images and fake labels, expressed as Gloss:

[0047]

[0048] in, represents the expected value, M is the total number of categories, G' lable (j) is the label of the generated image belonging to the jth class (if it is a multi-classification problem, it is a one-hot encoded vector), softmax(G' pred ) j is the model's predicted probability for the jth class.

[0049] Step 5.4: Perform one SGD iteration to get the pseudo gradient

[0050] Preferably, the loss value between the false gradient and the true gradient is mainly calculated in step 6. Specifically:

[0051] The Euclidean distance is introduced into the loss function to measure the difference between the true and false gradients. The WD loss is introduced to minimize the geometric difference between the two gradient vectors, and the TVLoss is applied to impose smoothness constraints on the generated fake image data. Therefore, the loss function of the malicious server is It is expressed as:

[0052]

[0053] The MSE loss and WD are weighted equally, and α is the weighting parameter of the smoothing regularization.

[0054] Compared with the prior art, the present invention has the following improvements and advantages:

[0055] 1. This paper proposes an adaptive over-parameterized network for the first time, which enhances the capacity of the network by adjusting the number of channels of the generator and enables it to capture more complex data distribution. This is aimed at improving the expressiveness of the model to meet the needs of high-resolution and large-volume data recovery.

[0056] 2. This paper innovatively proposes an image correction penalty term based on multi-seed optimization and image registration. This strategy improves the reconstruction quality and ensures the visual consistency of the generated images by penalizing the generated images that deviate greatly from the average image.

[0057] 3. The proposed model relies only on simple image priors, such as image smoothness, without the need to deeply understand the distribution of input data or adjust model parameters. This enhances the versatility of the model and makes it more applicable to diverse real-world settings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a structural schematic diagram of a gradient inversion attack method based on image correction penalty of the present invention.

[0059] Figure 2 It is a flow chart of a gradient inversion attack method based on image correction penalty of the present invention.

[0060] Figure 3 It is a structural schematic diagram of an adaptive over-parameterized network of a gradient inversion attack method based on image correction penalty of the present invention.

[0061] Figure 4 This is a comparison diagram of the effects of a gradient inversion attack method based on image correction penalty of the present invention and other gradient inversion attack methods. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0063] It should be noted here that in order to avoid unnecessary details that would obscure important parts of the present invention, the drawings only show structures or processing steps that are closely related to the present invention, while ignoring other details that are not very relevant to the present invention.

[0064] A gradient inversion attack method based on image correction penalty includes the following steps:

[0065] Step 1: Initialize the latent space random vector input and set the iteration number threshold.

[0066] Step 2: Initialize the network parameters of the generator.

[0067] Step 3: Get the client's true gradient.

[0068] A convolutional neural network (CNN)-based image classification model is used. At iteration t, the i-th client uses its local training data Calculate CNN model parameters θ t and local gradient F(·),θ t and L(·) are the global learning model, the network parameters at iteration t, and the loss function, respectively. To obtain the true gradient of the client, specifically:

[0069]

[0070] Step 4: Generate fake images and fake labels from the noise vector and conditional information through an adaptive over-parameterized network.

[0071] Adopting an adaptive over-parameterized network to generate fake images and fake labels, the aim is to dynamically adjust the number of channels of the generator so that the number of parameters of the generator exceeds a certain threshold to ensure that the model has sufficient capacity to learn complex data distributions and enhance its expressiveness. Specifically:

[0072] Step 4.1, over-parameterize the network. Given a latent vector z0 and a convolutional model G, define its generative model space as:

[0073]

[0074] Among them, Θ is the network parameter set, Generated images for the convolutional model.

[0075] Similarly, the optimal set of generated spaces can be defined as:

[0076]

[0077] A necessary condition for the success of the generative method is that the intersection of these two spaces is non-empty. Given a latent vector z0 and its corresponding generative model space, its intersection with the gradient constraint is defined as

[0078]

[0079] A necessary condition for successful gradient inversion is to In addition, Given a sufficiently large value, this non-emptiness condition always holds.

[0080] Step 4.2, adaptively update network parameters, update the model parameters W (including the weights of the convolutional layer and the linear layer) at specific moments in the model training process to ensure that the model does not overfit random noise, but is able to learn the key features in the data. Specifically, key parameter adjustments are made to the model when the training progress reaches 20%, 40%, and 60% to optimize model performance and promote its accurate learning of the real data distribution:

[0081] W pruned =W i -threshold(W i -W mean )

[0082]

[0083] Among them, threshold(x) is a threshold function that sets some smaller values ​​in the weights to zero, λ is a hyperparameter that determines the threshold of the pruned weights, and W mean is the weight matrix W i The mean of .

[0084] Step 4.3, based on the adaptive over-parameterized network obtained in the above steps 4.1 and 4.2, a fake image is output.

[0085] Step 4.4 uses a fully connected layer and a soffmax layer for classification. It randomly extracts a vector from the latent space as input and outputs its corresponding false label.

[0086] Step 5: Calculate the fake gradient based on the fake image and fake label.

[0087] Specific:

[0088] Step 5.1, use multiple random seeds to repeatedly generate multiple sets of images and calculate their average value, which is represented as the average image x mean :

[0089]

[0090] Among them, Gout Represents a single generated image.

[0091] Step 5.2, construct the image correction penalty term, which is defined by calculating the Euclidean distance (L2 norm) between a single generated image and the average image of a set of generated images, denoted as L IC :

[0092] L Ic =α group ·||G out (i)-x mean ||2

[0093] where α group is the regularization coefficient, which is used to control the strength of the regularization term.

[0094] Step 5.3, calculate the loss function of fake images and fake labels, expressed as Gloss:

[0095]

[0096] in, represents the expected value, M is the total number of categories, G' lable (j) is the label of the generated image belonging to the jth class (if it is a multi-classification problem, it is a one-hot encoded vector), softmax(G' pred ) j is the model's predicted probability for the jth class.

[0097] Step 5.4: Perform one SGD iteration to get the pseudo gradient

[0098] Step 6: Calculate the loss value between the false gradient and the true gradient.

[0099] Specific:

[0100] The Euclidean distance is introduced into the loss function to measure the difference between the true and false gradients. The WD loss is introduced to minimize the geometric difference between the two gradient vectors, and the TVLoss is applied to impose smoothness constraints on the generated fake image data. Therefore, the loss function of the malicious server is It is expressed as:

[0101]

[0102] The MSE loss and WD are weighted equally, and α is the weighting parameter of the smoothing regularization.

[0103] Step 7: Update the network parameters of the global model.

[0104] Step 8: Repeat steps 4 to 7 until the preset iteration number threshold is reached.

[0105] Step 9: Output the generated fake labels and fake images.

[0106] At this point, the gradient inversion attack method based on image correction penalty can be implemented.

[0107] In order to verify the effectiveness of the proposed method, the present invention and the traditional inversion attack algorithm are compared in an experiment:

[0108] DLG: A gradient inversion attack method that iteratively optimizes the data to be solved and the labels to be solved, and uses the Euclidean distance between the gradient of the model parameters on the data to be solved and the labels and the true value of the gradient as the gradient matching target.

[0109] GRNN: A gradient inversion attack method that formulates the data recovery task as a data generation problem guided by gradient information, introduces a GAN model as an image data generator, and introduces a simple fully connected (FC) layer as a label data generator. The generated false gradient can then be calculated given a shared global model, and the generator is jointly optimized to approximate the true gradient.

[0110] IORN: This paper proposes a gradient inversion attack method based on image correction penalty, which enhances the network capacity through over-parameterization design, enabling it to capture complex data distribution, thereby achieving higher model expression capabilities. At the same time, the image correction penalty term is introduced to constrain the deviation between the generated image and the average image, thereby improving the optimization stability and reducing the convergence deviation.

[0111] Figure 4 The average peak signal-to-noise ratio (PSNR) score obtained using the converged global model for DLG, GRNN, and IORN at a batch size of 1 is given to quantify the quality of the generated images, which is an objective criterion for image evaluation. The higher the PSNR value, the higher the similarity between the two images. DLG fails to restore the training images on the MNIST and LFW datasets because no visually recognizable images are generated. Overall, IORN's inversion performance on both networks and three datasets is better than the classic DLG algorithm and the GRNN method.

[0112] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A gradient inversion attack method based on image correction penalty is used to recover client data in a federated learning environment, characterized by: It includes a fake image generation branch and a fake label generation branch, which are used to generate fake images and fake labels; The fake image generation branch generates realistic virtual images similar to real images by inputting random noise into the adaptive over-parameterized network. The generator converts these input information into images through the forward propagation of the neural network, and optimizes its generation ability through adversarial learning with the discriminator during the training process. The purpose is to make the generated image gradually approach the feature distribution of the real image, so that it is visually difficult to distinguish between real and fake images. The generator can not only generate diverse samples and expand the data set, but also be used for data enhancement and improve the generalization ability of the model. The false label generation branch generates false label information based on the features or potential representations of the generated images, usually relying on a neural network or classifier to infer the labels. These labels are paired with the generated images in adversarial training, with the goal of deceiving the discriminator so that it cannot recognize the true relationship between the image and the label. The false label generation branch helps improve the quality and diversity of generated images, ensures the consistency between images and labels, and enhances the robustness of the model under diverse inputs and noisy data. The two branches are mutually optimized through adversarial training to ensure that the generated image is highly consistent with the label, thereby improving the quality of the generated image.

2. A gradient inversion attack method based on image correction penalty, characterized in that: It includes a fake image generation branch and a fake label generation branch, including the following steps: Step 1: Initialize the latent space random vector input and set the iteration number threshold. Step 2: Initialize the network parameters of the generator. Step 3: Get the client's true gradient. Step 4: Generate fake images and fake labels from the noise vector and conditional information through an adaptive over-parameterized network. Step 5: Calculate the fake gradient based on the fake image and fake label. Step 6: Calculate the loss value between the false gradient and the true gradient. Step 7: Update the network parameters of the global model. Step 8: Repeat steps 4 to 7 until the preset iteration number threshold is reached. Step 9: Output the generated fake labels and fake images.

3. The gradient inversion attack method based on image correction penalty according to claim 2 is characterized in that: In step 3, a convolutional neural network (CNN)-based image classification model is used to obtain the true gradient of the client. At iteration t, the i-th client is trained based on its local training data Calculate CNN model parameters θ t and local gradient F(·),θ t and L(·) are the global learning model, the network parameters at iteration t, and the loss function, respectively. To obtain the true gradient of the client, specifically:

4. The gradient inversion attack method based on image correction penalty according to claim 2, characterized in that: In step 4, an adaptive over-parameterized network is used to generate fake images and fake labels, aiming to dynamically adjust the number of channels of the generator so that the number of parameters of the generator exceeds a specific threshold, thereby ensuring that the model has sufficient capacity to learn complex data distributions and enhance its expressiveness. Specifically: Step 4.1, over-parameterize the network. Given a latent vector z0 and a convolutional model G, define its generative model space as: Among them, Θ is the network parameter set, Generated images for the convolutional model. Similarly, the optimal set of generated spaces can be defined as: A necessary condition for the success of the generative method is that the intersection of these two spaces is non-empty. Given a latent vector z0 and its corresponding generative model space, its intersection with the gradient constraint is defined as A necessary condition for successful gradient inversion is to In addition, Given a sufficiently large value, this non-emptiness condition always holds. Step 4.2, adaptively update network parameters, update the model parameters W (including the weights of the convolutional layer and the linear layer) at specific moments in the model training process to ensure that the model does not overfit random noise, but is able to learn the key features in the data. Specifically, key parameter adjustments are made to the model when the training progress reaches 20%, 40%, and 60% to optimize model performance and promote its accurate learning of the real data distribution: W pruned =W i -threshold(W i- W mean ) Among them, threshold(x) is a threshold function that sets some smaller values ​​in the weights to zero, λ is a hyperparameter that determines the threshold of the pruned weights, and W mean is the weight matrix W i The mean of . Step 4.3, based on the adaptive over-parameterized network obtained in the above steps 4.1 and 4.2, a fake image is output. Step 4.4 uses a fully connected layer and a softmax layer for classification. It randomly extracts a vector from the latent space as input and outputs its corresponding false label.

5. The gradient inversion attack method based on image correction penalty according to claim 2 is characterized in that: In step 5, the false gradient is calculated mainly based on the false image and the false label. Specifically: Step 5.1, use multiple random seeds to repeatedly generate multiple sets of images and calculate their average value, which is represented as the average image x mean : Among them, G out Represents a single generated image. Step 5.2, construct the image correction penalty term, which is defined by calculating the Euclidean distance (L2 norm) between a single generated image and the average image of a set of generated images, denoted as L IC : L IC =a group ·||G out (i)-x mean ||2 where α group is the regularization coefficient, which is used to control the strength of the regularization term. Step 5.3, calculate the loss function of fake images and fake labels, expressed as Gloss: in, represents the expected value, M is the total number of categories, G′ lable (j) is the label of the generated image belonging to the jth class (if it is a multi-classification problem, it is a one-hot encoded vector), softmax(G′ pred ) j is the model's predicted probability for the jth class. Step 5.4: Perform one SGD iteration to get the pseudo gradient

Images