Bulk image reconstruction method based on prior regularization

Abstract

The present application relates to a kind of mass image reconstruction methods based on prior regularization, for reconstructing image for training using exchanged gradient information in federal learning.First, as ordinary participant, participate in collaborative learning to obtain the model structure of collaborative learning and the gradient information received in training process;Simulated image with random value is generated, the simulated image is input into the obtained model to obtain loss, and then the simulated gradient is obtained according to the loss;The original gradient of global model is fitted using simulated gradient, and the distance between simulated gradient and original gradient is used as loss to optimize simulated image;In the process of optimization, according to the prior knowledge of image, prior knowledge regularization term is generated, and the prior knowledge regularization term is used to guide the progress of optimization process, so that the accuracy and speed of optimization process are greatly improved.The present application can only use gradient information to reconstruct and generate gradient image with mass, high pixel matching degree.

Description

Technical Field

[0001] This invention designs a large-scale image reconstruction method based on prior regularization, which belongs to the category of image reconstruction methods using gradient information. Background Technology

[0002] Federated learning resolves the conflict between privacy and efficiency in machine learning. Compared to traditional machine learning, federated learning uses a central parameter server to control the global training progress. All clients download the latest global model from the parameter server, train their local models on their own private data, and finally send the updated gradients to the central parameter server. The server aggregates all gradients to update the global model. Federated learning can fully utilize clients' private data to train the global model while protecting client data privacy by avoiding direct transmission of client private data; only gradients are transmitted between the parameter server and clients. Previous research generally suggests that federated learning better protects client privacy than traditional centralized machine learning, and therefore, federated learning has been widely applied in healthcare and financial scenarios.

[0003] In practical applications of federated learning, many training processes utilize temporary data from participants. For example, Google's GBoard uses users' temporary input history as the local training dataset for federated learning. In this case, the training data used for training may be deleted after training is complete, and it is impossible to obtain this dataset again if subsequent similar training requires the same dataset. Therefore, reconstructing the training data from intermediate and final training results is a real need.

[0004] Current research formalizes the reconstruction problem as an optimization process. After obtaining the gradients from the intermediate training process, a pair of simulated data and labels can be randomly generated. The simulated data is then input into the global model to obtain simulated gradients. By optimizing the simulated gradients to approximate the true gradients, the simulated data can be made to resemble the real training data. These optimization-based research methods can achieve pixel-level data reconstruction based on gradients. However, these methods only have good recovery effects for small batches of data; their performance deteriorates significantly as the batch size increases. Summary of the Invention

[0005] This paper addresses the poor performance of image reconstruction based on gradient information generated during training on large datasets by proposing a reconstruction method based on image prior regularization. Building upon the Deep Leakage from Gradients (DLG) method, this approach utilizes observations of universally existing image properties to design prior regularization terms that constrain the image reconstruction direction. These regularization terms are then used to optimize the image reconstruction process, thereby improving the performance of gradient reconstruction methods on large datasets.

[0006] To achieve the above objectives, the technical solution adopted in this invention is based on generating optimized simulated images using generative adversarial networks and using image prior regularization terms to optimize the sharpness of reconstructed images, as shown in the figure.

[0007] The implementation steps are as follows:

[0008] Step (1) Participants As an ordinary user, you participate in federated learning to obtain the global model F for federated learning training, where W is the parameter of the global model.

[0009] Step (2) Participants The obtained global model F is used as the discriminator model D in the locally trained adversarial neural network model. A corresponding generator G is then trained locally using this discriminator. For each image generated by generator G, the image is input into discriminator D for scoring, and the scoring result is used as the generator's loss to optimize the generator using gradient descent.

[0010] Step (3) Participants Use the trained generator G to generate the same number of images as the required reconstruction batch, and use these images as simulation data x′;

[0011] Step (4) Participants Obtain the original gradients existing in the global model F. The simulated gradient obtained from the simulation data

[0012] Step (5) Participants The obtained simulated gradient Compared with the original gradient To calculate the loss, we use gradient descent to reduce the distance between the original gradient and the simulated gradient, thereby reducing the distance between the simulated data and the original data. We then use a filter kernel to calculate the smoothness of the simulated image to constrain the direction of optimization. When the distance is sufficiently small, the simulated data and the original data will become very similar, indicating successful reconstruction.

[0013] Beneficial effects

[0014] This method, building upon gradient information-based image reconstruction, leverages the influence of the initial image distribution on reconstruction results and the constraint of smoothness regularization on the optimization direction to improve the reconstruction performance of existing methods, significantly enhancing reconstruction accuracy when reconstructing large batches of images. Image reconstruction essentially involves searching for correct feature points in the image's feature space. Using a trained generator to generate optimized initial images involves finding a starting point close to the target feature point in the feature space, while using prior image knowledge as a regularization term reduces the search range within the feature space. This method accelerates the search and improves accuracy by selecting closer starting points and reducing the search area within the feature space. Attached Figure Description

[0015] Figure 1 Diagram of the reconstruction process

[0016] Figure 2 This is a schematic diagram illustrating the process of calculating the smoothness of a simulated image using a filter kernel.

[0017] Figure 3 This document describes the process and results of reconstructing a large batch of images using this method, and compares it with other methods. Detailed Implementation

[0018] The purpose of this invention is to propose a method for large-scale image reconstruction based on generative adversarial networks and prior knowledge of images. This method utilizes generative adversarial networks to generate simulated images with distributions similar to the original images and image smoothness regularization terms to improve the accuracy of image reconstruction.

[0019] As shown in the figure, the implementation steps of this method are as follows:

[0020] Step (1) Obtain the global model F trained by federated learning

[0021] The global model F is trained collaboratively by all participants in the federated learning training. Each participant receives a parameter from the parameter server at the beginning of each training round. The latest global model F is obtained from the local dataset. Then, all participants use their respective datasets to backpropagate on the global model F to calculate the gradient.

[0022]

[0023] Where t is the t-th iteration of the federated learning, i is the i-th participant among all N participants, and x t,i Let y be the local data used by the i-th participant in the t-th round of global iteration. t,i These are the labels corresponding to this local data, W tThese are the parameters of the global model trained in round t. After calculating their own local gradients, all participants upload their gradients to the parameter server. The parameter server then aggregates all gradients to obtain the parameters.

[0024]

[0025] The parameter server then uses Update global model Where η is the learning rate

[0026] Step (2) Use the obtained global model F to train a generative adversarial network locally.

[0027] This step treats the global model F as the discriminator D in the generative adversarial network (GAN). In normal GAN ​​training, the discriminator and generator are updated in each iteration. However, in this step, each iteration of GAN training uses the latest global model as the latest discriminator instead of updating it through loss. Each training iteration of the GAN involves the locally trained generator G generating images. These generated images are then fed into the discriminator to obtain its prediction, which is used as the generator's loss to update it. This results in the generator producing images with very high accuracy in the federated learning-trained global model, meaning the generated images are similar in distribution to the images in the dataset to be reconstructed. The optimization objective function used to update the generator is:

[0028]

[0029] Where n represents the number of input image samples, θ G and θ D Let represent the parameters of the generator and discriminator, respectively, and z represent the latent vector of the random sample.

[0030] Step (3) Use the images generated by the trained generator G as simulation data x′

[0031] x′=G(z)

[0032] Step (4) obtains the gradients uploaded by other users. Use this gradient as the original gradient; calculate the loss by inputting the simulated data x′ into the global model F and comparing the output obtained with the simulated output y′; then use the calculated loss for backpropagation to obtain the simulated gradient.

[0033]

[0034] in In this invention, the cross-entropy loss function is used.

[0035] Step (5) uses the obtained simulated gradient Compared with the original gradient Calculate the loss, use this loss to optimize the simulation data, and when the loss is sufficiently low, the optimized simulation data x is complete. ′* The difference between the simulated data and the original data will become small enough that the optimized simulated data x is complete. ′* It's about rebuilding the data;

[0036] This invention introduces the use of a filter kernel Calculate the simulated data x ′* The image smoothness is used as a regularization term to constrain the optimization process, and the improved optimization objective function is:

[0037]

[0038]

[0039]

[0040]

[0041] Where α is a hyperparameter used to control the weight of the smoothness of the reconstructed image in the reconstruction optimization process; x′ i,j,k This represents the pixel value of x′ at position (i,j,k); The filter kernel is represented by `sum`; `sum` represents the sum of all elements in the matrix; `abs` represents taking the absolute value of each element in the matrix; `X` represents the expression with respect to x′. i,j,k central element A matrix representing the window size.

[0042] Instruction manual illustrations Figure 3 This figure compares the reconstruction results of our method with those of the DLG method and the Inverting Gradients (INV) method when reconstructing images in batches of 32. Since color is an important indicator for evaluating image reconstruction performance, the attached figure uses color images. The first row of the figure shows the initial images generated by the generator; the second and third rows show the reconstruction results of our method at rounds 50 and 100; the fourth row shows the final reconstruction result of our method; the fifth row shows the original images; the sixth row shows the reconstruction result of the DLG method; and the seventh row shows the reconstruction result of the INV method. The reconstruction results of the three methods clearly demonstrate that our method significantly improves upon the others.

Claims

1. A batch image reconstruction method based on prior regularization, characterized in that: Includes the following steps, step (1) participants As an ordinary user, you participate in federated learning to obtain the global model F for federated learning training, where W is the parameter of the global model. Step (2) Participants The obtained global model F is used as the discriminator model D in the locally trained adversarial neural network model, and a corresponding generator G is trained locally using this discriminator. Each image generated by generator G is input into discriminator D for scoring, and the scoring result is used as the loss of generator. The generator is then optimized using gradient descent algorithm. Step (3) Participants A batch of images is generated using the trained generator G. The size and number of these images are the same as the batch of images to be reconstructed. These generated images are then used as simulation data x. ' ; Step (4) Participants Obtain the original gradient based on the global model F. Obtain the simulated gradient from the simulated data. Step (5) Participants Use the obtained simulated gradient Compared with the original gradient To calculate the loss, we use gradient descent to reduce the distance between the original gradient and the simulated gradient, thereby reducing the distance between the simulated data and the original data. We also use a filter kernel to calculate the smoothness of the simulated image to constrain the direction of optimization. The objective function of the optimization process is: Where α is a hyperparameter used to control the weight of the smoothness of the reconstructed image in the reconstruction optimization process; ° represents the Adama product; x' i,j,k This represents the pixel value of x' at position (i,j,k); The filter kernel is represented by `sum`; `abs` represents the absolute value of each element in the matrix; `X` represents the expression with respect to x'. i,j,k central element Given a matrix of window size, the optimized simulation data x' * It involves reconstructing the data; where l is the cross-entropy loss function.

2. The method for large-scale image reconstruction based on prior regularization according to claim 1, characterized in that: Step (3) uses the trained generator to generate optimized initial images; Step (5) uses a filter kernel to calculate the smoothness of the simulated images, that is, uses prior knowledge of the image to constrain the optimization process.

Images