A privacy protection deep learning method based on diffusion model and label reconstruction

Abstract

The application discloses a privacy protection deep learning method based on a diffusion model and label reconstruction, and constructs a privacy protection deep learning model composed of a variational autoencoder, a diffusion model and a classifier; the variational autoencoder is trained by using original images and target images, and the original images are transformed into original latent layer signals by an encoding unit of the variational autoencoder; the diffusion model is trained by using the original latent layer signals, the original latent layer signals are transformed into reconstructed latent layer signals by the diffusion model, and a classifier is trained based on the reconstructed latent layer signals and category labels of the original images; the trained classifier is used to perform data classification on the reconstructed latent layer signals, and a decoding unit of the variational autoencoder is used to transform the reconstructed latent layer signals into reconstructed images. The above method can solve the image data privacy leakage problem in the deep neural network training process, so that a good balance between classification accuracy and data privacy protection performance is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a privacy protection deep learning method based on a diffusion model and label reconstruction. BACKGROUND

[0002] In a target classification application scenario, a data owner can use a deep neural network to perform a high-accuracy data classification task, but an attacker can reconstruct original data in a data set with high quality by analyzing parameters generated when a deep neural network is trained, thereby bringing great challenges to data privacy protection of the data owner, and the prior art lacks a solution to an image data privacy leakage problem based on a diffusion model. SUMMARY

[0003] The present application aims to provide a privacy protection deep learning method based on a diffusion model and label reconstruction, which can solve the problem of image data privacy leakage in the training process of a deep neural network, thereby achieving a good balance between classification accuracy and data privacy protection performance.

[0004] The present application is achieved by the following technical solutions:

[0005] A privacy protection deep learning method based on a diffusion model and label reconstruction, the method comprising:

[0006] Step 1, constructing a privacy protection deep learning model composed of a variational autoencoder, a diffusion model and a classifier;

[0007] Step 2, training the variational autoencoder using original images and target images, and transforming the original images into original latent layer signals by an encoding unit of the variational autoencoder;

[0008] Step 3, training the diffusion model using the original latent layer signals, transforming the original latent layer signals into reconstructed latent layer signals by the diffusion model, and training the classifier based on the reconstructed latent layer signals and class labels of the original images;

[0009] Step 4, performing data classification on the reconstructed latent layer signals using the trained classifier, and transforming the reconstructed latent layer signals into reconstructed images using a decoding unit of the variational autoencoder.

[0010] As can be seen from the above technical solutions of the present application, the above method can solve the problem of image data privacy leakage in the training process of a deep neural network, thereby achieving a good balance between classification accuracy and data privacy protection performance. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0012] Figure 1 A privacy protection deep learning method based on diffusion model and label reconstruction provided by the embodiments of the present application is shown in the flowchart.

[0013] Figure 2 A structure diagram of the privacy protection deep learning model described in the embodiments of the present application is shown.

[0014] Figure 3 Four original images randomly selected from the CIFAR-10 dataset and the corresponding class labels in the example provided by the present application are shown.

[0015] Figure 4 The reconstructed images corresponding to the original images in the example provided by the present application and the corresponding PSNR values are shown. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments of the present application, which do not constitute a limitation on the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the protection scope of the present application.

[0017] As Figure 1 A privacy protection deep learning method based on diffusion model and label reconstruction provided by the embodiments of the present application is shown in the flowchart, which comprises:

[0018] Step 1, constructing a privacy protection deep learning model composed of a variational autoencoder, a diffusion model and a classifier;

[0019] In this step, as Figure 2 A structure diagram of the privacy protection deep learning model described in the embodiments of the present application is shown. The input image of the constructed privacy protection deep learning model is a color image with a size of 32x32, which specifically comprises:

[0020] A two-dimensional convolution operation unit Conv(in_channels, out_channels, kernel_size, stride, padding) with an input channel number of in_channels, an output channel number of out_channels, a convolution kernel size of kernel_size, a convolution kernel step of stride, and a padding number of padding;

[0021] A max-pooling operation unit MaxPool2d;

[0022] A fully connected layer Linear(in_channels, out_channels) with an input channel number of in_channels and an output channel number of out_channels;

[0023] An encoding unit Encoder of a variational autoencoder;

[0024] A decoding unit Decoder of a variational autoencoder;

[0025] An attention mechanism-based down-sampling unit AttnDownBlock2D;

[0026] An attention mechanism-based intermediate unit AttnMidBlock2D;

[0027] An attention mechanism-based up-sampling unit AttnUpBlock2D;

[0028] A down-sampling unit DownBlock2D;

[0029] An up-sampling unit UpBlock2D;

[0030] A time step embedding unit Time embedding.

[0031] Step 2, training the variational autoencoder using the original image and the target image, and transforming the original image into an original latent layer signal by the encoding unit of the variational autoencoder;

[0032] In this step, the specific process is as follows:

[0033] (1) The original image set is represented as x, and the blocking and random block shuffling operation is performed on all images in the original image set x to obtain a target image set

[0034] (2) The loss function of the variational autoencoder is defined as :

[0035]

[0036] wherein x denotes an original image in the original image set x; denotes a target image in the target image set E is an encoding unit of the variational autoencoder; D is a decoding unit of the variational autoencoder; denotes an expectation operation based on the statistical distribution of the original image; denotes the square of the l2 norm of the difference between the image generated by the original image sequentially passing through the encoding unit and the decoding unit and the target image;

[0037] (3) training the variational autoencoder, i.e., minimizing the loss function of the variational autoencoder to obtain the trained encoding unit E * and the decoding unit D * , denoted as:

[0038]

[0039] wherein denotes the optimal encoding unit E * and the decoding unit D * obtained by training the variational autoencoder;

[0040] (4) using the trained encoding unit E * to transform the original image x into the original latent signal z = E * (x).

[0041] Step 3, training the diffusion model using the original latent signal, transforming the original latent signal into a reconstructed latent signal by the diffusion model, and training a classifier based on the reconstructed latent signal and the category label of the original image;

[0042] In this step, the specific process is as follows:

[0043] (1) setting the maximum value of the time step as T, and generating Gaussian noise

[0044]

[0045] (2) using the Gaussian noise to perform a noise addition operation on the original latent signal z step by step, denoted as:

[0046]

[0047] wherein is Gaussian noise obeying a distribution with a mean of 0 and a variance of I; z0 = z is the noisy latent signal when t = 0, and 0 < a t < 1; z t is the noisy latent signal;

[0048] (3) Define the diffusion model loss function is:

[0049]

[0050] where, is the predicted noise generated by the diffusion model with the noisy latent signal z t and the time step t as input signals.

[0051] (4) Train the diffusion model, i.e., minimize the diffusion model loss function to obtain the corresponding model parameters θ * :

[0052]

[0053] (5) Use the predicted noise to perform a noise removal operation on the noisy latent signal z t step by step to obtain the predicted noisy latent signal

[0054] where, is a Gaussian noise with mean 0 and variance I distribution; is the mean of the predicted noisy latent signal , denoted as:

[0055]

[0056] and is the variance of the predicted noisy latent signal , denoted as:

[0057]

[0058] (6) Obtain the reconstructed latent signal

[0059] where, is the predicted noisy latent signal when t = 0; is the mean of the predicted noisy latent signal when t = 1; is the variance of the predicted noisy latent signal when t = 1; is a Gaussian noise with mean 0 and variance I distribution when t = 1;

[0060] (7) Define the loss function of the classifier denoted as:

[0061]

[0062] wherein m represents the number of images in the original image set x; c represents the number of image categories in the original image set x; y represents the real category label set of the original image; represents the predicted category label set of the reconstructed latent signal through the classifier; y i (j) represents the real category label of the reconstructed latent signal corresponding to the jth category; represents the predicted category label of the reconstructed latent signal corresponding to the jth category;

[0063] (8) Training the classifier, i.e., minimizing the loss function of the classifier to obtain the model parameter ω of the classifier * , which is represented as:

[0064]

[0065] Step 4, using the trained classifier to perform data classification on the reconstructed latent signal, and using the decoding unit of the trained variational autoencoder to transform the reconstructed latent signal into a reconstructed image.

[0066] In this step, the specific process is as follows:

[0067] (1) using the obtained model parameter ω * for the classifier to perform data classification operation on the reconstructed latent signal to obtain the predicted category label

[0068] (2) using the decoding unit D * of the trained variational autoencoder to transform the reconstructed latent signal into a reconstructed image , which is represented as

[0069] wherein the input signal of the classifier when performing data classification operation is the reconstructed latent signal and the original image is not required for data classification operation, which is helpful for privacy protection of the original image.

[0070] It is worth noting that the contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art.

[0071] In order to test the privacy protection performance of the method described in the embodiments of the present application, the following specific examples are used for illustration, in which the peak signal-to-noise ratio (PSNR) is used to measure the quality of the reconstructed image, and the unit is dB. In order to test the classification accuracy of the method described in the embodiments of the present application, the classifier is used to perform data classification on the reconstructed latent signal.

[0072] The data set used in testing the privacy protection performance and classification accuracy is CIFAR-10, and AdamW is used as the optimization algorithm of the privacy protection deep learning model. The number of training rounds of the privacy protection deep learning model is set to 50, the size of the batch sampling data set is set to 64, and the learning rate of the privacy protection deep learning model is set to 0.0004.

[0073] As shown in Figure 3 The four original images and the corresponding class labels randomly selected from the data set CIFAR-10 in the example of the present application are shown in Fig. 1, which are plane, car, cat and horse, respectively.

[0074] As shown in Figure 4 The reconstructed images corresponding to the original images and the corresponding PSNR values in the example of the present application are shown in Fig. 2. Figure 4 It can be seen that the content of the reconstructed image cannot be perceived.

[0075] The data classification of 10,000 reconstructed latent signals is performed using the classifier, and the classification accuracy is 53.36%. Table 1 below shows the classification accuracy of the reconstructed latent signals with different class labels.

[0076] Table 1 Classification accuracy of reconstructed latent signals with different class labels

[0077]

[0078] As can be seen from Table 1, the classifier has classification performance for reconstructed latent signals with different class labels, and does not need the original image for data classification operation, which is helpful for the privacy protection of the original image.

[0079] In summary, the method described in the embodiment of the present application can solve the problem of image data privacy leakage in the training process of the deep neural network, thereby achieving a good balance between classification accuracy and data privacy protection performance.

[0080] In addition, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be instructed by a program to complete the relevant hardware, and the corresponding program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0081] The above description is merely preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions easily conceived by those skilled in the art within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. The information disclosed in the background section of the present application is merely intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known to those skilled in the art.

Claims

1. A privacy-preserving deep learning method based on diffusion model and label reconstruction, characterized in that, The method comprises: Step 1, constructing a privacy protection deep learning model composed of a variational autoencoder, a diffusion model and a classifier; Wherein, the input image of the constructed privacy protection deep learning model is a color image with a size of 32x32, specifically comprising: The classifier is composed of the following units, respectively: a two-dimensional convolution operation unit Conv(in_channels, out_channels, kernel_size, stride, padding) with an input channel number of in_channels, an output channel number of out_channels, a convolution kernel size of kernel_size, a convolution kernel step of stride and a padding number of padding; a maximum value pooling operation unit MaxPool2d; a fully connected layer Linear(in_channels, out_channels) with an input channel number of in_channels and an output channel number of out_channels; The variational autoencoder is composed of an encoding unit Encoder of the variational autoencoder and a decoding unit Decoder of the variational autoencoder; The diffusion model comprises an attention mechanism-based down-sampling unit AttnDownBlock2D; an attention mechanism-based intermediate unit AttnMidBlock2D; an attention mechanism-based up-sampling unit AttnUpBlock2D; a down-sampling unit DownBlock2D; an up-sampling unit UpBlock2D; and a time step embedding unit Time embedding; Step 2, training the variational autoencoder using the original image and the target image, and transforming the original image into an original latent layer signal by the encoding unit of the variational autoencoder; The specific process of step 2 is: (1) The original image set is represented as x, and the blocking and random block scrambling operations are performed on all images in the original image set x to obtain a target image set (2) Defining the loss function of the variational autoencoder is defined as: where x denotes an original image in the original image set x; denotes a target image in the target image set E is an encoding unit of the variational autoencoder; D is a decoding unit of the variational autoencoder; denotes an expectation operation based on a statistical distribution of the original images; denotes a square of an l2 norm of a difference between an image generated by the original image sequentially passing through the encoding unit and the decoding unit and the target image. (3) training the variational autoencoder, i.e. minimizing the loss function of the variational autoencoder obtaining the trained encoding unit E * and the decoding unit D * is denoted as: wherein, denotes training the variational autoencoder to obtain an optimal encoding unit E * and a decoding unit D * operations; (4) using the trained encoding unit E * transforming the original image x into an original latent signal z = E * (x); Step 3, training the diffusion model using the original latent layer signal, transforming the original latent layer signal into a reconstructed latent layer signal by the diffusion model, and training the classifier based on the reconstructed latent layer signal and the category label of the original image; Step 4, performing data classification on the reconstructed latent layer signal using the trained classifier, and transforming the reconstructed latent layer signal into a reconstructed image using the decoding unit of the variational autoencoder. 2.The privacy-preserving deep learning method based on diffusion model and label reconstruction of claim 1, wherein, The specific process of step 3 is: (1) Set the maximum time step to T and generate Gaussian noise that obeys a distribution with mean 0 and variance I (2) Using Gaussian noise The noise addition operation is performed stepwise on the original latent layer signal z, denoted as: wherein is a Gaussian noise subject to a mean of 0 and a variance of I; z0= z is the noisy latent signal when t = 0, and 0 < a < 1 t < 1, z t is a noisy latent signal; (3) defining a diffusion model loss function is: wherein, is a diffusion model to add noise to the latent signal z t and time step t is a predicted noise generated by the input signal; (4) training the diffusion model, i.e., minimizing the diffusion model loss function obtaining the corresponding model parameters θ * : (5) Use predictive noise For the noisy latent layer signal z t The noise removal operation is performed step by step to obtain the predicted noisy latent layer signal. wherein is a Gaussian noise subject to a mean of 0 and a variance of I; is the mean of the predicted noisy latent signal is expressed as: And is the predicted noisy latent signal variance of the predicted noisy latent signal, denoted as: (6) obtaining a reconstructed latent signal wherein is the predicted noisy latent signal when t = 0; is the predicted noisy latent signal when t = 1 is the mean of is the predicted noisy latent signal when t = 1 is the variance of is a Gaussian noise with mean 0 and variance I when t = 1; (7) defining a loss function for the classifier is represented as: wherein m represents the number of images in the original image set x; c represents the number of image categories in the original image set x; y represents the real category label set of the original image; represents the predicted category label set obtained by the classifier through the reconstructed latent signal; y i (j) represents the real category label corresponding to the j-th category of the reconstructed latent signal; represents the predicted category label corresponding to the j-th category of the reconstructed latent signal; (8) training the classifier, i.e. minimizing the loss function of the classifier obtaining model parameters ω of the classifier * is denoted as 3.The privacy-preserving deep learning method based on diffusion model and label reconstruction of claim 2, wherein, The specific process of step 4 is: (1) using the resulting model parameters for the classifier * to perform a data classification operation on the reconstructed latent signal to obtain a predicted class label (2) the decoding unit D after training using a variational autoencoder * reconstructing the latent signal transforming into a reconstructed image is represented as The input signal of the classifier during the data classification operation is the reconstructed latent layer signal and the data classification operation does not require the original image, which helps to protect the privacy of the original image.

Images