A synthetic method of generative adversarial network data based on adaptive differential privacy

Abstract

The application discloses a kind of based on adaptive differential privacy's synthetic method of generative adversarial network data, can solve data synthesis process data utility and privacy problem, through the MAD-GAN and adaptive differential privacy technical scheme of the method, reach better privacy, improve the utility of data;Through adaptive gradient clipping threshold, the performance fluctuation in the process of deep neural network training is reduced, and the generalization ability of the model is improved;According to the corresponding noise scale, an adaptive noise allocation strategy is used to prevent excessive noise from being added, which can ensure the effectiveness of the generator during model training and guarantee the privacy protection process of parameters involving sensitive information;After the generator is trained, synthetic data that satisfies differential privacy can be generated;The synthetic data that satisfies differential privacy protection and the real data have high usability in the same task.

Description

Technical Field

[0001] This invention belongs to the field of edge device data generation methods, and specifically relates to a method for synthesizing generative adversarial network data based on adaptive differential privacy. Background Technology

[0002] Industry 4.0 is characterized by the integration of the Industrial Internet, artificial intelligence, big data distribution, machine learning, and deep learning, driving the next wave of digital transformation. In recent years, with the development of the Industrial Internet, vast amounts of collected data are being used through machine learning techniques to predict the aging or failure of production tools. Examples include equipment fault diagnosis, supply chain optimization, performance monitoring, and predictive maintenance. However, in the past few years, it has become clear that companies / factories are reluctant to share data for cross-organizational release and use, fearing the loss of competitive advantage by disclosing their trade secrets and sensitive information.

[0003] Therefore, establishing high-quality industrial internet data generation and applying it to data publishing and sharing remains a very challenging task.

[0004] Generative Adversarial Networks (GANs) and their variants, by combining the complexity of deep neural networks with game theory, have demonstrated impressive performance in modeling the underlying data distribution, generating synthetic data that is highly similar to the original dataset distribution, making it difficult to distinguish between real and fake data. Li et al. found that original GANs consistently suffer from modality collapse, insufficient diversity of generated samples, and vanishing gradients. Modality collapse refers to the fact that the generator network during GAN training can only generate limited sample diversity, failing to cover all patterns in the data distribution. This can lead to overly similar or undifferentiated samples output by the generator. To address these issues, this patent proposes a Multi-Agent Diversified Generative Adversarial Network (MAD-GAN) utilizing multiple generators. Ideally, we can generate high-quality synthetic data to protect the privacy of the original data. However, adversaries can still make further inferences about the generated data, allowing sensitive information to be inferred from the neural network model parameters through advanced inference attacks, thus causing data privacy breaches. To solve these problems, differential privacy is introduced into machine learning models to protect model privacy. However, adding differential privacy Gaussian random noise leads to the following problems: i) the synthetic data often has low utility, for example, unrealistic data; ii) its convergence speed is slower than that of regular GANs, resulting in excessive privacy loss, and sometimes even divergence. The reason behind these problems may lie in the gradient pruning threshold C: (i) due to the pruning operation, useful information is removed, especially for small C, which is included in the local parameter updates; (ii) the addition of random noise severely affects the local updates, leading to large differences between local update models, especially for large C. In other words, setting a pruning threshold that is too low or too high may lead to large differences between local models, resulting in performance degradation and reduced practicality. Summary of the Invention

[0005] The purpose of this invention is to address the aforementioned deficiencies in industrial internet data privacy protection technologies by providing a generative adversarial network data synthesis method based on adaptive differential privacy, which offers high privacy protection and strong data availability.

[0006] To address the issues of pattern collapse and data utility, this invention proposes a MAD-GAN containing k generators and one discriminator. Unlike traditional GANs, the discriminator's output is modified to output a k+1 softmax score. Specifically, k generators are provided, and the discriminator distinguishes between k types of generated samples and one type of real samples. To address the data utility issue, this patent proposes an adaptive gradient pruning method. Specifically, it adaptively thresholds the L2 norm of the discriminator network's gradient, ensuring that the MAD-GAN, after incorporating differential privacy, can generate an unlimited amount of high-quality synthetic data that conforms to the statistical characteristics of the source data without revealing privacy. Furthermore, MAD-GAN introduces a large amount of noise during training, and excessive noise can degrade model performance. To improve model performance and achieve a balance between privacy and usability, an adaptive noise allocation strategy is designed, allowing noise to be dynamically injected into the gradient at each iteration. Therefore, this invention proposes a data synthesis method based on adaptive gradient pruning and adaptive noise addition of a differential privacy-multi-agent diverse generative adversarial network (Adap DP-MAD-GAN).

[0007] This invention provides a method for synthesizing generative adversarial network data based on adaptive differential privacy, the method comprising the following steps: Step 1: Data Acquisition - Training Dataset and Discriminator The MAD-GAN model is used to normalize and standardize the initial data collected from the Industrial Internet to obtain a dataset; a dataset of size is selected from the dataset. Random sample data is fed into each discriminator network in the initial generative adversarial network;

[0008] Step 2 uses a discriminator to distinguish between the generated synthetic data and the randomly input real data, obtaining the discrimination result output by each discriminator network in the initial generative adversarial network; Step 3: Calculate the loss function of each discriminator network in the initial generator network and the L2 norm of the gradient information of each discriminator network in the initial generator adversarial network. Step 4 utilizes differential privacy to adaptively prune the gradient threshold and add noise to the discriminator network. The generator takes random noise as input to generate synthetic data similar to the training data. The synthetic data and real data are used as input to the discriminator network. Differential privacy is used to adaptively prune the gradient of the discriminator network and adaptively add noise. Step 5: Update the discriminator and generator; The discriminator network gradient information is adaptively pruned and noise-added using a pruning threshold to obtain the discriminator network in the next iteration. The generator network in the initial generative adversarial network is updated according to the updated discriminator network, and the updated generator adversarial network is used as the generator network in the next iteration; until the iteration reaches the current privacy budget threshold or the iteration round is reached to obtain the trained adversarial generative network.

[0009] Preferably, step 1 involves the following steps for data acquisition: training dataset and discriminator. Step 101 Initialize generator parameters Discriminator parameters ;

[0010] Step 102 Construct a multi-generational generative adversarial network based on the MAD-GAN model, consisting of k generators and one discriminator. The generators use random noise as input to generate the i-th batch of generated data. Step 103: Send the generator-generated data from the initial generative adversarial network to each discriminator network in each of the generative adversarial networks; Step 104 Select a dataset of size [size missing] from the actual dataset. Random sample data is fed into each discriminator network in the initial generative adversarial network;

[0011] Step 105: Through the discriminator The synthesized data and its corresponding real data are distinguished. K synthesized data are generated by k generators, and the discriminator... The output is k+1 softmax scores.

[0012] Preferably, step 3, calculating the loss function of each discriminator network in the initial generator network, is as follows: Step 201: Calculate the loss function of each discriminator in the initial generator network based on the output discrimination results. During the training phase, the generated samples need to be used with the j-th discriminator. We use the 0th label to distinguish between the real sample and the actual sample;

[0013] Step 202 Discriminator The differences between k classes of samples need to be evaluated, where the k classes of samples are synthetic data generated by k generators; the discriminator The loss function is adjusted to a multi-class cross-entropy loss function:

[0014] (1)

[0015] in, This represents the negative value of the cross-entropy function. Represents the Dirac distribution;

[0016] Step 203 When the discriminator When the model parameters remain constant, the loss of the j-th generator can be expressed as:

[0017] (2)

[0018] Gradient information refers to the calculation of the discriminator gradient. Let represent the gradient of the discriminator in the (t-1)th round.

[0019] Preferably, the L2 norm of calculating the gradient information of each discriminator network in the initial generative adversarial network in step 3 is as follows: Step 301: Solve for the adaptive gradient clipping threshold based on the gradient L2 norm. :

[0020] (3)

[0021] Where: the formula for solving the L2 norm of the gradient is: (4)

[0022] in, Let i be the i-th batch of samples. For the previous round The gradient value C calculated after training t-1 Indicates the first The wheel passes the clipping threshold;

[0023] The gradient solution formula is: (5)

[0024] Where, x i-1 This represents the samples used in round t-1. These represent the model parameters of the discriminator. It represents the loss function of the discriminator in the (t-1)th round;

[0025] Variance of Gaussian noise: (6)

[0026] in, For the first Wheel through clipping threshold Calculate the variance of the Gaussian noise. The size of the noise scale;

[0027] Step 302: Adaptively prune and add noise to the gradient information of the discriminator network using a pruning threshold. (7)

[0028] Step 303: Obtain the discriminator network for the next iteration according to formula (7): (8)

[0029] in, and Let these represent the discriminator parameters for the i-th round and the (i-1)-th round, respectively. This represents the gradient descent decay rate. This represents the gradient of the discriminator network after adaptive pruning thresholding and adding noise in round t.

[0030] Preferably, step 5 updates the generator and discriminator by updating the parameter information of the discriminator and generator according to formula (8); The parameters of the k generators are updated as follows: (9)

[0031] in, This represents the model parameters of the j-th generator; This represents the gradient of the j-th generator; This represents the learning rate of the j-th generator;

[0032] This discriminator is used to classify the j-th index... The parameter update is simplified to:

[0033] (10)

[0034] in, This represents the model parameters of the discriminator; Discriminator The gradient with respect to the j-th index; Discriminator The learning rate for the j-th metric.

[0035] Preferably, the update generator and discriminator in step 5 further include the following steps: Step 501: Dynamically adjust the noise scale based on the changes in the loss function of the adversarial generative network; Adjusting the noise scale: (11)

[0036] Where k is the current round, For the current noise scale, Let represent the loss function for round k. Then, to obtain the noise scale for round k+1;

[0037] Step 502 uses Rényi Accountant to calculate about Privacy Budget This continues until the current privacy budget threshold is reached or the trained generative adversarial network is obtained after a certain number of iterations.

[0038] Preferably, step 503 uses Rényi Accountant to calculate about Privacy Budget The process continues until the current privacy budget threshold is reached or the trained generative adversarial network is obtained after a certain number of iterations.

[0039] For two probability distributions P and Q defined on R, the Renyi divergence of order α>1 is: (12)

[0040] Where all logarithms are natural; P(x) is the probability density of P at x; if If the value is ∞, then the original definition of differential privacy is restored.

[0041] Preferably, the MAD-GAN model in step 1 consists of K generators and one discriminator, wherein the feature extractor is built based on a one-dimensional CNN, including three convolutional (Conv) layers, three batch normalization (BN) layers and three max pooling (MP) layers.

[0042] Preferably, the generator parameters are initialized in step 101. Discriminator parameters First, a discriminator is trained by inputting initial data into the j-th batch of real data. After n samplings, the initial data is used to make predictions and obtain... Score;

[0043] The generator uses random noise As input, generate the i-th batch of images of equal number and same dimensions, which are used by the discriminator for prediction, and the score is... .

[0044] Preferably, in step 1, the MAD-GAN model is used to normalize and standardize the initial industrial internet data collected. Obtaining the dataset refers to normalizing and standardizing the input industrial internet data containing sensitive information. The normalization is performed using the Min-Max normalization method, and the specific formula is as follows: (13)

[0045] The data standardization methods include linear, polygonal, and curvilinear methods; different standardization methods will have different impacts on the evaluation results of the system; among them, Z-Score standardization; (14)

[0046] in, Represents the original data. This represents the standard deviation. This represents the mean.

[0047] This invention provides a method for synthesizing differential privacy generative adversarial network (GAN) data, which addresses the issues of data utility and privacy in the data synthesis process. By employing MAD-GAN and an adaptive differential privacy scheme, this method achieves better privacy and improves data utility. Adaptive gradient pruning thresholds reduce performance fluctuations during deep neural network training, enhancing the model's generalization ability. An adaptive noise allocation strategy based on the corresponding noise scale prevents the addition of excessive noise, ensuring both the generator's performance during model training and the privacy protection of parameters involving sensitive information. After final training, the generator can produce differential privacy-compliant data, unaffected by the size of the original dataset. The resulting differential privacy-compliant synthesized data exhibits high usability on the same task as real data. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating an embodiment of a generative adversarial network data synthesis method based on adaptive differential privacy. Figure 2 This is a flowchart illustrating an embodiment of a generative adversarial network data synthesis method based on adaptive differential privacy. Figure 3 This is a flowchart illustrating the differential privacy noise adaptive threshold calculation method implemented in a generative adversarial network data synthesis method based on adaptive differential privacy. Figure 4 This is a flowchart illustrating the noise scale attenuation calculation process for a generative adversarial network data synthesis method based on adaptive differential privacy. Figure 5 This is a schematic diagram of the confusion matrix result of an application example of a generative adversarial network data synthesis method based on adaptive differential privacy; Figure 6 This is a flowchart of a generative adversarial network data synthesis method based on adaptive differential privacy. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention.

[0050] This invention provides a method for synthesizing generative adversarial network data based on adaptive differential privacy, such as... Figure 6 As shown, the method includes the following steps: Step 1: Data Acquisition - Training Dataset and Discriminator; The MAD-GAN model is used to normalize and standardize the initial data collected from the Industrial Internet to obtain a dataset; a dataset of size is selected from the dataset. Random sample data is fed into each discriminator network in the initial generative adversarial network;

[0051] Step 2 uses a discriminator to distinguish between the generated synthetic data and the randomly input real data, obtaining the discrimination result output by each discriminator network in the initial generative adversarial network; Step 3: Calculate the loss function of each discriminator network in the initial generator network and the L2 norm of the gradient information of each discriminator network in the initial generator adversarial network. Step 4 utilizes differential privacy to adaptively prune the gradient threshold and add noise to the discriminator network. The generator takes random noise as input to generate generated data similar to the training data. The synthesized generated data and real data are used as input to the discriminator network. Differential privacy is used to adaptively prune the gradient of the discriminator network and adaptively add noise. Step 5 updates the discriminator and generator by adaptively pruning and adding noise to the gradient information of the discriminator network using a pruning threshold, thus obtaining the discriminator network for the next iteration. The generator network in the initial generative adversarial network is updated according to the updated discriminator network, and the updated generator adversarial network is used as the generator network in the next iteration; until the iteration reaches the current privacy budget threshold or the iteration round is reached to obtain the trained adversarial generative network.

[0052] In one embodiment, step 1 involves data acquisition of the training dataset and discriminator as follows: Step 101 Initialize generator parameters and discriminator parameters ;

[0053] Step 102: Construct a multi-agent generative adversarial network based on the MAD-GAN model, consisting of k generators and one discriminator. The generators use random noise as input to generate the i-th batch of generated data. Step 103: Send the generator-generated data from the initial generative adversarial network to each discriminator network in each of the generative adversarial networks; Step 104: Select a dataset of size [size missing] from the actual dataset. Random sample data is fed into each discriminator network in the initial generative adversarial network;

[0054] Step 105: Through the discriminator The synthesized data and its corresponding real data are distinguished. Since k generators generate k synthesized data, the discriminator... The output is modified to output k+1 (k represents the number of generators) softmax scores.

[0055] In one embodiment, step 3, calculating the loss function for each discriminator network in the initial generator network, is as follows: Step 201: Calculate the loss function of each discriminator in the initial generator network based on the output discrimination results. During the training phase, the generated samples need to be used with the j-th discriminator. We use the 0th label to distinguish between the real sample and the actual sample;

[0056] Step 202 Discriminator The differences between k classes of samples need to be evaluated, where the k classes are synthetic data generated by k generators; the discriminator... The loss function is adjusted to a multi-class cross-entropy loss function:

[0057] (1)

[0058] in, This represents the negative value of the cross-entropy function. Represents the Dirac distribution; discriminator This means that it can not only distinguish between real samples and generated samples, but also distinguish between k types of generated synthetic data samples;

[0059] Step 203 When the discriminator When the model parameters remain constant, the loss of the j-th generator can be expressed as:

[0060] (2)

[0061] Gradient information refers to the calculation of the discriminator gradient. Let represent the gradient of the discriminator in the (t-1)th round.

[0062] In one embodiment, the step 3 of calculating the L2 norm of the gradient information of each discriminator network in the initial generative adversarial network is as follows: Step 301: Solve for the adaptive gradient clipping threshold based on the gradient L2 norm. :

[0063] (3)

[0064] Where: the formula for solving the L2 norm of the gradient is: (4)

[0065] in, Let i be the i-th batch of samples. For the previous round The gradient value C calculated after training t-1 Indicates the first The wheel passes the clipping threshold;

[0066] The gradient solution formula is: (5)

[0067] Among them, X i-1 Let represent the samples used in round t-1, and Ꞷ represent the model parameters of the discriminator. It represents the loss function of the discriminator in the (t-1)th round;

[0068] Variance of Gaussian noise: (6)

[0069] in, For the first Wheel through clipping threshold Calculate the variance of the Gaussian noise. The size of the noise scale;

[0070] Step 302: Adaptively prune and add noise to the gradient information of the discriminator network using a pruning threshold. (7)

[0071] Step 303: Obtain the discriminator network for the next iteration according to formula (7): (8)

[0072] in, and These represent the discriminator parameters for the i-th and (i-1)-th rounds, respectively. This represents the gradient descent decay rate. This represents the gradient of the discriminator network after adaptive pruning thresholding and adding noise in round t.

[0073] In one embodiment, step 5, updating the generator and discriminator, involves updating the parameter information of the discriminator and generator according to formula (8).

[0074] The parameters of the k generators are updated as follows: (9)

[0075] in, This represents the model parameters of the j-th generator; This represents the gradient of the j-th generator; This represents the learning rate of the j-th generator;

[0076] This discriminator is used to classify the j-th index... The parameter update is simplified to:

[0077] (10)

[0078] This represents the model parameters of the discriminator; Discriminator The gradient with respect to the j-th index; Discriminator The learning rate for the j-th metric.

[0079] In one embodiment, the update generator and discriminator in step 5 further include the following steps: Step 501: Dynamically adjust the noise scale based on the changes in the loss function of the adversarial generative network; Adjusting the noise scale: (11)

[0080] Where k is the current round, For the current noise scale, Let represent the loss function for round k. Then, to obtain the noise scale for round k+1;

[0081] Step 502 uses Rényi Accountant to calculate about Privacy Budget This continues until the current privacy budget threshold is reached or the trained generative adversarial network is obtained after a certain number of iterations.

[0082] In one embodiment, step 503 uses Rényi Accountant to calculate about Privacy Budget The process continues until the current privacy budget threshold is reached or the trained generative adversarial network is obtained after a certain number of iterations.

[0083] For two probability distributions P and Q defined on R, the Renyi divergence of order α>1 is: (12)

[0084] Where all logarithms are natural; P(x) is the probability density of P at x; if If the value is ∞, then the original definition of differential privacy is restored.

[0085] In one embodiment, the MAD-GAN model in step 1 consists of K generators and one discriminator, wherein the feature extractor is built based on a one-dimensional CNN, including three convolutional (Conv) layers, three batch normalization (BN) layers and three max pooling (MP) layers.

[0086] Among them, the initialization generator parameters in step 101 and discriminator parameters First, a discriminator is trained by inputting initial data into the j-th batch of real data. After n samplings, the initial data is used to make predictions and obtain... Score;

[0087] The generator uses random noise As input, generate the i-th batch of images of equal number and same dimensions, which are used by the discriminator for prediction, and the score is... .

[0088] In one embodiment, step 1 uses the MAD-GAN model to normalize and standardize the initial industrial internet data to obtain the dataset. This means normalizing and standardizing the input industrial internet data containing sensitive information. The normalization is performed using the Min-Max normalization method, and the specific formula is as follows: (13)

[0089] Data standardization methods include linear methods (such as extreme value method and standard deviation method), piecewise linear methods (such as three-segment method), and curvilinear methods (such as semi-normal distribution); different standardization methods will have different effects on the evaluation results of the system; among them, the most commonly used is Z-Score standardization; (14)

[0090] in, Represents the original data. This represents the standard deviation. This represents the mean.

[0091] This invention provides a method for synthesizing differential privacy generative adversarial network (GAN) data. By employing adaptive differential privacy and MAD-GAN techniques, this method addresses the issues of data utility and privacy during the data synthesis process, achieving better privacy and improving data utility.

[0092] Example 1 To address the aforementioned issues, differential privacy has been introduced into machine learning models to protect model privacy. However, adding Gaussian random noise to differential privacy models can lead to the following problems: i) synthetic data often has low utility, such as unrealistic data; ii) its convergence speed is slower than that of regular GANs, resulting in excessive privacy loss and sometimes even divergence. The underlying reason for these problems may lie in the gradient pruning threshold C: (i) due to the pruning operation, useful information is removed, especially for small C values, which are included in local parameter updates; (ii) the addition of random noise severely affects local updates, leading to significant differences between local update models, especially for larger C values. In other words, setting a pruning threshold that is too low or too high can cause large differences between local models, resulting in performance degradation and reduced practicality.

[0093] This invention provides a method for synthesizing generative adversarial network data based on adaptive differential privacy. The working principle of this invention is as follows: Figure 1 As shown, differential privacy is used to adaptively prune the gradient of the discriminator network and add noise to update the corresponding gradient information. Specifically, it can be summarized as: "Calculate the adaptive gradient pruning threshold based on the L2 norm of the discriminator in the generative adversarial network -> Adaptively prune the gradient information of the discriminator network using the pruning threshold -> Adaptively add noise -> Update the discriminator -> Update the generator."

[0094] The MAD-GAN model used in this embodiment of the invention consists of multiple generators and one discriminator. The feature extractor is built on a one-dimensional CNN, including three convolutional (Conv) layers, three batch normalization (BN) layers, and three max pooling (MP) layers. Additionally, a flattened layer and two fully connected layers use convolutional modules to obtain deep features. The discriminator is connected to the two fully connected layers and uses binary cross-entropy to calculate the loss value on the output layer. This technique can be used to solve the mode collapse problem. By calculating the gradient of each discriminator, the L2 norm of the gradient in that round is solved. If the L2 norm of the gradient of the training sample in that round is greater than the pruning threshold C, then each gradient of the sample is proportionally reduced by dividing the coefficient C by the L2 norm of the sample's gradient. By calculating the adaptive gradient pruning threshold for each round, we can obtain the sensitivity of the overall gradient to the increase or decrease of samples in each round, thus adding Gaussian noise to the overall gradient to achieve high data privacy and utility.

[0095] like Figure 2 As shown, this invention provides a data synthesis method based on generative adversarial networks, which includes the following steps: Step 1: Data Acquisition and Discriminator Training (Initializing Generator Parameters) and discriminator parameters );

[0096] 1. Normalize and standardize the input industrial internet data containing sensitive information; Common methods for normalization include Min-Max normalization. (13)

[0097] There are various data standardization methods, such as linear methods (e.g., extreme value method, standard deviation method), piecewise linear methods (e.g., three-segment method), and curvilinear methods (e.g., semi-normal distribution). Different standardization methods will have different effects on the evaluation results of the system. Among them, Z-Score standardization is the most commonly used.

[0098] (14)

[0099] 2. The generator uses noise As input, a batch of images of equal number and same dimensions are generated for the discriminator to predict scores.

[0100] 3. First, train the discriminator. Its input includes a batch of real data. The discriminator downsamples the data multiple times and makes predictions on the batch of data to obtain a score.

[0101] 4. Based on the scores, calculate the loss value of each discriminator network in the initial generative adversarial network using the multi-class cross-entropy loss function; (1)

[0102] in, This represents the negative value of the cross-entropy function. Represents the Dirac distribution. Discriminator. It can not only distinguish between real samples and generated samples, but also distinguish between k types of generated synthetic data samples.

[0103] Step 2: Construct a multi-agent generative adversarial network (MAD-GAN) model, in which multiple generators generate a batch of data with random noise distribution; Step 3: Send the synthesized data of the generator in the initial generative adversarial network to each discriminator network in each of the generative adversarial networks; Step 4: Select a dataset of size [size missing] from the dataset. Random sample data is fed into each discriminator network in the initial generative adversarial network;

[0104] Step 5: The generated data and the corresponding local data (real data) are distinguished by the discriminator to obtain the discrimination result output by each discriminator network in the initial generative adversarial network; Step 6: Calculate the loss function of each discriminator in the initial generator network based on the output discrimination results; Step 7: Solve for the adaptive gradient clipping threshold based on the gradient L2 norm. Step 8: Calculate the L2 norm of the gradient information of each discriminator network in the initial generative adversarial network according to the loss function of each discriminator network; Calculate the gradient information of each discriminator network in the initial generative adversarial network based on the loss function of each discriminator network in the initial generative adversarial network; The loss of the generator can be expressed as: (2)

[0105] Gradient information mainly refers to the calculation of the discriminator gradient. Let represent the gradient of the discriminator in the (t-1)th round.

[0106] To balance the relationship between data privacy and availability, differential privacy perturbation is applied to the discriminator gradient information, and an adaptive pruning threshold is first performed. The adaptive gradient threshold clipping mainly involves adaptive threshold clipping of the gradient L2 norm. Calculate the adaptive gradient clipping threshold based on the L2 norm of the gradient. (3)

[0107] in: The formula for solving the L2 norm of the gradient is as follows: (4)

[0108] in, Let i be the i-th batch of samples. For round t-1 The gradient value calculated after training.

[0109] The gradient solution formula is as follows (5)

[0110] Variance of Gaussian noise: (6)

[0111] in, For the first Wheel through clipping threshold Calculate the variance of the Gaussian noise. The size of the noise scale;

[0112] This invention presents an adaptive differential privacy-oriented generative adversarial network (GAN) data generation method. It proposes a MAD-GAN comprising k generators and one discriminator, modifying the discriminator's output to produce a k+1 softmax score to address the mode collapse problem. The interaction between the discriminator and generator training on the original data is analyzed. Adaptive threshold pruning and the addition of noise at appropriate scales are performed based on the L2 norm of the discriminator's gradient information, ensuring that the perturbation takes privacy into account, thereby improving the effectiveness of data synthesis.

[0113] like Figure 3 As shown, the adaptive gradient clipping threshold for each round is solved. The specific steps are as follows:

[0114] a. Calculate the L2 range of each discriminator network in the adversarial generative network; b. Calculate the gradient pruning threshold for the current round based on the L2 range. The formula for calculating the pruning threshold can be expressed as follows: (3)

[0115] The formula for solving the L2 norm of the gradient is as follows: (4)

[0116] c. Update the gradient of the discriminator in the current generative adversarial network and add it according to the corresponding noise scale; in this step... (6)

[0117] in, The cropping threshold, For the previous round The gradient values ​​calculated after training. The size of the noise scale. For the first Wheel through clipping threshold Calculate the variance of the Gaussian noise;

[0118] d. Obtain the gradient information after perturbation.

[0119] Step 9: Use differential privacy to adaptively prune and add noise to the gradient information of the discriminator network to obtain the discriminator network in the next iteration; Step 10: Update the generator network in the initial generative adversarial network according to the discriminator network with updated parameters, and obtain the updated generator adversarial network as the generator network in the next iteration. Step 11: Dynamically adjust the noise scale based on the changes in the loss function of the adversarial generative network. The specific steps are as follows: Figure 4 As shown; a. Determine and compare the loss functions of the adversarial generative network at rounds t, t-1, t-2, and t-3, respectively. b. Calculate the attenuation rate of the corresponding noise scale based on the proportion of the loss function; Noise scale adjustment formula as follows (11)

[0120] Where k is the current round, For the current noise scale, Let represent the loss function for the k-th round. Then, to obtain the noise scale for the (k+1)-th round, the above formula is used for updating.

[0121] c. Obtain the noise scale attenuation rate for each round; When it is found that the loss function of the adversarial generative network has decreased three times consecutively, then according to the decay rate... We performed a decay process. We found that the loss function might not decrease monotonically with the number of training iterations, and its fluctuations could lead to unnecessary noise scaling reduction, resulting in a large privacy budget. This prompted us to use three consecutive decreases in the loss function as the criterion for noise scaling reduction.

[0122] Step 12: Calculate the privacy budget using Rényi Accountant. Privacy Budget This continues until the current privacy budget threshold is reached or the trained generative adversarial network is obtained after a certain number of iterations.

[0123] Using Rényi Accountant to calculate about Privacy Budget This process continues until the current privacy budget threshold is reached or the trained generative adversarial network is obtained after a certain number of iterations.

[0124] For two probability distributions P and Q defined on R, the Renyi divergence of order α>1 is: (12)

[0125] (All logarithms are natural; P(x) is the probability density of P at x.) If If the value is ∞, then the original definition of differential privacy is restored.

[0126] Example 2 The effects of the present invention will be further explained below with reference to simulation experiments.

[0127] 1. Simulation experimental conditions: The hardware platform for the simulation experiment of this invention is: a Tesla T4 GPU processor and 16GB of memory.

[0128] The software platform for the simulation experiment of this invention is: Linux 64-bit distribution Linux Ubuntu 20.04 operating system and Python 3.6.

[0129] The dataset used in the simulation experiments of this invention is the experimental bearing dataset from the Bearing Data Center of Case Western Reserve University (CWRU).

[0130] The bearing test platform consists of: 1) a motor, 2) an accelerometer, 3) a torque sensor, and 4) a force gauge. An accelerometer was installed on the SKF6205 drive-end bearing, which recorded vibration signals at a sampling rate of 12 kHz. Four speed conditions and ten fault categories were included. The collected fault data were categorized into internal faults, ball faults, and external faults, with defect sizes of 0.1778, 0.3556, and 0.5334 mm, respectively. Furthermore, to improve fault identification capabilities, outer ring fault data was obtained from the fault point at the 6 o'clock position of the outer ring. According to experimental requirements, 20 samples with 400 vibration points were selected from 300 samples in each healthy state.

[0131] This invention sets the local momentum coefficient to 0.5, the local learning rate to η=0.1, and the batch size to... =20, local update period is 10 epochs, communication rounds T=200. Privacy parameters: For the CWRU dataset, the noise multiplier is set to σ=1.5 by default.

[0132] The performance of the proposed method at different rotational speeds in this experimental data was evaluated using accuracy (Acc), precision (Precision), recall (Recall), and F1 score, as shown in Table 1.

[0133] Table 1. Performance analysis of different comparison methods on the CWRU dataset.

[0134] To clearly demonstrate the effectiveness of this method, we used a confusion matrix, such as... Figure 5 As shown, this method performs well across all categories, generating efficient synthetic data with privacy protection, which is of great significance for engineering applications.

[0135] This invention provides a method for synthesizing differential privacy generative adversarial network (GAN) data. By employing adaptive differential privacy and MAD-GAN techniques, this method addresses the issues of data utility and privacy during the data synthesis process, achieving better privacy and improved data utility. Specific beneficial effects are as follows: 1. This invention proposes a differential privacy adversarial generative network scheme with adaptive gradient pruning. This scheme reduces the performance fluctuation during the training process of deep neural networks by using an adaptive gradient pruning threshold, improves the generalization ability of the model, and balances the trade-off between privacy loss and model performance. 2. This invention proposes a method for adaptive noise allocation based on the corresponding noise scale, which aims to prevent the addition of excessive noise, thereby ensuring both the generator's performance during model training and the privacy protection of parameters involving sensitive information. 3. In this invention, the generator module can generate data that satisfies differential privacy after the final training is completed, without being limited by the size of the original dataset; 4. The privacy-de-identified synthetic data obtained by this invention exhibits high usability on the same task as randomly input real data.

Claims

1. A method for synthesizing generative adversarial network data based on adaptive differential privacy, characterized in that, The method includes the following steps: Step 1: Data Acquisition - Training Dataset and Discriminator; The MAD-GAN model is used to normalize and standardize the initial data collected from the Industrial Internet to obtain a dataset; random sample data is then selected from the dataset. To each discriminator network in the initial generative adversarial network; Step 2 uses a discriminator to distinguish between the generated synthetic data and the randomly input real data, obtaining the discrimination result output by each discriminator network in the initial generative adversarial network; Step 3: Calculate the loss function and L2 norm of the gradient information for each discriminator network in the initial generator network. Step 301 involves solving for the adaptive gradient clipping threshold based on the gradient L2 norm. : （3） Where: the formula for solving the L2 norm of the gradient is: （4） in, Let i be the i-th batch of samples. For the previous round The gradient value C calculated after training t-1 Indicates the first The wheel passes the clipping threshold; The gradient solution formula is: （5） Where, x i-1 This represents the samples used in round t-1. These represent the model parameters of the discriminator. It represents the loss function of the discriminator in the (t-1)th round; Variance of Gaussian noise: (6) in, For the first Wheel through clipping threshold Calculate the variance of the Gaussian noise. The size of the noise scale; Step 4 utilizes differential privacy to adaptively prune the threshold and add noise to the discriminator network gradient; The generator takes random noise as input to generate synthetic data similar to the training data. The synthetic data and real data are used as input to the discriminator network. Differential privacy is used to adaptively prune the gradient of the discriminator network and adaptively add noise. Step 5: Update the discriminator and generator The discriminator network gradient information is adaptively pruned and noise-added using a pruning threshold to obtain the discriminator network for the next iteration; the iteration is performed by calculating the gradient information of the discriminator network using the Rényi Accountant algorithm. Privacy Budget The iteration continues until the current privacy budget threshold is reached or the trained adversarial generative network is obtained after a certain number of iterations. The generator network in the initial generative adversarial network is updated according to the updated discriminator network, and the updated generator adversarial network is used as the generator network in the next iteration; until the iteration reaches the current privacy budget threshold or the iteration round is reached to obtain the trained adversarial generative network.

2. The method for synthesizing generative adversarial network data based on adaptive differential privacy according to claim 1, characterized in that, Step 1: Data Acquisition and Training of the Dataset and Discriminator. The steps are as follows: Step 101 Initialize generator parameters Discriminator parameters ; Step 102 Construct a multi-generational generative adversarial network based on the MAD-GAN model, consisting of k generators and one discriminator. The generators use random noise as input to generate the i-th batch of generated data. Step 103: Send the generator-generated data from the initial generative adversarial network to each discriminator network in each of the generative adversarial networks; Step 104 Select a dataset of size [size missing] from the actual dataset. Random sample data is fed into each discriminator network in the initial generative adversarial network; Step 105: Through the discriminator The synthesized data and corresponding real data are distinguished. K synthesized data are generated by k generators. The distinguisher... The output is k+1 softmax scores.

3. The method for synthesizing generative adversarial network data based on adaptive differential privacy according to claim 1, characterized in that, The steps for calculating the loss function of each discriminator network in the initial generator network in step 3 are as follows: Step 201: Calculate the loss function of each discriminator in the initial generator network based on the output discrimination results. During the training phase, the generated samples need to be used with the j-th discriminator. We use the 0th label to distinguish between the real sample and the actual sample; Step 202 Discriminator The differences between k classes of samples need to be evaluated, where the k classes of samples are synthetic data generated by k generators; the discriminator The loss function is adjusted to a multi-class cross-entropy loss function: （1） in, This represents the negative value of the cross-entropy function. Represents the Dirac distribution; Step 203 When the discriminator When the model parameters remain constant, the loss of the j-th generator can be expressed as: （2） Gradient information refers to the calculation of the discriminator gradient. Let represent the gradient of the discriminator in the (t-1)th round.

4. The method for synthesizing generative adversarial network data based on adaptive differential privacy according to claim 1, characterized in that, The step of calculating the L2 norm of the gradient information of each discriminator network in the initial generative adversarial network in step 3 further includes: Step 302: Adaptively prune and add noise to the gradient information of the discriminator network using a pruning threshold. （7） Step 303: Obtain the discriminator network for the next iteration according to formula (7): （8） in, and Let these represent the discriminator parameters for the i-th round and the (i-1)-th round, respectively. This represents the gradient descent decay rate. This represents the gradient of the discriminator network after adaptive pruning thresholding and adding noise in round t.

5. The method for synthesizing generative adversarial network data based on adaptive differential privacy according to claim 1, characterized in that, Step 5 updates the generator and discriminator by updating the parameter information of the discriminator and generator according to formula (8); The parameters of the k generators are updated as follows: （9） in, This represents the model parameters of the j-th generator; This represents the gradient of the j-th generator; This represents the learning rate of the j-th generator; The discriminator is used to classify the j-th index... The parameter update is simplified to: （10） This represents the model parameters of the discriminator; Discriminator The gradient with respect to the j-th index; Discriminator The learning rate for the j-th metric.

6. The method for synthesizing generative adversarial network data based on adaptive differential privacy according to claim 5, characterized in that, The update generator and discriminator in step 5 also include the following steps: Step 501: Dynamically adjust the noise scale based on the changes in the loss function of the adversarial generative network; Adjusting the noise scale: (11) Where k is the current round, For the current noise scale, Let represent the loss function for round k; then, in order to obtain the noise scale for round k+1.

7. The method for synthesizing generative adversarial network data based on adaptive differential privacy according to claim 6, characterized in that, Step 502 uses Rényi Accountant to calculate about Privacy Budget The process continues until the current privacy budget threshold is reached or the number of iterations is reached to obtain the trained adversarial generative network: for two probability distributions P and Q defined on R, The Renyi divergence of order is: （12） Where all logarithms are natural; P(x) is the probability density of P at x; if If the value is ∞, then the original definition of differential privacy is restored.

8. The method for synthesizing generative adversarial network data based on adaptive differential privacy according to claim 1, characterized in that, The MAD-GAN model in step 1 consists of K generators and one discriminator. The feature extractor is built based on a one-dimensional CNN, which includes three convolutional Conv layers, three batch normalized (BN) layers, and three max pooling (MP) layers.

9. The method for synthesizing generative adversarial network data based on adaptive differential privacy according to claim 2, characterized in that, Initializing generator parameters in step 101 Discriminator parameters First, a discriminator is trained by inputting initial data into the j-th batch of real data. After n samplings, the initial data is used to make predictions and obtain... Score; The generator uses random noise As input, generate the i-th batch of images of equal number and same dimensions, which are used by the discriminator for prediction, and the score is... .

10. The method for synthesizing generative adversarial network data based on adaptive differential privacy according to claim 1, characterized in that, Step 1 employs the MAD-GAN model to normalize and standardize the initial industrial internet data collected. Obtaining the dataset involves normalizing and standardizing the input industrial internet data containing sensitive information. The normalization is performed using the Min-Max normalization method, with the specific formula as follows: （13） The data standardization methods include linear, polygonal, and curvilinear methods; different standardization methods will have different impacts on the evaluation results of the system; among them, Z-Score standardization; （14） in, Represents the original data. This represents the standard deviation. This represents the mean.

Images