Differential privacy deep learning method for hierarchical adaptive noise addition

Abstract

The invention provides a differential privacy deep learning method for hierarchical adaptive noise addition, and belongs to the technical field of differential privacy deep learning. The method comprises the following steps: representing the influence degree of gradient change of each layer of a model on a loss function by conductivity of each layer, and taking the influence degree as the influence weight of each layer of the model on final output; distributing a privacy budget for each layer according to the influence weight; independently cutting different gradient norms of each layer; and by minimizing the model loss and the sharpness of the model loss, searching a flat minimum value to complete model training. According to the method, through differential noise addition based on hierarchy importance, excessive damage of unified noise addition to key features is avoided, and the model performance is effectively reserved while privacy is protected; the robustness and generalization ability of the model are further improved in combination with hierarchical cutting and sharpness perception optimization; meanwhile, through hierarchical privacy dependency modeling, interlayer sensitivity is strictly constrained, and it is ensured that RDP privacy guarantee is met globally.

Description

Technical Field

[0001] This invention relates to the field of differential privacy deep learning technology, and in particular to a hierarchical adaptive noise-adding differential privacy deep learning method. Background Technology

[0002] Differential privacy deep learning techniques protect individual data privacy by injecting specific noise during model training. This method requires that the output be insensitive to the addition or removal of any single record in the input dataset, thus ensuring that attackers cannot infer sensitive information about specific individuals by analyzing the model output. In deep learning, differential privacy is typically achieved through gradient perturbation mechanisms. This involves adding random noise that meets differential privacy requirements to the calculated gradient during backpropagation, and then using the perturbed gradient to update the model parameters. The core of this mechanism lies in reasonably controlling the noise intensity to satisfy both privacy budget constraints and maintain model usability.

[0003] Current differential privacy deep learning schemes generally employ a uniform noise strategy, applying the same intensity of noise perturbation to the gradients of all network layers throughout the training process. This neglects two key characteristics of deep learning models: First, different network layers have different functional importance in feature extraction; shallow networks typically capture general features, while deep networks are responsible for high-level features specific to the task. Second, data distribution in real-world applications is often non-uniform, including significant differences between feature dimensions and imbalances in sample classes. The uniform noise strategy cannot adapt to these complexities, making it difficult to achieve an optimal balance between privacy protection and model utility. Specifically, applying excessive noise to important network layers can destroy key gradient information, leading to decreased model accuracy and slower convergence; while insufficient noise to non-critical layers results in inadequate privacy protection, posing a potential risk of sensitive information leakage. These problems are particularly prominent in applications such as medical image analysis and financial risk control, where both privacy protection and model accuracy are critical.

[0004] Therefore, a hierarchical adaptive noise-adding differential privacy deep learning method is needed. Summary of the Invention

[0005] In view of this, the present invention provides a differential privacy deep learning method with hierarchical adaptive noise addition, which balances privacy protection and model utility by adding noise differentially based on hierarchical importance.

[0006] Therefore, the present invention provides the following technical solution: A hierarchical adaptive noise-adding differential privacy deep learning method includes: The transmissibility of each layer is used to characterize the degree of influence of gradient changes in each layer of the model on the loss function, and is used as the weight of the influence of each layer of the model on the final output. A privacy budget is allocated to each layer based on the aforementioned influence weights; Independently prune the gradient norms of each layer; By minimizing both the model loss and the sharpness of the model loss, a flat minimum is found to complete model training.

[0007] Furthermore, the conductivity includes: Calculate the integral gradient of the input features; The activation values ​​of each layer's path are obtained based on the completeness of the integral gradient; Calculate the conductivity of each layer based on the path activation values ​​of each layer; The discrete form of the conductance of each layer is obtained based on the completeness of the integral gradient, which serves as the influence weight of each layer on the final output.

[0008] Further, the calculation of the integral gradient of the input features includes: The integral gradient of an input feature is equal to the difference between the feature and the baseline sample multiplied by the integral of the gradient of that feature on the linear interpolation path from the baseline sample to the input sample.

[0009] Furthermore, obtaining the path activation values ​​of each layer based on the completeness of the integral gradient includes: The activation value on the path is equal to the activation value of the layer on the baseline sample plus the interpolation path parameter multiplied by the difference between the activation value of the layer on the input sample and the baseline activation value.

[0010] Further, the calculation of the conductance of each layer based on the path activation values ​​of each layer includes: No. The conductivity of a layer is the sum of the products of the activation difference of each activation unit in that layer and the corresponding elements of the path-sensitive gradient; Activation difference refers to the difference between the activation values ​​of the activation unit on the input sample and the baseline sample. Path-sensitive gradient refers to the integral of the gradient of the activation unit by the model output on the linear interpolation path from the baseline to the input.

[0011] Furthermore, obtaining the discrete form of the conductance of each layer based on the completeness of the integral gradient includes: No. The discrete form of the conductance of a layer is equal to the activation difference of each activation unit in that layer multiplied by the average gradient of that activation unit to the model output along the discretization path, and then summed over all activation units.

[0012] Advantages and positive effects of the present invention: This method avoids excessive destruction of key features by applying noise differentially based on hierarchical importance, effectively preserving model performance while protecting privacy. Combining hierarchical pruning and sharpness-aware optimization further enhances the model's robustness and generalization ability. Simultaneously, hierarchical privacy dependency modeling strictly constrains inter-layer sensitivity, ensuring global compliance with RDP privacy protection. This achieves a better balance between privacy protection and model utility, making it suitable for deep learning applications in scenarios with high privacy requirements. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of the differential privacy deep learning method with hierarchical adaptive noise addition according to an embodiment of the present invention. Figure 2 This is a framework diagram of the differential privacy deep learning method with hierarchical adaptive noise addition according to an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] This invention provides a hierarchical adaptive noise-adding differential privacy deep learning method, which dynamically adjusts the noise intensity based on the importance of each layer of the model, rather than using uniform noise addition. Specifically, it includes the following steps: 1. Hierarchical Importance Assessment Mechanism: The influence of gradient changes at each layer on the loss function is calculated using an interpretable algorithm, thus quantifying the weights of each layer of the deep learning model on the final output.

[0018] 1) According to the integral gradient theory, the integral gradient of the input layer features is calculated as follows: The integral gradient of an input feature is equal to the difference between that feature and the baseline sample multiplied by the integral of the gradient of that feature on the linear interpolation path from the baseline sample to the input sample. The formula is as follows:

[0019] in, This represents the model's predicted probability for class c. For the input sample, For baseline samples, The feature is the difference between the input sample and the baseline sample; The interpolation path is defined by parameters ranging from 0 to 1; from the baseline To input The linear interpolation path can be represented as: ; This is the integral gradient of the input layer features.

[0020] 2) Obtain the path activation values ​​of each layer based on the integral gradient: For any differentiable model prediction function The sum of the integral gradients of all input features equals the sum of the integral gradients of the model on the input samples. The predicted probability on the baseline sample The difference in predicted probabilities.

[0021] The layer activation path should be: The activation path of a layer is a linear interpolation between the activation values ​​of that layer on the baseline sample and the activation values ​​on the input sample.

[0022] The activation value on the path is equal to: the activation value of this layer on the baseline sample plus the interpolation path parameter multiplied by (the difference between the activation value of this layer on the input sample and the baseline activation value), expressed by the formula:

[0023] in, Activation value on the path; Activation value of this layer on the baseline sample; Activation value of this layer on the input sample; The interpolation path is defined by parameters, ranging from 0 to 1.

[0024] 3) Layer conductivity: The conductance of a layer is the sum of the products of the activation difference and the corresponding element of the path-sensitive gradient for each activation unit in that layer. Here, the activation difference refers to the difference in activation value between the input sample and the baseline sample for that activation unit; the path-sensitive gradient refers to the integral of the gradient of the model output for that activation unit along the linear interpolation path from the baseline to the input, expressed by the formula:

[0025] in, For the first The conductivity of the layer; Multiply corresponding elements; This is the index of the activated unit in this layer. This represents the total number of activated units in this layer.

[0026] 4) Based on the completeness of layer conductivity, the discretized form of layer conductivity is obtained, which can be approximated by discretized integral: The first... The conductivity of a layer is approximately equal to the activation difference of each activation unit in that layer, multiplied by the average gradient of that activation unit on the discretization path to the model output, and then summed over all activation units.

[0027] For convolutional layers in convolutional neural networks Its conductivity can be decomposed into spatial and channel components: First, for each spatial location (height) ,width ) channel dimension (total) Calculate the channel-level conductance (for each channel), which is the product of the activation difference of the channel at that location and the gradient expectation. Then take the average of all spatial locations (divided by) To ensure translation invariance, the conductivity of the convolutional layer is finally obtained.

[0028] 2. Adaptive noise adjustment mechanism: Based on the importance of each layer, i.e. the conductivity of each layer, the noise intensity is dynamically allocated. Lower intensity noise is added to high-importance layers to reduce the impact on model performance; higher intensity noise is added to low-importance layers to enhance privacy protection.

[0029] The specific implementation process is as follows: scores from each layer are calculated using an interpretability algorithm, and then the scores are summed, sorted, and the ratio is calculated. A privacy budget is allocated according to the ratio, and noise is dynamically adjusted accordingly.

[0030] 3. Hierarchical gradient pruning mechanism: Independent pruning is performed on gradients of different norms at each layer to avoid privacy risks caused by excessively large local gradients, while reducing over-pruning of effective gradient information.

[0031] 4. Sharpness-aware optimization mechanism: The sharpness-aware optimizer improves the traditional stochastic gradient descent optimizer by minimizing the sharpness of the model loss, thereby enhancing the model's generalization ability in noisy scenarios and mitigating the impact of noise on convergence stability.

[0032] Example Using a classic CNN model, the MNIST image classification task was performed on a platform with an NVIDIA GeForce RTX4090 GPU and a 64-bit Ubuntu 20.04.6LTS operating system to verify the effectiveness of the proposed method.

[0033] In privacy budget , It satisfies the RDP (Rényi Differential Privacy) constraint; Comparison of approaches: traditional unified noisy DP-SGD (Differentially Private Stochastic Gradient Descent) and adaptive noisy strategy DPDLDA.

[0034] The accuracy rate of this method was 72.85%, while that of DP-SGD was 60.05% and DPDLDA was 69.55%. Therefore, this method achieves a 19.1% improvement over DP-SGD and a 3.3% improvement over DPDLDA; Verified by RDP, global privacy budget While maintaining strict differential privacy, preset privacy constraints are met.

[0035] In summary, this method significantly improves model performance while preserving privacy, supporting the effectiveness of beneficial effects. Balancing privacy and performance: within the same privacy budget... While maintaining strict differential privacy, the model accuracy is improved by 19.1% compared to the baseline of the traditional uniform noise addition scheme, resolving the contradiction between "excessive noise reducing utility" and "insufficient noise leaking privacy." Convergence efficiency is improved: through hierarchical gradient pruning and sharpness-aware optimization, the model convergence speed is significantly accelerated, and the loss value after convergence is lower, enhancing stability on complex datasets. Strict privacy guarantees are ensured: through hierarchical sensitivity constraints and RDP theory derivation, it is proven that the global privacy budget is satisfied. - Differential privacy (maintains high model performance even when ε≤1.0) has a better privacy-performance Pareto boundary than existing schemes.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hierarchical adaptive noise-adding differential privacy deep learning method, characterized in that, include: The transmissibility of each layer is used to characterize the degree of influence of gradient changes in each layer of the model on the loss function, and is used as the weight of the influence of each layer of the model on the final output. A privacy budget is allocated to each layer based on the aforementioned influence weights; Independently prune the gradient norms of each layer; By minimizing both the model loss and the sharpness of the model loss, a flat minimum is found to complete model training.

2. The method according to claim 1, characterized in that, The conductivity includes: Calculate the integral gradient of the input features; The activation values ​​of each layer's path are obtained based on the completeness of the integral gradient; Calculate the conductivity of each layer based on the path activation values ​​of each layer; The discrete form of the conductance of each layer is obtained based on the completeness of the integral gradient, which serves as the influence weight of each layer on the final output.

3. The method according to claim 2, characterized in that, The calculation of the integral gradient of the input features includes: The integral gradient of an input feature is equal to the difference between the feature and the baseline sample multiplied by the integral of the gradient of that feature on the linear interpolation path from the baseline sample to the input sample.

4. The method according to claim 2, characterized in that, The step of obtaining the path activation values ​​of each layer based on the completeness of the integral gradient includes: The activation value on the path is equal to the activation value of the layer on the baseline sample plus the interpolation path parameter multiplied by the difference between the activation value of the layer on the input sample and the baseline activation value.

5. The method according to claim 2, characterized in that, The calculation of the conductivity of each layer based on the path activation values ​​of each layer includes: No. The conductivity of a layer is the sum of the products of the activation difference of each activation unit in that layer and the corresponding elements of the path-sensitive gradient; Activation difference refers to the difference between the activation values ​​of the activation unit on the input sample and the baseline sample. Path-sensitive gradient refers to the integral of the gradient of the activation unit by the model output on the linear interpolation path from the baseline to the input.

6. The method according to claim 2, characterized in that, The process of obtaining the discrete form of conductance of each layer based on the completeness of the integral gradient includes: No. The discrete form of the conductance of a layer is equal to the activation difference of each activation unit in that layer multiplied by the average gradient of that activation unit to the model output along the discretization path, and then summed over all activation units.

Images