Differential privacy noise distribution method based on layer contribution degree

Abstract

The invention discloses a differential privacy noise distribution method based on a layer contribution degree, and relates to the technical field of privacy protection, and the method comprises the steps: receiving a current deep neural network model parameter issued by a server at a federated learning client, executing forward propagation based on local training data, and obtaining an activation output; using activation output to calculate the loss change of the deep neural network model in a layer-by-layer zero disturbance mode, and generating a dynamic layer contribution degree; according to the dynamic layer contribution degree, determining a global importance score fusing the direct disturbance effect and the cross-layer information flow influence in combination with an inter-layer correlation back propagation mechanism; aggregating the global importance scores to obtain a group contribution degree; and uploading the group contribution degree to a server, aggregating the group contribution degree of the federated learning client by the server, calculating historical volatility of the contribution degree based on an aggregation result, further adjusting a privacy budget distribution proportion, and issuing the distribution proportion to the federated learning client.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of privacy protection, and particularly relates to a differential privacy noise allocation method based on layer contribution degree. BACKGROUND

[0002] As a privacy protection mechanism with strict mathematical guarantee, differential privacy has been widely applied in federated learning systems, which usually prevents data reverse by injecting random noise in local gradient or model update. Existing methods mainly include two categories: one aims to improve model robustness, such as adopting defensive strategies like adversarial training or gradient clipping; the other attempts to allocate privacy budget in layers, for example, weighting based on gradient magnitude or applying uniform noise to all network layers.

[0003] However, these schemes have key defects: first, they ignore the dynamic contribution difference of different functional modules in the training process, leading to insufficient protection of key layers or excessive disturbance of redundant layers; second, they rely on gradient magnitude to evaluate layer importance, which is easily disturbed by gradient vanishing, activation saturation or normalization operation, and is difficult to truly reflect the causal influence of each layer on model output. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a differential privacy noise allocation method based on layer contribution degree to solve the problem that existing technologies ignore the dynamic contribution difference of different functional modules in the training process, leading to insufficient protection of key layers or excessive disturbance of redundant layers, rely on gradient magnitude to evaluate layer importance, which is easily disturbed by gradient vanishing, activation saturation or normalization operation, and is difficult to truly reflect the causal influence of each layer on model output.

[0006] To solve the above technical problems, the present application provides the following technical solutions: In the first aspect, the present application provides a differential privacy noise allocation method based on layer contribution degree, which comprises: The federated learning client receives the current deep neural network model parameters issued by the server, and performs forward propagation based on the local training data to obtain the activation output; Using the activation output, the deep neural network model loss change is calculated by the way of zeroing disturbance layer by layer to generate dynamic layer contribution degree; According to the dynamic layer contribution degree, the global importance score which integrates the direct disturbance effect and the influence of cross-layer information flow is determined combined with the interlayer correlation backpropagation mechanism; The global importance score is aggregated to obtain group contribution degree; The group contribution is uploaded to the server, which aggregates the group contribution of the federated learning client and calculates the historical volatility of the contribution based on the aggregation result. Then, the privacy budget allocation ratio is adjusted and the allocation ratio is distributed to the federated learning client. The federated learning client configures the corresponding noise intensity according to the privacy budget allocation ratio, and applies random perturbations to the local gradients respectively. The gradient perturbations adopt a cooperative noise injection method. The gradients that have completed noise injection are uploaded to the server and participate in the aggregation update.

[0007] As a preferred embodiment of the differential privacy noise allocation method based on layer contribution as described in this invention, the step of calculating the loss change of the deep neural network model by using activation output and perturbing by setting zero layer by layer to generate dynamic layer contribution is as follows: The activation output of each layer is operated on by the layer-by-layer activation zeroing perturbation method. For each layer in the deep neural network, the activation output generated during the forward propagation is replaced with a zero tensor with the same shape as the original tensor, while keeping the activation output of each layer unchanged. The remaining forward propagation process from the current layer to the output layer is then completed to obtain the model prediction result under the perturbation state. The model's predictions under perturbation are compared with the true labels using a loss function to obtain the model loss after perturbation. ; Simultaneously, a loss function is used to calculate the loss of the original model by comparing the normal forward propagation results with the true labels. ; Loss after disturbance Compared with the original loss Perform the absolute value operation on the difference to obtain the first... Dynamic layer contribution of the current training batch Its calculation expression is: ; in, For the first Dynamic layer contribution of the layer To set to zero The model loss calculated after layer activation, The loss is the original model loss obtained from normal forward propagation under undisturbed conditions; The federated learning process is executed sequentially across all layers to ensure that the contribution of dynamic layers can reflect the sensitivity of each layer to the current local data in real time.

[0008] As a preferred embodiment of the differential privacy noise allocation method based on layer contribution as described in this invention, the specific steps for determining the global importance score by fusing direct perturbation effects and cross-layer information flow influences based on dynamic layer contribution and combined with the inter-layer correlation backpropagation mechanism are as follows: The contribution of the dynamic layer is determined by using the importance backpropagation fusion method. To enhance the process, first apply enhancements to all layers. Perform normalization operation, Divide by the largest dynamic layer contribution among all layers Normalized direct contribution items were obtained. ; Define the set of predecessor layers To directly propagate to the layer during forward propagation Provides all layers of input, for the set Each layer Obtain the gradient tensor for the current training batch. and model weight tensor ; For gradient tensor with weight tensor Perform element-wise multiplication to obtain ; right Perform L1 norm calculation to obtain , used to measure the joint activation strength of gradient and weight; For gradient tensor Perform L1 norm calculation to obtain ; Will Divide by ,in To prevent small constants with zero denominators, we obtain Gradient-weight interaction strength factor; Define the successor set For layer The output of all connected layers, for the set Each layer Obtain the global importance score. and all Summing, we get ; Precursor layer Global importance score Divide by ,in As a stability constant, the structure propagation weights are obtained; Multiplying the gradient-weight interaction strength factor by the structure propagation weights yields the layer. For the current layer Indirect contributions; right The indirect contribution items of all predecessor layers are summed to obtain the cross-layer information flow contribution item. Normalize direct contribution items Adding it to the cross-layer information flow contribution item, we get the first... Global importance score of the layer Its expression is: ; in, For the first The global importance score of the layer. This is the contribution of the direct perturbation after normalization. For layer Precursor layer set, For the first The gradient tensor of the layer, For the first The model weight tensor of the layer, For Hadamard product, It is an L1 norm. It is a smoothing constant. For the precursor layer Global importance score, For layer The set of direct successor layers, For the successor layer Global importance score, It is the stability constant. = 1 As a smoothing factor, = 0.01 is the stability constant.

[0009] As a preferred embodiment of the differential privacy noise allocation method based on layer contribution as described in this invention, the specific steps for aggregating global importance scores to obtain group contribution are as follows: Based on the network functional module grouping rules, deep neural networks are divided into logical functional groups. Each group contains functionally related layers; Global importance scores for each layer Filter and extract those belonging to the same functional module group. All ; Assign weights to each layer based on its structural position within the group. Layers along the jump path are assigned higher weights; For all layers within the group and Summing the products of , we get the . Group contribution of each functional module group Its calculation expression is: ; in, For the first The group's functional module group contribution. For the first The collection of all layers contained in a functional module group For layer The weighting coefficient within the group, For a preset positive real number, The value range is (0,1), and its specific value is determined based on whether the layer is located on a skip connection, residual path, or backbone information flow. The weight of a layer on a skip path is not less than 0.5, and the weight of other layers is not more than 0.3. For the first Global importance score of the layer; The fine-grained hierarchical importance is integrated into a coarse-grained module-level contribution.

[0010] As a preferred embodiment of the differential privacy noise allocation method based on layer contribution as described in this invention, the steps of uploading group contribution values ​​to the server, aggregating the group contribution values ​​of the federated learning client, calculating the historical volatility of the contribution values ​​based on the aggregation results, adjusting the privacy budget allocation ratio, and then distributing the allocation ratio to the federated learning client are as follows: The Federated Learning Client will group contribution Uploaded to the server in encrypted form; The server provides support to all participating clients in this round. The global group contribution was calculated using median aggregation. To resist outlier interference; The server maintenance teams in the past In round iteration Historical records are analyzed, and variance is calculated to obtain the historical volatility of each group's contribution. Its expression is: ; in, For the first Historical volatility of group contribution For the first Group 1 in The global group contribution of the round, For the past wheel The mean; Server based on and Construct budget adjustment factors, Divide by ,in The variance suppression coefficient is used as the basis for calculation. The same operation is then performed on all groups and summed. The individual group results are then divided by the sum to obtain the updated privacy budget allocation ratio. Its expression is: ; in, For the first The group's privacy budget allocation ratio, =0.5; The server will be updated. The encrypted data is sent to the federated learning client.

[0011] As a preferred embodiment of the differential privacy noise allocation method based on layer contribution described in this invention, the federated learning client configures the corresponding noise intensity according to the privacy budget allocation ratio, and applies random perturbations to the local gradients respectively. The gradient perturbations adopt a cooperative noise injection method. The specific steps are as follows: Federated Learning Client's Privacy Budget Allocation Perform analysis, based on Calculate each functional module group Allocable privacy budget ,in For the overall privacy budget, For the first Privacy budget allocation ratio for each functional module group; Local gradients for each group Perform L2 clipping to ensure the norm does not exceed a preset threshold. Obtain the gradient after clipping. ; Crop threshold L2 sensitivity as a gradient ; The standard deviation of the noise to be added is calculated using the Gaussian mechanism. Its expression is: ; in, For the first The noise standard deviation of the group The failure probability parameter is allocated proportionally. To be assigned to the Local privacy budget for each functional module group; For the same functional module group Gradient perturbation within the vector to construct a multivariate Gaussian noise vector with a shared covariance structure. The covariance matrix Based on the internal connection structure of the module is preset; Will Add to Get the disturbed gradient .

[0012] As a preferred scheme of the layer contribution-based differential privacy noise allocation method of the application, wherein: the gradient with completed noise injection is uploaded to the server to participate in aggregated updating, and the specific steps are: The disturbed gradient of each functional module group is completed Integration, all groups of According to the model parameter sequence splicing, the local disturbed gradient vector is obtained ; The secure aggregation protocol is used for encrypted fragmentation processing of Generate multiple encrypted gradient fragments; The encrypted gradient fragments are uploaded to the server respectively; After receiving the encrypted gradient fragments of the participating federated learning clients, the server performs decryption operation, and sums the gradient fragments of the same parameter position to obtain the global aggregated disturbed gradient of each functional module group ; The server uses Update the current global deep neural network model parameters The update mode is that the parameters are subtracted by the product of the learning rate And the aggregated gradient, to obtain the updated global model parameters ; The server distributes the updated global model parameters To the participating federated learning clients.

[0013] As a preferred scheme of the layer contribution-based differential privacy noise allocation method of the application, wherein: the local gradient is respectively subjected to random disturbance, and the gradient disturbance adopts a collaborative noise injection mode, and the specific steps are: The preset covariance matrix Dynamic optimization is carried out; In each round of training, the client calculates the pair-wise Pearson correlation coefficient of each layer gradient Of the current functional module group , and constructs a dynamic correlation matrix ; The dynamic correlation matrix And the structure prior matrix based on the network architecture preset Weighted fusion is carried out, and a convex combination method is adopted to generate the actual used covariance structure , and the calculation expression is: ; wherein, is the final adopted covariance matrix of the group, is a dynamic adaptation coefficient, and the value range is is a correlation matrix calculated based on the current gradient, is a prior structure matrix reflecting the internal connection relationship of the module, =0.7, used to balance the data-driven correlation and the network structure prior; based on the updated reconstructs a multivariate Gaussian noise vector . adds the to the clipped local gradient , to obtain a perturbed gradient with dynamic structure adaptation.

[0014] In a second aspect, the present application provides a computer device comprising a memory and a processor, the memory storing a computer program, wherein: the computer program is executed by the processor to implement any step of the layer contribution-based differential privacy noise allocation method according to the first aspect of the present application.

[0015] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein: the computer program is executed by the processor to implement any step of the layer contribution-based differential privacy noise allocation method according to the first aspect of the present application.

[0016] The present application has the following beneficial effects: by receiving the current deep neural network model parameters issued by the server at the federal learning client, and performing forward propagation based on the local training data to obtain the activation output of each layer, the accurate capture of the model internal intermediate representation is realized, the process ensures that the local model of the client is consistent with the global model state, avoids the distortion of subsequent evaluation caused by parameter deviation, and at the same time, the structured caching of the key activation information of the convolution feature map, nonlinear response, normalized output and residual path transmission value provides a real data basis for subsequent layer-by-layer perturbation analysis, thereby achieving the purpose of accurately reflecting the dynamic behavior of the model under the current local data distribution, and providing a reliable basis for the formulation of fine privacy protection strategy. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0018] Figure 1 A flowchart of a layer-contribution-based differential privacy noise allocation method; Figure 2 A branch flowchart of a layer-contribution-based differential privacy noise allocation method. DETAILED DESCRIPTION

[0019] In order to make the above objectives, features and advantages of the present application more apparent, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0020] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0021] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0022] Reference Figure 1 For one embodiment of the present application, the embodiment provides a layer-contribution-based differential privacy noise allocation method, comprising the following steps: S1, receiving the current deep neural network model parameters issued by the server at the federated learning client, and performing forward propagation based on the local training data to obtain the activation output.

[0023] Further, the global model parameters are used to initialize the local deep neural network, ensuring that the client local model is consistent with the global model state of the current communication round; a data batch is randomly selected from the local private training dataset as the input sample of the current iteration; the input sample is fed into the model, and the complete forward propagation process from the input layer to the output layer is completed through linear transformation, nonlinear activation function, normalization operation and possible pooling or attention mechanism of each layer; during the forward propagation process, the intermediate output results generated by each layer of network module are captured and cached, and the intermediate output results are the activation outputs of the layer; the activation outputs include: feature maps of convolutional layers, vector outputs of fully connected layers, nonlinear responses after activation function processing, normalized outputs of normalization layers, weighted representations in attention mechanisms, and direct transmission values of jump paths in residual connections; the activation outputs of all layers are organized and stored according to their hierarchical order in the network to form a structured activation data set for subsequent layer contribution calculation; the process is repeated in each training iteration period to ensure that the obtained activation outputs can reflect the dynamic behavior characteristics of the model under the current local data distribution.

[0024] It should be noted that the process of obtaining activation outputs provides basic data for subsequent layer contribution analysis. This step ensures that the client model is synchronized with the global model, and accurately captures the intermediate outputs of each layer, including convolutional feature maps, fully connected responses, normalized results and residual path transmission values, to form a structured activation set, ensuring the accuracy and traceability of the perturbation analysis.

[0025] S2, using the activation output, the loss change of the deep neural network model is calculated by zeroing the perturbation layer by layer, and the dynamic layer contribution is generated.

[0026] Further, the zero-perturbation method is used to operate the activation outputs of each layer, and the activation outputs generated in the forward propagation process are replaced with zero tensors of the same shape as the original tensors, while keeping the activation outputs of each layer unchanged, and the remaining forward propagation process from the current layer to the output layer is continued to obtain the model prediction result under the perturbation state; the loss function is used to calculate the model prediction result under the perturbation state and the true label, and the perturbed model loss is obtained; at the same time, the loss function is used to calculate the normal forward propagation result and the true label, and the original model loss is obtained; the absolute value operation of the difference between the perturbed loss and the original loss is performed, and the dynamic layer contribution of the first layer under the current training batch is obtained , and the calculation expression is: ; in, For the first Dynamic layer contribution of the layer To set to zero The model loss calculated after layer activation, The loss is the original model loss obtained from normal forward propagation under undisturbed conditions. In the federated learning iteration, this loss is applied sequentially to all layers to ensure that the dynamic layer contribution reflects the sensitivity of each layer to the current local data in real time. It should be noted that by setting the activation to zero layer by layer and calculating the change in loss, the impact of each layer on model performance can be quantified. This method does not rely on gradient magnitude, avoiding misjudgments of importance due to gradient vanishing or saturation, and evaluates based on actual forward behavior, reflecting the true sensitivity of each layer under the current input, generating a layer contribution index with dynamic characteristics.

[0027] S3. Based on the contribution of the dynamic layer and combined with the backpropagation mechanism of inter-layer correlation, determine the global importance score of the fusion direct disturbance effect and the cross-layer information flow influence.

[0028] Furthermore, an importance backpropagation fusion method is used to assess the contribution of the dynamic layer. To enhance the process, first apply enhancements to all layers. Perform normalization operation, Divide by the largest dynamic layer contribution among all layers Normalized direct contribution items were obtained. Define the set of predecessor layers. To directly propagate to the layer during forward propagation Provides all layers of input, for the set Each layer Obtain the gradient tensor for the current training batch. and model weight tensor For gradient tensors with weight tensor Perform element-wise multiplication to obtain ;right Perform L1 norm calculation to obtain This is used to measure the joint activation strength of the gradient and weights; for the gradient tensor Perform L1 norm calculation to obtain ;Will Divide by ,in To prevent small constants with zero denominators, we obtain Gradient-weight interaction strength factor; Define the set of successor layers. For layer The output of all connected layers, for the set Each layer Obtain the global importance score. and all Summing, we get ; the precursor layer Global importance score Divide by ,in As a stability constant, the structure propagation weights are obtained; the gradient-weight interaction strength factor is multiplied by the structure propagation weights to obtain the layer... For the current layer Indirect contributions; to The indirect contributions of all preceding layers are summed to obtain the cross-layer information flow contribution; the direct contributions are then normalized. Adding it to the cross-layer information flow contribution item, we get the first... Global importance score of the layer Its expression is: ; in, For the first The global importance score of the layer. This is the contribution of the direct perturbation after normalization. For layer Precursor layer set, For the first The gradient tensor of the layer, For the first The model weight tensor of the layer, For Hadamard product, It is an L1 norm. It is a smoothing constant. For the precursor layer Global importance score, For layer The set of direct successor layers, For the successor layer Global importance score, It is the stability constant. = 1 As a smoothing factor, = 0.01 is the stability constant.

[0029] It should be noted that the introduction of the backpropagation mechanism integrates cross-layer dependencies, so that the global importance score not only reflects the impact of perturbations in this layer, but also incorporates the gradient-weight interaction strength of the predecessor layer and the importance feedback of the successor layer, thereby more comprehensively depicting the overall position of the layer in the network structure and improving the systematicness and robustness of importance assessment.

[0030] S4. Aggregate the global importance scores to obtain the group contribution.

[0031] Furthermore, based on the network functional module grouping rules, deep neural networks are divided into logical functional groups. Each group contains functionally related layers; a global importance score is given for each layer. Filter and extract those belonging to the same functional module group. All Assign weights to each layer based on its structural position within the group. Layers along the jump path are assigned higher weights; all layers within the group are weighted accordingly. and Summing the products of , we get the . Group contribution of each functional module group Its calculation expression is: ; in, For the first The group's functional module group contribution. For the first The collection of all layers contained in a functional module group For layer The weighting coefficient within the group, For a preset positive real number, The value range is (0,1), and its specific value is determined based on whether the layer is located on a skip connection, residual path, or backbone information flow. The weight of a layer on a skip path is not less than 0.5, and the weight of other layers is not more than 0.3. For the first The global importance score of the layer is calculated; fine-grained hierarchical importance is integrated into coarse-grained module-level contribution.

[0032] It should be noted that by aggregating hierarchical importance into group contribution based on functional modules, the abstraction from fine-grained parameters to coarse-grained components is achieved, reducing the complexity of communication and control. By setting structure-aware weights, the aggregation results can better reflect the overall functional stability of the modules, facilitating the implementation of subsequent differentiated privacy protection strategies.

[0033] S5. Upload the group contribution to the server, where the server aggregates the group contribution of the federated learning client, calculates the historical volatility of the contribution based on the aggregation result, adjusts the privacy budget allocation ratio, and then distributes the allocation ratio to the federated learning client.

[0034] Furthermore, the federated learning client will group contribution levels. The data is encrypted and uploaded to the server; the server records the data from all participating clients in this round. The global group contribution was calculated using median aggregation. To resist outlier interference; server maintenance groups in the past In round iteration Historical records are analyzed, and variance is calculated to obtain the historical volatility of each group's contribution. Its expression is: ; in, For the first Historical volatility of group contribution For the first Group 1 in The global group contribution of the round, For the past wheel The mean; server based and Construct budget adjustment coefficients, Divide by ,in The variance suppression coefficient is used as the basis for calculation. The same operation is then performed on all groups and summed. The individual group results are then divided by the sum to obtain the updated privacy budget allocation ratio. Its expression is: ; in, For the first The group's privacy budget allocation ratio, =0.5; the server will update to... The encrypted data is sent to the federated learning client.

[0035] It should be noted that the server's median aggregation of group contributions can suppress interference from abnormal reporting. Combined with historical volatility analysis, it can achieve dynamic and smooth adjustment of budget allocation. By suppressing the budget growth of high-volatility modules, it can prevent short-term noise from causing resource misallocation and ensure that privacy resources are continuously allocated to stable and critical functional units.

[0036] S6. The federated learning client configures the corresponding noise intensity according to the privacy budget allocation ratio and applies random perturbations to the local gradients respectively. The gradient perturbation adopts a cooperative noise injection method.

[0037] Furthermore, the Federated Learning client's allocation of the privacy budget... Perform analysis, based on Calculate each functional module group Allocable privacy budget ,in For the overall privacy budget, For the first Privacy budget allocation ratios for each functional module group; local gradients for each group. Perform L2 clipping to ensure the norm does not exceed a preset threshold. , to obtain the clipped gradient ; the clipping threshold is set as the L2 sensitivity of the group of gradients ; the standard deviation of noise to be added is calculated using a Gaussian mechanism , whose expression is ; wherein is the standard deviation of noise of the th group, is the proportionally allocated failure probability parameter, is the local privacy budget allocated to the th group of functional modules; for gradient perturbation within the same group of functional modules , a multivariate Gaussian noise vector with a shared covariance structure is constructed , wherein the covariance matrix is preset based on the internal connection structure of the module; the is added to the , to obtain the perturbed gradient .

[0038] It should be noted that the noise intensity is configured according to the allocation proportion, and multivariate Gaussian noise injection with a shared covariance structure is adopted, which breaks through the limitation of traditional independent noise injection, maintains the statistical correlation between parameters within the module, reduces semantic distortion, and improves model usability and convergence stability at the same privacy cost.

[0039] S7, upload the gradient after completing noise injection to the server to participate in aggregated update.

[0040] Further, the perturbed gradients of each group of functional modules after completing noise injection are integrated, and the of all groups are spliced according to the order of model parameters, to obtain a local perturbed gradient vector ; the is encrypted and fragmented using a secure aggregation protocol, to generate multiple encrypted gradient fragments; the encrypted gradient fragments are uploaded to the server respectively; after receiving the encrypted gradient fragments of the participating federated learning clients, the server performs decryption operation and summing operation on the gradient fragments of the same parameter position, to obtain the global aggregated perturbed gradient of each group of functional modules; the server updates the current global deep neural network model parameters using the , and the update mode is that the parameters are subtracted by the product of the learning rate and the aggregated gradient, to obtain the updated global model parameters ; the server distributes the updated global model parameters to the participating federated learning clients; the preset covariance matrix Dynamic optimization is performed; in each round of training, the client calculates the pair-wise Pearson correlation coefficient of the gradients of each layer in the current function module group to construct a dynamic correlation matrix ; the dynamic correlation matrix is weighted and fused with a structure prior matrix based on the preset network architecture , and a convex combination method is used to generate the actual covariance structure used in this round , and the calculation expression is as follows: ; wherein, is the final covariance matrix used by the first group, is a dynamic adaptation coefficient, and the value range is is the correlation matrix calculated based on the current gradient, is a prior structure matrix reflecting the internal connection relationship of the module, = 0.7, used to balance the data-driven correlation and the network structure prior; based on the updated , a multivariate Gaussian noise vector is reconstructed ; the is added to the clipped local gradient to obtain a perturbed gradient with dynamic structure adaptation.

[0041] It should be noted that after the perturbed gradient is uploaded and aggregated, a covariance matrix dynamic optimization mechanism is further introduced to adjust the noise structure using the current gradient correlation, realize data-driven adaptive perturbation, balance flexibility and reliability by combining the preset prior, make the noise injection more suitable for the actual training dynamics, improve the overall learning efficiency and privacy utility ratio, and by setting the dynamic adaptation coefficient λ = 0.7, the network topology prior is retained while the current training dynamics is effectively integrated, and the adaptability of noise injection is improved.

[0042] The embodiment also provides a computer device suitable for the differential privacy noise allocation method based on layer contribution degree, which comprises a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the differential privacy noise allocation method based on layer contribution degree proposed in the above embodiment.

[0043] ​The computer device can be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved by WIFI, an operator network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0044] The embodiment also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the method for allocating differential privacy noise based on layer contribution degree proposed in the above embodiment. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0045] To sum up, the present application achieves accurate capture of the model internal intermediate representation by receiving the current deep neural network model parameters issued by the server at the federated learning client and performing forward propagation based on the local training data to obtain the activation output of each layer, ensures that the local model of the client is consistent with the global model state, avoids distortion of subsequent evaluation caused by parameter deviation, and provides a true data basis for subsequent layer-by-layer perturbation analysis by structuring the cache of key activation information such as convolution feature maps, nonlinear response, normalized output and residual path transmission values, thereby achieving the purpose of accurately reflecting the dynamic behavior of the model under the current local data distribution and providing a reliable basis for the development of fine privacy protection strategies.

[0046] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, and all these modifications and equivalents should be included in the scope of the claims of the present application.

Claims

1. A differential privacy noise allocation method based on layer contribution, characterized in that: include: The federated learning client receives the current deep neural network model parameters from the server and performs forward propagation based on local training data to obtain the activation output; By utilizing the activation output, the loss change of the deep neural network model is calculated through a layer-by-layer zeroing perturbation, and the dynamic layer contribution is generated. Based on the contribution of the dynamic layer and combined with the backpropagation mechanism of inter-layer correlation, the global importance score of the fusion direct disturbance effect and the cross-layer information flow influence is determined; The overall importance scores are aggregated to obtain the group contribution. The group contribution is uploaded to the server, which aggregates the group contribution of the federated learning client and calculates the historical volatility of the contribution based on the aggregation result. Then, the privacy budget allocation ratio is adjusted and the allocation ratio is distributed to the federated learning client. The federated learning client configures the corresponding noise intensity according to the privacy budget allocation ratio, and applies random perturbations to the local gradients respectively. The gradient perturbations adopt a cooperative noise injection method. The gradients that have completed noise injection are uploaded to the server and participate in the aggregation update.

2. The differential privacy noise allocation method based on layer contribution as described in claim 1, characterized in that: The method of utilizing activation outputs to calculate the loss change of the deep neural network model and generate dynamic layer contribution by perturbating by setting zero layer by layer is as follows: The activation output of each layer is operated on by the layer-by-layer activation zeroing perturbation method. For each layer in the deep neural network, the activation output generated during the forward propagation is replaced with a zero tensor with the same shape as the original tensor, while keeping the activation output of each layer unchanged. The remaining forward propagation process from the current layer to the output layer is then completed to obtain the model prediction result under the perturbation state. The model's predictions under perturbation are compared with the true labels using a loss function to obtain the model loss after perturbation. ; Simultaneously, a loss function is used to calculate the loss of the original model by comparing the normal forward propagation results with the true labels. ; Loss after disturbance Compared with the original loss Perform the absolute value operation on the difference to obtain the first... Dynamic layer contribution of the current training batch Its calculation expression is: ; in, For the first Dynamic layer contribution of the layer To set to zero The model loss calculated after layer activation, The loss is the original model loss obtained from normal forward propagation under undisturbed conditions; The federated learning process is executed sequentially across all layers to ensure that the contribution of dynamic layers can reflect the sensitivity of each layer to the current local data in real time.

3. The differential privacy noise allocation method based on layer contribution as described in claim 2, characterized in that: The global importance score is determined based on the contribution of the dynamic layer and the backpropagation mechanism of inter-layer correlation, by fusing the direct perturbation effect and the cross-layer information flow influence. The specific steps are as follows: The contribution of the dynamic layer is determined by using the importance backpropagation fusion method. To enhance the process, first apply enhancements to all layers. Perform normalization operation, Divide by the largest dynamic layer contribution among all layers Normalized direct contribution items were obtained. ; Define the set of predecessor layers To directly propagate to the layer during forward propagation Provides all layers of input for the set Each layer Obtain the gradient tensor for the current training batch. and model weight tensor ; For gradient tensor with weight tensor Perform element-wise multiplication to obtain ; right Perform L1 norm calculation to obtain , used to measure the joint activation strength of gradient and weight; For gradient tensor Perform L1 norm calculation to obtain ; Will Divide by ,in To prevent small constants with zero denominators, we obtain Gradient-weight interaction strength factor; Define the successor set For layer The output of all connected layers, for the set Each layer Obtain the global importance score. and all Summing, we get ; Precursor layer Global importance score Divide by ,in As a stability constant, the structure propagation weights are obtained; Multiplying the gradient-weight interaction strength factor by the structure propagation weights yields the layer. For the current layer Indirect contributions; right The indirect contribution items of all predecessor layers are summed to obtain the cross-layer information flow contribution item. Normalize direct contribution items Adding it to the cross-layer information flow contribution item, we get the first... Global importance score of the layer Its expression is: ; in, For the first The global importance score of the layer. This is the contribution of the direct perturbation after normalization. For layer Precursor layer set, For the first The gradient tensor of the layer, For the first The model weight tensor of the layer, For Hadamard product, It is an L1 norm. It is a smoothing constant. For the precursor layer Global importance score, For layer The set of direct successor layers, For the successor layer Global importance score, It is the stability constant. = 1 As a smoothing factor, = 0.01 is the stability constant.

4. The differential privacy noise allocation method based on layer contribution as described in claim 3, characterized in that: The specific steps for aggregating the global importance scores to obtain the group contribution are as follows: Based on the network functional module grouping rules, deep neural networks are divided into logical functional groups. Each group contains functionally related layers; Global importance scores for each layer Filter and extract those belonging to the same functional module group. All ; Assign weights to each layer based on its structural position within the group. Layers along the jump path are assigned higher weights; For all layers within the group and Summing the products of , we get the . Group contribution of each functional module group Its calculation expression is: ; in, For the first The group's functional module group contribution. For the first The collection of all layers contained in a functional module group For layer The weighting coefficient within the group, For a preset positive real number, The value range is (0,1), and its specific value is determined based on whether the layer is located on a skip connection, residual path, or backbone information flow. The weight of a layer on a skip path is not less than 0.5, and the weight of other layers is not more than 0.

3. For the first The global importance score of the layer; The fine-grained hierarchical importance is integrated into a coarse-grained module-level contribution.

5. The differential privacy noise allocation method based on layer contribution as described in claim 4, characterized in that: The process involves uploading group contributions to the server, aggregating the group contributions from the federated learning clients, calculating the historical volatility of the contributions based on the aggregation results, adjusting the privacy budget allocation ratio, and then distributing the allocation ratio back to the federated learning clients. The specific steps are as follows: The Federated Learning Client will group contribution Uploaded to the server in encrypted form; The server provides support to all participating clients in this round. The global group contribution was calculated using median aggregation. To resist outlier interference; The server maintenance teams in the past In round iteration Historical records are analyzed, and variance is calculated to obtain the historical volatility of each group's contribution. Its expression is: ; in, For the first Historical volatility of group contribution For the first Group 1 in The global group contribution of the round, For the past wheel The mean; Server based on and Construct budget adjustment factors, Divide by ,in The variance suppression coefficient is used as the basis for calculation. The same operation is then performed on all groups and summed. The individual group results are then divided by the sum to obtain the updated privacy budget allocation ratio. Its expression is: ; in, For the first The group's privacy budget allocation ratio, =0.5; The server will be updated. The encrypted data is sent to the federated learning client.

6. The differential privacy noise allocation method based on layer contribution as described in claim 5, characterized in that: The federated learning client configures the corresponding noise intensity according to the privacy budget allocation ratio, and applies random perturbations to the local gradients respectively. The gradient perturbation adopts a cooperative noise injection method, and the specific steps are as follows: Federated Learning Client's Privacy Budget Allocation Perform analysis, based on Calculate each functional module group Allocable privacy budget ,in For the overall privacy budget, For the first Privacy budget allocation ratio for each functional module group; Local gradients for each group Perform L2 clipping to ensure the norm does not exceed a preset threshold. Obtain the gradient after clipping. ; Crop threshold L2 sensitivity as a gradient ; The standard deviation of the noise to be added is calculated using the Gaussian mechanism. Its expression is: ; in, For the first The noise standard deviation of the group The failure probability parameter is allocated proportionally. To be assigned to the Local privacy budget for each functional module group; For the same functional module group Gradient perturbation within the vector to construct a multivariate Gaussian noise vector with a shared covariance structure. The covariance matrix Based on the pre-defined internal connection structure of the module; Will Add to The gradient after perturbation is obtained above. .

7. The differential privacy noise allocation method based on layer contribution as described in claim 6, characterized in that: The steps for uploading the noise-injected gradient to the server for aggregation and update are as follows: Perturbation gradients for each functional module group that has completed noise injection Integrate all groups By concatenating the parameters in the order of the model parameters, we obtain the local perturbation gradient vector. ; Using secure aggregation protocol Perform encrypted sharding processing to generate multiple encrypted gradient shards; The encrypted gradient fragments are uploaded to the server separately. After receiving the encrypted gradient fragments from the clients participating in federated learning, the server performs decryption and sums the gradient fragments at the same parameter position to obtain the global aggregated perturbation gradient for each functional module group. ; Server usage For the current global deep neural network model parameters The update is performed by subtracting the learning rate from the parameters. The updated global model parameters are obtained by multiplying the aggregate gradient with the aggregate gradient. ; The server will update the global model parameters. Distribute to the client application for participating in federal learning.

8. The differential privacy noise allocation method based on layer contribution as described in claim 7, characterized in that, The covariance matrix This was achieved through dynamic optimization, with the following specific steps: In each round of training, the client computes the current functional module group. Inner layer gradients Paired Pearson correlation coefficients were used to construct a dynamic correlation matrix. ; Dynamic correlation matrix Compared with the pre-defined structural prior matrix based on the network architecture Weighted fusion is performed, and a convex combination method is used to generate the covariance structure actually used in this round. Its calculation expression is: ; in, For the first The covariance matrix ultimately adopted by the group This is the dynamic adaptation coefficient, and its value range is... , The correlation matrix is ​​calculated based on the current gradient. A priori structure matrix reflecting the internal connection relationships of the modules. =0.7, used to balance the correlation of data-driven factors with prior knowledge of network structure; Based on the updated Reconstructing the multivariate Gaussian noise vector ; Will Add to the local gradient after clipping The perturbation gradient with dynamic structural adaptability is obtained above. .

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the differential privacy noise allocation method based on layer contribution as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the differential privacy noise allocation method based on layer contribution as described in any one of claims 1 to 8.

Images