Differential privacy-based federated learning gradient protection aggregation method and system thereof

Abstract

The application discloses a federal learning gradient protection aggregation method and system based on differential privacy, relates to the technical field of information security, and comprises the following steps: obtaining a local gradient vector of a current communication round, and calling a historical gradient sequence to construct an inertia tensor; wherein the inertia tensor is constructed based on data correlation and time evolution law of the historical gradient sequence, and is used for accurately capturing gradient change inertia; the local gradient vector is decomposed into a trend feature flow in a virtual evolution hyperplane and a disturbance feature flow perpendicular to the virtual evolution hyperplane by using the inertia tensor; the beneficial effects are that the non-convergence failure caused by noise interference is effectively avoided, strict privacy protection requirements are met, and the negative influence on model accuracy is significantly reduced.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a federated learning gradient protection aggregation method and system based on differential privacy. Background Technology

[0002] In federated learning, differential privacy techniques are typically introduced to prevent central servers or eavesdroppers from back-engineering the participants' original privacy data by aggregating gradients. This is achieved through mathematical desensitization by adding random noise of a specific distribution to the local gradient.

[0003] Existing technologies for gradient protection typically employ isotropic noise injection strategies, which apply uniform perturbations to all dimensions of the gradient vector. This approach ignores the physical evolutionary inertia of the gradient vector during federated learning iterations, making it easy for injected random noise to cause drastic deflections in the gradient vector's direction in the hyperspace. Since model convergence is highly dependent on the accuracy of the gradient direction, this isotropic protection mechanism, while providing privacy strength, inevitably disrupts the physical directionality of gradient evolution, leading to low global model convergence efficiency or even non-convergence technical failures. It is difficult to achieve an effective balance between privacy protection strength and model training utility. Summary of the Invention

[0004] In view of the above-mentioned prior art, this application is hereby proposed. Embodiments of this application provide a differential privacy-based federated learning gradient-protected aggregation method and system, which can effectively avoid non-convergence faults caused by noise interference, satisfying strict privacy protection requirements while significantly reducing the negative impact on model accuracy.

[0005] According to one aspect of this application, a differential privacy-based federated learning gradient-preserving aggregation method is provided, comprising:

[0006] Obtain the local gradient vector of the current communication round and retrieve the historical gradient sequence to construct an inertia tensor; wherein, the inertia tensor is constructed based on the data correlation and time evolution law of the historical gradient sequence, and is used to accurately capture the gradient change inertia;

[0007] The local gradient vector is deconstructed using the inertia tensor into a trend feature flow within the virtual evolutionary hyperplane and a perturbation feature flow perpendicular to the virtual evolutionary hyperplane.

[0008] Based on a preset total privacy budget, a first privacy sub-budget is allocated to the trend feature stream, and a second privacy sub-budget is allocated to the disturbance feature stream, establishing an asymmetric allocation relationship; wherein, the first privacy sub-budget is higher than the second privacy sub-budget;

[0009] The first privacy sub-budget and the second privacy sub-budget are used to inject noise into the trend feature stream and the perturbation feature stream respectively, and vector synthesis is performed to obtain the desensitized gradient vector. The desensitized gradient vector is then subjected to multi-party aggregation to obtain the global update increment.

[0010] Obtain the deviation residual vector between the global update increment and the trend feature flow, use the deviation residual vector to numerically correct the weight factor of the historical gradient sequence, and use the corrected weight factor to perform a weighted operation with the historical gradient sequence retrieved in the next communication round to generate the inertia tensor of the next communication round.

[0011] According to another aspect of this application, a differential privacy-based federated learning gradient-protected aggregation system is provided, comprising:

[0012] Inertia Construction Module: Used to obtain the local gradient vector of the current communication round and retrieve historical gradient sequences to construct an inertia tensor; wherein, the inertia tensor is constructed based on the data correlation and time evolution law of historical gradient sequences, and is used to accurately capture gradient change inertia;

[0013] Feature deconstruction module: used to deconstruct the local gradient vector into a trend feature flow within the virtual evolutionary hyperplane and a perturbation feature flow perpendicular to the virtual evolutionary hyperplane using the inertia tensor;

[0014] Budget allocation module: used to allocate a first privacy sub-budget to the trend feature stream and a second privacy sub-budget to the disturbance feature stream according to a preset total privacy budget, and establish an asymmetric allocation relationship; wherein, the first privacy sub-budget is higher than the second privacy sub-budget;

[0015] Desensitization aggregation module: used to inject noise into the trend feature stream and the perturbation feature stream using the first privacy sub-budget and the second privacy sub-budget respectively, and to perform vector synthesis to obtain a desensitized gradient vector, and to perform multi-face aggregation processing on the desensitized gradient vector to obtain a global update increment;

[0016] Correction module: used to obtain the deviation residual vector between the global update increment and the trend feature flow, use the deviation residual vector to numerically correct the weight factor of the historical gradient sequence, and use the corrected weight factor to perform a weighted operation with the historical gradient sequence retrieved in the next communication round to generate the inertia tensor of the next communication round.

[0017] According to another aspect of this application, an electronic device is provided, including a memory and a processor, the memory being used to store computer-executable instructions, and the processor being used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method described above.

[0018] According to another aspect of this application, a computer storage medium is provided that stores computer-executable instructions thereon, which, when executed by a processor, implement the steps of the method described above.

[0019] Compared with existing technologies, the differential privacy-based federated learning gradient protection aggregation method and system according to the embodiments of this application preserve the physical evolution inertia of gradients in hyperspace, ensuring that the desensitized gradient vectors can still maintain accurate update directions after aggregation, effectively avoiding non-convergence faults caused by noise interference, and significantly shortening the training cycle. Through the "trend-perturbation" dual-stream deconstruction mechanism, differentiated noise is applied to feature streams with different physical attributes. The first privacy sub-budget focuses more on preserving the fidelity of the trend feature stream, while the second privacy sub-budget focuses more on masking the perturbation feature stream. Thus, under the same total privacy budget, it satisfies the strict privacy protection requirements and significantly reduces the negative impact on model accuracy, achieving the optimal trade-off between the two.

[0020] By calculating the residual vector of the deviation between the global update increment and the local trend feature flow, and feeding it back into the correction process of the weight factor, this cross-round data interaction mechanism enables the system to dynamically adjust the construction logic of the inertial tensor in the next round based on the real-time feedback of global aggregation. This closed-loop correction design can continuously eliminate the systematic cumulative deviation caused by individual differences, ensuring that the system can still maintain extremely high stability and generalization performance in large-scale, long-cycle federated training tasks. Attached Figure Description

[0021] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 This is a schematic diagram of the overall process of the federated learning gradient protection aggregation method based on differential privacy of the present invention.

[0023] Figure 2 This is a schematic diagram of the overall framework of the federated learning gradient protection aggregation method based on differential privacy of the present invention.

[0024] Figure 3 This is an extended schematic diagram of the differential privacy-based federated learning gradient protection aggregation method of the present invention. Detailed Implementation

[0025] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0026] Example 1:

[0027] In existing technologies, federated learning, as a privacy-preserving computational paradigm for multi-party collaborative modeling, often suffers from drastic deviations in gradient vector direction when introducing differential privacy mechanisms to protect gradients, due to the typically isotropic nature of the injected noise. This directional deviation accumulates during iteration, leading to low global model convergence efficiency or even non-convergence, making it difficult for the system to achieve an effective balance between privacy protection strength and model training utility.

[0028] To address the aforementioned issues, the applicant discovered that the core reason for model failure caused by noise injection lies in disrupting the evolutionary inertia of gradients in the feature space. Analysis revealed that historical gradient sequences contain stable trends in model optimization. Based on this, a technical approach based on inertial tensor space deconstruction and asymmetric privacy protection is proposed: local gradients are deconstructed into trend feature flows and perturbation feature flows using inertial tensors, and an asymmetric privacy budget allocation relationship is established to ensure the directional stability of the trend feature flows in a low-noise environment. Simultaneously, the weight factors for the next round are corrected in real-time by obtaining the residual deviation between the global update increment and the trend flow. This application significantly improves the convergence speed and model accuracy of federated learning in privacy-preserving scenarios through precise capture and differentiated desensitization of gradient evolution trajectories, effectively eliminating non-convergence failures caused by noise injection.

[0029] Reference Figures 1-3 As an embodiment of the present invention, a federated learning gradient protection aggregation method based on differential privacy is provided, comprising: S1-S5.

[0030] To facilitate understanding, we will use a federated learning image classification task as an example. Assume the federated learning system contains three participating nodes (denoted as node A, node B, and node C), each holding its own image training dataset, collaboratively training a shared image classification model. Assume the model's gradient vector has a dimension of... (Using a smaller dimension for easier example calculations), currently at the [number]th [dimension]. Round of communication, preset historical backtracking window depth (That is, using historical gradient data from rounds 6 to 10 for calculation), the preset total privacy budget The preset total number of communication rounds .

[0031] Figure 1 The illustration shows a differential privacy-based federated learning gradient-protected aggregation method according to an embodiment of this application, specifically including:

[0032] S1: Obtain the local gradient vector of the current communication round and retrieve the historical gradient sequence to construct the inertia tensor; the inertia tensor is constructed based on the data correlation and time evolution law of the historical gradient sequence, and is used to accurately capture the gradient change inertia;

[0033] In this step, the local gradient vector refers to the gradient vector obtained by each participating node in the current communication round after performing backpropagation calculation on the shared model using the training data held locally. The dimension of this vector is consistent with the dimension of the model parameters. The historical gradient sequence refers to the ordered sequence of local gradient vectors calculated and saved by each participating node in the previous communication rounds. The vectors in this sequence are arranged in chronological order of the communication rounds.

[0034] The inertia tensor is a second-order statistic constructed based on historical gradient sequences, reflecting the evolution trend and dispersion characteristics of gradient vectors in a multi-dimensional feature space. Its mathematical form is a symmetric positive definite matrix. The construction of the inertia tensor is based on the fact that in the iterative optimization process of federated learning, gradient vectors usually exhibit a certain directional continuity and amplitude variation pattern between adjacent communication rounds. That is, the evolution of gradients has inertial characteristics. By performing covariance matrix operations on historical gradient sequences, the data correlation between each dimension and the time evolution pattern of gradients with the advancement of communication rounds can be extracted, thereby representing the gradient change inertia in the form of a tensor.

[0035] It should be noted that in the initial stage of federated learning training, when the total number of communication rounds completed is less than the preset historical backtracking window depth L, all existing historical gradient vectors can be used to construct the inertia tensor. That is, the number of communication rounds currently available can be used to replace L in the covariance matrix operation. As training progresses, when the length of the historical gradient sequence reaches or exceeds L, the historical gradient vectors of the most recent L rounds are used to construct the inertia tensor. The historical backtracking window depth L is set according to the communication frequency and model size of the federated learning task. In this application, its specific value is not limited. It can usually be selected in the range of 3 to 20 according to actual needs.

[0036] Continuing with the previous federated learning example, taking node A as an example, in the... In each round of communication, node A uses its local image dataset to perform backpropagation on the current global model, obtaining a local gradient vector with dimension 5. Simultaneously, node A retrieves its historical gradient vectors saved from rounds 6 to 10. , , , , This constitutes a historical gradient sequence. Based on this historical gradient sequence, a covariance matrix operation is used to construct a... inertial tensor The inertial tensor encodes, in matrix form, the directional evolution trend of the gradient vector of node A in the most recent 5 communication rounds and the correlation structure between its dimensions.

[0037] This step utilizes the gradient evolution trend information encoded by the inertial tensor to separate the local gradient vector of the current communication round into two feature components with different physical properties, providing a structural basis for subsequent differentiated privacy protection.

[0038] S2: Using the inertia tensor, the local gradient vector is deconstructed into a trend feature flow within the virtual evolutionary hyperplane and a perturbation feature flow perpendicular to the virtual evolutionary hyperplane;

[0039] In this step, the virtual evolutionary hyperplane refers to the subspace spanned by the principal eigenvectors of the inertial tensor. Specifically, performing eigenvalue decomposition (or singular value decomposition, or equivalent operations) on the inertial tensor yields a set of eigenvectors arranged in order of eigenvalue magnitude. Among these, several eigenvectors with larger eigenvalues ​​correspond to the main directions in which the gradient changes in magnitude during historical evolution. The subspace spanned by these directions constitutes the virtual evolutionary hyperplane. Geometrically, the virtual evolutionary hyperplane represents the low-dimensional manifold along which the gradient vector tends to evolve in the multidimensional eigenspace.

[0040] The trend feature flow refers to the orthogonal projection component of the local gradient vector onto the virtual evolution hyperplane. This component captures the part of the local gradient vector that is consistent with the historical gradient evolution trend and reflects the main direction information of model optimization. The perturbation feature flow refers to the residual component of the local gradient vector that is perpendicular to the virtual evolution hyperplane. This component reflects the fluctuation part of the local gradient vector that deviates from the historical evolution trend. It usually contains perturbation information caused by individual data differences, sampling randomness, etc. The sum of the vectors of the trend feature flow and the perturbation feature flow is equal to the original local gradient vector.

[0041] It should be noted that the dimension of the virtual evolving hyperplane depends on the number of principal eigenvectors retained after the eigenvalue decomposition of the inertia tensor. This number can be determined by accumulating energy percentage: after arranging all eigenvalues ​​of the inertia tensor in descending order, the eigenvalues ​​are added sequentially starting from the largest eigenvalue. When the sum of the accumulated eigenvalues ​​reaches a preset energy retention threshold (e.g., 90% or 95%), the number of accumulated eigenvectors is the dimension of the virtual evolving hyperplane. This method can adaptively determine the dimension of the hyperplane based on the actual distribution characteristics of the gradient data. The dimension of the hyperplane may differ at different training stages and at different participating nodes.

[0042] Continuing with the aforementioned federated learning example, taking node A as an example, the resulting... Inertia tensor Performing eigenvalue decomposition yields five eigenvalues ​​and their corresponding five eigenvectors. Assume the five eigenvalues ​​are ordered from largest to smallest as follows: , , ,

[0043] , The sum of the eigenvalues ​​is 7.0. Taking the sum of the first two eigenvalues... This accounts for approximately 81.4% of the total; the sum of the first three eigenvalues ​​is... This accounts for approximately 92.9% of the total. If the preset energy retention threshold is 90%, then the first three eigenvectors are retained. The virtual evolutionary hyperplane has a dimension of 3. The local gradient vector of node A is... Projecting onto the subspace spanned by these three eigenvectors yields the trend feature flow; The remaining component after subtracting the trend characteristic flow is the perturbation characteristic flow;

[0044] Through the aforementioned trend-perturbation feature deconstruction, this application splits the originally isotropic gradient vector into two feature components carrying different physical properties, providing a structural premise for subsequently implementing differentiated privacy protection strengths for different components. Compared with the existing technology that applies uniform intensity noise to the entire gradient vector, this deconstruction allows noise injection to be targeted at the perturbation feature flow that has a smaller impact on the model's convergence direction, thereby applying less noise interference to the trend feature flow that carries the main direction information, thus reducing the degree of damage to the gradient evolution direction by noise injection under the same total privacy budget.

[0045] After completing the trend-perturbation feature deconstruction of the gradient vector, the method enters the privacy budget allocation stage. This step is based on the different roles of the trend feature stream and the perturbation feature stream in privacy protection. The trend feature stream carries the main direction information on which the model convergence depends and is more sensitive to noise perturbation. The information carried by the perturbation feature stream has less impact on the model convergence direction, but may contain more individual privacy information. Therefore, a differentiated privacy budget allocation strategy is established.

[0046] S3: Based on the preset total privacy budget, allocate a first privacy sub-budget to the trend feature flow and a second privacy sub-budget to the disturbance feature flow, establishing an asymmetric allocation relationship; wherein, the first privacy sub-budget is higher than the second privacy sub-budget;

[0047] In this step, the total privacy budget is a parameter used in the differential privacy framework to quantify the strength of privacy protection. The smaller the value, the stricter the privacy protection, and the greater the amount of noise injected. The first privacy sub-budget and the second privacy sub-budget are the privacy allowances allocated from the total privacy budget to the trend feature stream and the perturbation feature stream, respectively. The asymmetric allocation relationship means that the value of the first privacy sub-budget is lower than that of the second privacy sub-budget. This means that the trend feature stream is allocated a larger privacy allowance (i.e., the value of the first privacy sub-budget is larger), thereby introducing less random noise in the noise injection stage to ensure the fidelity of the trend feature stream and the physical directionality of its evolution. On the other hand, the perturbation feature stream is allocated a smaller privacy allowance (i.e., the value of the second privacy sub-budget is smaller), thereby masking the privacy features of individual data by injecting higher intensity noise.

[0048] Specifically, establishing asymmetric allocation relationships includes:

[0049] Extract the orientational representation parameters of the trend feature flow in the vector space, and determine the sensitive feature values ​​of the trend feature flow to external disturbances;

[0050] In this step, the orientation characterization parameter refers to the direction and amplitude characteristics of the trend feature flow in the vector space, including but not limited to the alignment degree between the trend feature flow and the principal feature direction of the virtual evolution hyperplane, and the degree of distribution concentration of the components of each dimension of the trend feature flow; the sensitivity feature value is a scalar index determined based on the orientation characterization parameter to measure the degree of response of the trend feature flow to external disturbances. The larger the sensitivity feature value, the more sensitive the trend feature flow is to disturbances, that is, a small amount of noise may cause a shift in the trend direction; the smaller the sensitivity feature value, the more stable the current direction of the trend feature flow is, and the stronger its anti-disturbance ability.

[0051] A negative correlation mapping relationship is established between sensitive feature values ​​and the first privacy sub-budget, as shown in the following formula:

[0052] ;

[0053] in, Assign the first privacy sub-budget to the trend feature stream for the current communication round t. For the preset total privacy budget, These are the sensitive eigenvalues ​​of the trend characteristic flow to external disturbances. The preset sensitivity reference peak value, Assign an upper limit coefficient to the preset trend feature flow budget. This indicates the relative degree to which the trend characteristic flow is affected by external disturbances, and its value varies with the sensitivity characteristic value. The value increases and approaches 1, thereby reducing the first privacy sub-budget;

[0054] Sensitivity reference peak The determination method is as follows: During the initialization or pre-training phase of the federated learning system, each participating node uses its local data to perform several rounds of pre-training iterations, statistically analyzes the numerical distribution of sensitive feature values ​​corresponding to the trend feature flow in each round, and takes the upper quantile of this distribution (e.g., the value corresponding to the 95th percentile) as... ;

[0055] Adjust the value of the first privacy sub-budget based on the increase in the value of the sensitive feature;

[0056] It should be noted that the design intent of the above negative correlation mapping relationship is that when the trend feature flow is in a state sensitive to disturbances (i.e., (larger), reduce the privacy sub-budget allocated to the trend feature stream (i.e. (The noise is reduced), thus injecting less noise into the trend feature flow during subsequent noise injection to protect the trend direction from being disturbed and deflected; conversely, when the stability of the trend feature flow is good (i.e., (If the privacy subbudget is relatively small, its privacy subbudget can be appropriately increased to release more privacy budget for use by the perturbation feature stream;)

[0057] Obtain the mean of the numerical components of the perturbation feature stream, and use the mean of the numerical components to determine the privacy exposure gain of the perturbation feature stream;

[0058] In this step, the mean of the numerical components refers to the arithmetic mean of the values ​​of each dimension component in the perturbation feature flow vector. Privacy exposure gain is a scalar indicator determined based on the mean of the numerical components, used to measure the degree of risk of individual privacy information exposure in the current state of the perturbation feature flow. Its determination logic is as follows: when the mean of the numerical components of the perturbation feature flow is large, it means that the perturbation components carry more individual data-specific information; in this case, the privacy exposure gain is high, and the privacy protection for this part needs to be strengthened. When the mean of the numerical components is small, the perturbation components carry less individual-specific information, and the privacy exposure gain is low.

[0059] The privacy exposure gain is determined as follows: The arithmetic mean of the absolute values ​​of each component of the perturbation feature flow is calculated. The ratio of this mean to a preset exposure gain benchmark (determined by taking the median after statistically analyzing the distribution of the absolute values ​​of the perturbation feature flow components during the initialization or pre-training phase) is used as the privacy exposure gain. The privacy exposure gain is a dimensionless scalar; when it is greater than 1, it indicates that the privacy exposure risk of the current perturbation feature flow is higher than the benchmark level; when it is less than 1, it indicates that it is lower than the benchmark level.

[0060] Based on the total privacy budget, after deducting the first privacy sub-budget, the remaining budget is allocated to the perturbation feature stream as the second privacy sub-budget, and the second privacy sub-budget is adjusted by segmented weights according to the privacy exposure gain;

[0061] Furthermore, segment weight adjustment is performed, including:

[0062] Obtain the numerical distribution density of the perturbation feature stream in the current communication round, and determine the cumulative privacy consumption rate of the perturbation feature stream in historical communication rounds;

[0063] In this step, numerical distribution density refers to the degree of concentration of the numerical distribution of each dimension component in the perturbation feature flow vector. It can be obtained by calculating the reciprocal of the standard deviation of the numerical values ​​of each dimension component or by taking the density peak after kernel density estimation of the numerical distribution. The higher the numerical distribution density, the more concentrated the numerical distribution of each dimension component of the perturbation feature flow is, and more compensation needs to be given in the subsequent budget compensation. The cumulative privacy consumption rate refers to the average rate at which the privacy budget actually consumed by the perturbation feature flow in each round from the start of federated learning training to the current communication round. It can be obtained by summing the privacy sub-budget allocated to the perturbation feature flow in each round and dividing by the number of communication rounds completed.

[0064] Establish a three-dimensional logical relationship between privacy exposure gain, numerical distribution density, and cumulative privacy consumption rate. When the privacy exposure gain exceeds a preset risk threshold, determine the target correction interval corresponding to the second privacy sub-budget.

[0065] In this step, the preset risk threshold is determined as follows: based on the security level requirements for privacy protection in the federated learning task, and combined with the numerical distribution of privacy exposure gain obtained from the initialization or pre-training phase, the quantile corresponding to the higher privacy exposure risk in this distribution (e.g., the value corresponding to the 80th or 90th percentile) is taken as the preset risk threshold; when the privacy exposure gain is lower than the preset risk threshold, the second privacy sub-budget directly takes the remaining value after deducting the first privacy sub-budget from the total privacy budget, and no segmented weight correction is performed; only when the privacy exposure gain exceeds the preset risk threshold is the subsequent exponential weight decay and step-by-step numerical compensation triggered. The target correction range refers to the numerical range that the second privacy sub-budget is allowed to be adjusted. The lower bound of this range is the minimum budget value that ensures the basic effectiveness of the differential privacy mechanism, and the upper bound is the remaining value after deducting the first privacy sub-budget from the total privacy budget;

[0066] Within the target correction interval, the second privacy sub-budget is subjected to exponential weight decay using the cumulative privacy consumption rate. The decayed budget value is then compensated in steps based on the density of the numerical distribution, as shown in the following formula:

[0067] ;

[0068] in, The second privacy sub-budget is allocated to the perturbation feature stream for the current communication round t. For the preset total privacy budget, To determine the cumulative privacy consumption rate of the perturbation feature stream over historical communication rounds, The baseline constant for privacy consumption rate (and) (Dimensional consistency) For unit compensation step length, To determine the density of the numerical distribution of the perturbation feature flow in the current communication round t, As a reference distribution density threshold (and) (Dimensional consistency) For the preset step mapping function, The compensation order is determined based on the density of the current data distribution, and the budget that has been excessively attenuated is replenished in a stepwise manner.

[0069] In this formula, the privacy consumption rate benchmark constant The determination method is as follows: based on the preset total number of communication rounds for the federated learning task. and total privacy budget The theoretical privacy consumption rate per unit round is calculated based on the uniform consumption assumption, i.e. , As a measure of the actual cumulative rate of privacy consumption A reference benchmark deviating from the ideal uniform consumption level; during the initialization or pre-training phase, statistically analyze the numerical distribution density of the perturbation feature flow in each round, and take its mean or median as... Step mapping function It is a piecewise function that maps continuous non-negative real number inputs to discrete non-negative integer outputs. Its mapping rule is: when the input value is in... The output is 0 when the input value is in the range, and 0 when the input value is in the range. The output is 1 when the input value is in the range, and 1 when the input value is in the range. When the interval is in, output 2, and so on. ,in This is a floor function; the output value represents the compensation order. The larger the compensation order, the more privacy budget is compensated.

[0070] Continuing with the previous federated learning example, taking node A as an example, suppose that at the... In each round of communication, the sensitivity eigenvalues ​​of the trend feature flow obtained by deconstruction in step S2 to external disturbances are... Preset sensitivity reference peak (The 95th percentile value obtained during the initialization phase), substituted into the calculation formula for the first privacy sub-budget:

[0071] ;

[0072] At this point, the first privacy sub-budget is approximately 0.714, lower than the total privacy budget of 1.0. Further assumptions are made regarding the cumulative privacy consumption rate of the perturbation feature stream over historical communication rounds. , The density of the numerical distribution of the perturbation feature flow in the current round , (The mean value obtained from the initialization phase). The maximum compensation order , Substituting into the formula for calculating the second privacy sub-budget:

[0073] ;

[0074] It can be verified that the first privacy sub-budget of 0.714 is higher than the second privacy sub-budget of 0.102 (Note: here). The larger the value, the lower the protection strength and the less noise. That is, the trend feature flow is assigned less noise to maintain its directional stability, while the perturbation feature flow is assigned more noise to enhance privacy masking.

[0075] By establishing the aforementioned asymmetric allocation relationship, this application ensures that the allocation of the privacy budget is no longer a uniform apportionment of each component of the gradient vector, but rather a differentiated configuration based on the different roles each component plays in model convergence and privacy exposure. For trend feature streams, a lower privacy sub-budget means less noise injection, which helps maintain the stability of the main direction of gradient evolution after desensitization and reduces the risk of global model update direction deviation after aggregation. For perturbation feature streams, a higher privacy sub-budget means stronger noise masking, which can more effectively mask the data-specific information of individual participants. In addition, the segmented weight correction mechanism introduces the correlation between privacy exposure gain, numerical distribution density, and cumulative privacy consumption rate, enabling the allocation of the second privacy sub-budget to be dynamically adjusted with the training process. This avoids protection failure due to privacy budget exhaustion in the later stages of training, and also avoids wasting privacy allowance due to excessively conservative budget in the early stages of training.

[0076] After completing the asymmetric allocation of the privacy budget, the method enters the noise injection and aggregation stage. In this stage, the allocated privacy sub-budget is used to perform differentiated noise injection on the trend feature stream and the perturbation feature stream respectively. The noise-added feature stream is then resynthesized into a desensitized gradient vector and aggregated among the participating nodes.

[0077] S4: Noise injection is performed on the trend feature stream and the perturbation feature stream using the first privacy sub-budget and the second privacy sub-budget respectively, and vector synthesis is performed to obtain the desensitized gradient vector. The desensitized gradient vector is then subjected to multi-face aggregation to obtain the global update increment.

[0078] In this step, noise injection refers to adding random noise that meets a specific distribution to the trend feature stream and the perturbation feature stream, respectively, according to the requirements of the differential privacy mechanism. The noise distribution type can be a Laplace distribution or a Gaussian distribution that meets the definition of differential privacy. The magnitude of the noise is jointly determined by the corresponding privacy sub-budget and the sensitivity of the data to be protected. Taking the Laplace mechanism as an example, the scale parameter of the noise is equal to the data sensitivity divided by the corresponding privacy sub-budget. The smaller the privacy sub-budget, the larger the scale parameter of the noise and the higher the magnitude of the injected noise. Since the first privacy sub-budget is higher than the second privacy sub-budget, under the same data sensitivity conditions, the noise scale parameter corresponding to the trend feature stream is greater than the noise scale parameter of the perturbation feature stream. However, since the trend feature stream is mainly located in the main direction of gradient evolution, its sensitivity along the main direction is usually low (the gradients of multiple participants converge in the main direction, and the individual differences are small), while the perturbation feature stream carries individual difference information, and its sensitivity is usually higher. After combining the two, the actual noise level injected into the trend feature stream is lower than that of the perturbation feature stream.

[0079] Vector synthesis refers to combining the trend feature stream and the perturbation feature stream, which have been injected with noise, into a desensitized gradient vector with the same dimension as the original local gradient vector by vector addition. Multi-party aggregation processing refers to the process by which the central server (or aggregation coordinator) collects the desensitized gradient vectors uploaded by each participating node and performs aggregation operations on them to obtain the global update increment. The aggregation operation methods include, but are not limited to, taking the arithmetic mean of the desensitized gradient vectors of each participating node or performing a weighted average according to the proportion of data volume of each participating node. The global update increment is used to update the parameters of the shared model in the current communication round.

[0080] It should be noted that during the multi-party aggregation process, the central server receives the gradient vectors of each participating node after they have been anonymized, rather than the original local gradient vectors. This enables collaborative model updates without exposing the original individual data.

[0081] Following the aforementioned example of federated learning, in the first... In each round of communication, nodes A, B, and C each use the steps described above to inject noise into the trend feature stream and the disturbance feature stream, and then synthesize the noisy trend feature stream and the disturbance feature stream into vectors to obtain their respective desensitized gradient vectors. The three nodes upload their respective desensitized gradient vectors to the central server, which takes the arithmetic mean of the three desensitized gradient vectors to obtain the global update increment for the 10th round.

[0082] After completing the multi-party aggregation and obtaining the global update increment, the method enters the closed-loop feedback correction stage across rounds. The core of this stage is to feed the result of global aggregation back to the local system and correct the construction logic of the inertial tensor for the next round by comparing the difference between the global update increment and the local trend feature flow, so as to eliminate the cumulative effect of individual bias in long-term training.

[0083] S5: Obtain the deviation residual vector between the global update increment and the trend feature flow, and use the deviation residual vector to numerically correct the weight factors of the historical gradient sequence;

[0084] In this step, the deviation residual vector refers to the vector difference between the global update increment and the trend feature flow of the current participant node. That is, the vector obtained by subtracting the trend feature flow from the global update increment. This vector reflects the degree and direction of deviation between the local trend direction of the current participant node and the global aggregation result.

[0085] The weighting factor refers to the numerical weight assigned to the gradient vectors of each communication round in the historical gradient sequence when constructing the inertia tensor. This weighting factor is used to control the contribution of each historical gradient vector in the construction of the inertia tensor. The numerical adjustment of the weighting factor is as follows: when the magnitude of the deviation residual vector is large, it indicates that the current local trend direction deviates significantly from the global direction. In this case, the weight of the recent historical gradient vector is increased and the weight of the distant historical gradient vector is decreased so that the inertia tensor constructed in the next round reflects the recent gradient evolution trend more. When the magnitude of the deviation residual vector is small, it indicates that the local trend direction is basically consistent with the global direction, and the adjustment range of the weighting factor is reduced accordingly.

[0086] The inertial tensor for the next communication round is generated by weighting the corrected weighting factor with the historical gradient sequence retrieved in the next communication round.

[0087] Specifically, generating the inertia tensor for the next communication round includes:

[0088] The rate of change of the angle between adjacent gradient vectors in the historical gradient sequence is calculated, and the curvature feature parameters of the gradient evolution trajectory are extracted. The specific formula is as follows:

[0089] ;

[0090] in, The curvature characteristic parameter of the gradient evolution trajectory reflects the acceleration of the directional change of the gradient evolution trajectory. , and These are the local gradient vectors for the current communication round t, the previous communication round t-1, and the two previous communication rounds t-2, respectively. , and These are the magnitudes of the local gradient vectors for the current communication round t, the previous communication round t-1, and the second previous communication round t-2, respectively. The time step between adjacent communication rounds. This is a spatial angular displacement operator used to calculate the angle between two adjacent gradient vectors in a multidimensional feature space.

[0091] It should be noted that the calculation of the curvature feature parameters mentioned above depends on the local gradient vectors of the current communication round t and the two previous communication rounds t−1 and t−2. Therefore, in the initial stage of federated learning training, it is necessary to handle the case of less than three rounds of historical data. Specifically: when the current communication round is the first round (i.e., t=1), only the local gradient vector of the current round exists. Since there are no gradient vectors from the previous two communication rounds, the angle between adjacent rounds cannot be calculated. The curvature feature parameter is set to 0, and the corresponding component weight vector degenerates into the weight factor itself. That is, no additional curvature adaptive adjustment is applied to the historical gradient sequence, and the inertia tensor is directly constructed based on the currently available gradient vectors. When the current communication round is the second round (t=2), there exists... and It can calculate the angle between the gradient vectors of the second round and the first round, but it does not exist. This is used to calculate the angle between round 1 and round 0. In this case, the angle of the first segment is considered to be 0, i.e., let... It only uses an existing segment of angle information as an approximate estimate of the curvature characteristic parameters;

[0092] In this formula, A positive value indicates that the curvature of the gradient evolution trajectory is increasing, a negative value indicates that the curvature is decreasing, and zero indicates that the gradient is turning at a constant angle and speed. The time step between adjacent communication rounds is... The value of can be set to 1 (i.e., the time unit is the number of communication rounds), at which point... It is directly equal to the difference between the included angles of two adjacent segments;

[0093] Establish a numerical mapping relationship between weighting factors and azimuth curvature parameters, and determine the component weight vectors for each dimension component in the historical gradient sequence. The specific formula is as follows:

[0094] ;

[0095] in, This refers to the component weight vector for each dimension of the historical gradient sequence. The historical gradient sequence weight factor is the result of the correction for the current communication round t. The curvature characteristic parameters of the gradient evolution trajectory, The preset reference curvature constant (and) (Dimensions consistent), "1" represents a vector of all 1s; the physical meaning of this formula is: when the curvature of the gradient evolution trajectory... When it is large, Approaching 1, component weight vector The elements tend to be This is twice the weight of the gradient, meaning that greater reconstruction weights are assigned to each dimensional component to enhance the tracking response to changes in gradient direction; when When the component weight vector is small, Approaching The reconstructed weights themselves remain stable;

[0096] The physical meaning of this formula lies in: the degree of curvature of the gradient evolution trajectory When it is large, Approaching 1, component weight vector The elements tend to be This is twice the weight of the gradient, meaning that greater reconstruction weights are assigned to each dimensional component to enhance the tracking response to changes in gradient direction; when When the component weight vector is small, Approaching The reconstructed weights themselves remain stable;

[0097] Dimensional-level numerical reconstruction is performed on the historical gradient sequence retrieved in the next communication round using the component weight vector;

[0098] Among them, performing dimensional-level numerical reconstruction includes:

[0099] The gain of the numerical magnitude of each dimension component in the historical gradient sequence is adjusted by using the component weight vector to obtain the preliminary reconstructed sequence;

[0100] In this step, gain adjustment refers to multiplying the component weight vector with the historical gradient vector element by element, so that the numerical magnitude of each dimension component is scaled according to the weight of the corresponding dimension. The component magnitude corresponding to the dimension with larger weight is amplified, and the component magnitude corresponding to the dimension with smaller weight is reduced, thereby achieving differentiated adjustment of the feature contribution of each dimension of the gradient vector.

[0101] Calculate the contribution index of the preliminary reconstructed sequence in each dimension, and identify redundant dimensions whose contribution is lower than the dimension contribution threshold;

[0102] In this step, the contribution index refers to the proportion of the absolute value of the numerical amplitude of each dimension component in the preliminary reconstructed sequence to the sum of the absolute values ​​of the numerical amplitudes of all dimension components. The dimension contribution threshold is determined by sorting the contribution indices of each dimension from high to low and accumulating them sequentially. When the accumulated value reaches the preset energy retention ratio (e.g., 95%), the maximum value of the contribution index corresponding to the remaining unaccumulated dimensions is the dimension contribution threshold. Dimensions with contribution indices below this threshold are identified as redundant dimensions. This determination method allows the dimension contribution threshold to be adaptively adjusted according to the actual distribution of the data.

[0103] Obtain the projection residuals of the initial reconstructed sequence in a preset orthogonal basis space. Use the projection residuals to zero out the component values ​​of redundant dimensions, and perform step-wise energy compensation on the component values ​​of non-redundant dimensions to obtain the reconstructed historical gradient sequence. The specific formula is as follows:

[0104] ;

[0105] in, The reconstructed historical gradient vector is obtained after performing dimension-level numerical reconstruction in the t-th communication round. This refers to the component weight vector for each dimension of the historical gradient sequence. Let be the local gradient vector of the current communication round t. " indicates the operation of multiplying corresponding elements of two vectors of the same dimension.

[0106] This means using component weight vectors to stretch or compress historical gradient sequences at the dimensional level. By changing the magnitude contribution of each dimension feature, discrete noise in trajectory evolution is eliminated, thereby achieving geometric correction of the original gradient evolution direction.

[0107] In this step, the preset orthogonal basis space refers to the space spanned by orthogonal bases composed of eigenvectors of the inertial tensor. After projecting the preliminary reconstructed sequence onto this orthogonal basis space, the projection residual is the component in the preliminary reconstructed sequence that is not covered by the orthogonal basis. For redundant dimensions, the projection residual in the orthogonal basis space reflects the noise component that contributes very little to the gradient evolution direction in that dimension. It is set to zero to eliminate redundant information. For non-redundant dimensions, in order to compensate for the overall energy loss that may be caused by setting the redundant dimensions to zero, the component values ​​of the non-redundant dimensions are scaled proportionally to perform step energy compensation, so that the reconstructed gradient vector maintains a reasonable numerical amplitude while maintaining the direction correction effect.

[0108] The reconstructed sequence is then subjected to covariance matrix operations to generate the inertia tensor for the next communication round, as shown in the following formula:

[0109] ;

[0110] in, The generated inertial tensor for the next communication round is represented as A symmetric positive definite matrix of dimension 1. The preset historical backtracking window depth represents the total number of historical communication rounds involved in the calculation. This is the index subscript for the communication round iteration, with a value range from... Up to the current communication round t, For the first The reconstructed historical gradient vector obtained after performing dimension-level numerical reconstruction in each communication round. The reconstructed historical gradient sequence is the local mean vector of the central axis within a preset historical backtracking window. The "" operator is used to transpose a matrix, converting a column vector into a row vector for outer product operations.

[0111] It reflects the discrete distribution characteristics of the reconstructed historical gradient vector relative to the average evolution center within a preset historical backtracking window, and is used to extract the second-order statistical properties of the data flow in the feature space.

[0112] Continuing with the previous federated learning example, taking node A as an example, in the... After obtaining the global update increment, node A calculates the deviation residual vector between the global update increment and its local trend feature flow. Assuming the magnitude of this deviation residual vector is 0.15, it indicates a certain deviation between the local trend direction and the global aggregation result. Node A adjusts the weight factors of the historical gradient sequence based on the magnitude of the deviation residual vector, so that the gradient vectors of recent rounds (such as rounds 9 and 10) receive greater weight in the construction of the inertia tensor in the next round. Subsequently, node A calculates the angle between the gradient vectors of round 10 and round 9, and the angle between the gradient vectors of round 9 and round 8, to obtain the curvature feature parameters. Thus, the component weight vector is determined. ,use Perform dimension-level numerical reconstruction on the historical gradient vectors from rounds 6 to 10, setting redundant dimensions to zero and compensating for non-redundant dimensions to obtain the reconstructed historical gradient sequence. Finally, covariance matrix operations are performed on the reconstructed sequence to generate the inertia tensor for the 11th round of communication. .

[0113] Through the aforementioned deviation residual feedback correction mechanism, this application establishes a cross-round information loop from the global aggregation layer to the local inertial tensor construction layer. In the multi-round iteration process of federated learning, due to the differences in local data distribution, the trend feature flow of each participating node may gradually deviate from the global optimization direction. If not corrected, this deviation will continue to accumulate in the construction of the inertial tensor, eventually leading to a decrease in the accuracy of trend-perturbation deconstruction. By feeding back the deviation between the global update increment and the local trend feature flow to the correction of the weight factor, the inertial tensor can be adaptively adjusted according to the actual results of global aggregation, establishing a continuous calibration relationship between the local perspective and the global aggregation perspective of each participating node, thereby maintaining the effectiveness of gradient deconstruction and privacy protection in long-term federated training tasks.

[0114] Through the closed-loop process from S1 to S5 described above, this application completes the full iteration of "inertia tensor construction → trend-perturbation deconstruction → asymmetric privacy budget allocation → differentiated noise injection and aggregation → deviation feedback correction" in each round of communication. This allows the inertia tensor to be continuously optimized with the continuous injection of global aggregation feedback, and a continuous closed-loop correction relationship is formed between each round of communication.

[0115] Reference Figure 3In the basic process of S1 to S5 above, the inertia tensor constructed in step S1 and the asymmetric allocation relationship established in step S3 both use fixed parameter settings. However, in the actual training process of federated learning, the gradient fluctuation characteristics may change with different training stages—for example, the gradient fluctuation is more severe in the early stage of training and gradually becomes more stable in the middle and later stages of training. In order to adapt the construction of the inertia tensor and the allocation of the privacy budget to this change, this application further proposes an adaptive adjustment scheme based on the numerical fluctuation frequency; specifically, after the construction of the inertia tensor is completed in step S1, the following steps are also included:

[0116] Obtain the frequency of numerical fluctuations in historical gradient sequences within a preset time window;

[0117] In this step, the numerical fluctuation frequency refers to the average number of times the values ​​of each dimension component of the historical gradient sequence reverse direction (i.e., from increasing to decreasing or from decreasing to increasing) within a preset time window. The preset time window can be equal to or less than the depth of the historical backtracking window. This is used to focus on the fluctuation characteristics of recent gradients. The higher the frequency of numerical fluctuations, the more violently the recent gradient is oscillating, and the poorer the stability of the gradient evolution trend.

[0118] The target dimension of the virtual evolutionary hyperplane is determined based on the numerical fluctuation frequency, and the inertial tensor is projected and compressed using the target dimension.

[0119] In this step, the target dimension refers to the adjustment value of the virtual evolution hyperplane dimension under the current numerical fluctuation frequency. When the numerical fluctuation frequency is high, the evolution trend of the historical gradient is disturbed by frequent fluctuations. At this time, reducing the target dimension of the virtual evolution hyperplane (i.e., retaining fewer main feature directions) can filter out more fluctuation components and make the trend feature flow more concentrated in the core evolution direction. When the numerical fluctuation frequency is low, the target dimension can be appropriately increased to retain more trend information. Projection compression processing refers to truncating the feature space of the inertial tensor to the target dimension and retaining only the feature directions corresponding to the first few largest eigenvalues.

[0120] A mapping relationship is established between numerical fluctuation frequency and asymmetric distribution, as shown in the following formula:

[0121] ;

[0122] in, This represents the sensitivity reference peak value after numerical fluctuation frequency correction at the current communication round t. The preset initial sensitivity reference constant, To determine the frequency of numerical fluctuations in the historical gradient sequence within a preset time window, For reference fluctuation frequency benchmark (and) (Uniformity consistent); During the initialization or pre-training phase, statistically analyze the numerical distribution of the frequency of numerical fluctuations in the historical gradient sequence within a preset time window in each round, and take its mean or median as... ;

[0123] The physical meaning of this formula is: when the numerical fluctuation frequency... At higher levels, The amount was increased (up to nearly double). ), making The denominator increases, thus Decrease This increase ultimately leads to an increase in the first privacy sub-budget—that is, when gradient fluctuations are severe, more privacy budget is allocated to the trend feature flow (corresponding to less noise) to strengthen the protection of the trend direction; conversely, when... At lower levels, near The asymmetric allocation relationship returns to the baseline state; based on the increase in the frequency of numerical fluctuations, the allocation ratio of the first privacy sub-budget in the total privacy budget is increased, while the allocation ratio of the second privacy sub-budget is reduced simultaneously.

[0124] Noise injection processing is performed on the trend feature stream and perturbation feature stream after projection compression using the adjusted first privacy sub-budget and second privacy sub-budget;

[0125] Specifically, after projection compression, the feature space of the inertial tensor is truncated to the target dimension. The corresponding trend feature stream retains only the components located in the compressed virtual evolution hyperplane, while the perturbation feature stream contains the original perturbation components and the trend components truncated by projection compression. Based on this, noise is injected into the trend feature stream after projection compression using the adjusted first privacy sub-budget according to the noise injection method in step S4, and noise is injected into the perturbation feature stream using the adjusted second privacy sub-budget. Then, the two noisy feature streams are vector synthesized to obtain the desensitized gradient vector.

[0126] In the basic process of S1 to S5 above, the multi-party aggregation process in step S4 adopts an equal-weight aggregation method on the desensitized gradient vectors of each participating node. However, in the actual deployment environment of federated learning, the data quality, participation frequency, and contribution to the global model of each participating node often differ. If all nodes are aggregated equally, nodes with low contribution or unstable communication may have an undue impact on the global update increment. In addition, privacy budget, as a limited resource, needs to be reasonably allocated across rounds of training to avoid budget shortage in the later stages of training. To this end, this application further proposes an aggregation optimization scheme based on the differentiated evaluation of participating nodes and privacy budget prediction; specifically, before the multi-party aggregation process in step S4, the following steps are also included:

[0127] The specific formula for calculating the directional evolution slope of the historical gradient sequence across consecutive communication rounds is as follows:

[0128] ;

[0129] in, Let be the slope of the directional evolution in the t-th communication round (dimensionless). This is the local gradient vector for the current communication round t. This is the local gradient vector from the previous communication round t-1. This is the Euclidean norm (modulus) operator; in this formula, the directional evolution slope... The physical meaning of is to measure the ratio of the change in gradient vector between two adjacent gradient rounds to the magnitude of the gradient in the previous round. The larger the value, the more drastic the gradient change in that round; The smaller the value, the more gradual the gradient change.

[0130] And use the directional evolution slope to establish a privacy budget prediction sequence for the next communication round;

[0131] In this step, the privacy budget prediction sequence refers to the sequence generated by predicting the privacy budget consumption trend of several future communication rounds using the directional evolution slope. The logic behind this is that the directional evolution slope reflects the drasticness of gradient changes. When the gradient changes drastically, subsequent rounds may need to reserve more privacy budget for the trend feature flow to maintain directional stability. When the gradient changes gently, more privacy budget can be appropriately used to strengthen the privacy protection of the perturbation feature flow. Based on the changing trend of the directional evolution slope in the current and recent rounds, the privacy budget requirement for the next communication round can be estimated through prediction methods such as linear extrapolation or exponential smoothing.

[0132] The privacy budget prediction sequence is established as follows: using the directional evolution slope of the current communication round and several previous rounds as observation samples, the trend of the directional evolution slope is smoothed by an exponentially weighted moving average. The predicted slope value of the next communication round obtained after smoothing is used as an estimate of the drasticness of future gradient changes. Based on the predicted slope value, according to the logic that the larger the slope, the more privacy budget is required for the trend feature flow of the next round, the privacy budget demand prediction value of the next communication round is generated. The prediction values ​​of several consecutive rounds are used to form a privacy budget prediction sequence.

[0133] Obtain the cumulative contribution index and communication activity parameter of each participating node in the current communication round, and determine the personalized weighting coefficient of each participating node based on the cumulative contribution index and communication activity parameter;

[0134] In this step, the cumulative contribution metric refers to the quantitative evaluation of the cumulative contribution made by each participating node to the global model update from the start of federated learning training to the current communication round. The cumulative contribution metric can be determined by calculating the cumulative average of the consistency (e.g., cosine similarity) between the direction of the desensitized gradient vector uploaded by each participating node in each round and the global update increment. Higher consistency indicates a greater contribution from that node. The communication activity parameter refers to the frequency or number of consecutive rounds each participating node participates in during federated learning training, reflecting the stability of that node's participation in training. Personalized weighting coefficients are also used. It is a scalar coefficient determined by combining the cumulative contribution index and the communication activity parameter. Participant nodes with higher contribution and better communication activity receive a larger weighting coefficient, and vice versa.

[0135] The total privacy budget is allocated across rounds using a privacy budget prediction sequence.

[0136] The total allocation between rounds refers to the pre-allocation of the total privacy budget between the current round and future rounds based on the predicted demand values ​​of each round in the privacy budget prediction sequence. When the privacy budget prediction sequence indicates that the budget demand in future rounds will increase, the actual available privacy budget in the current round is appropriately reduced to reserve more funds for subsequent rounds; when the prediction sequence indicates that the demand in future rounds will decrease, the available budget in the current round can be appropriately increased. This total allocation between rounds ensures that the consumption rhythm of the privacy budget matches the actual trend of gradient changes during training, avoiding insufficient budget in the later stages due to excessive consumption in the early stages.

[0137] The desensitization gradient vector is then weighted and mapped according to personalized weighting coefficients, as shown in the following formula:

[0138] ;

[0139] in, The desensitized gradient vector of participant node i after mapping processing. Let i be the original desensitization gradient vector of the participating node. The personalized weighting coefficient is determined based on the cumulative contribution index. Let i be the communication activity parameter of participating node i in the current communication round t. This is an activity reference constant (usually set to 1). Let t be the directional evolution slope of the t-th communication round;

[0140] In this formula, the activity reference constant The value of is usually set to 1, and its function is to serve as a normalization benchmark for the communication activity parameter. The value of is typically between 0 and 1, representing the participating node. The communication activity level in the current communication round; the physical meaning of this formula lies in: personalized weighting coefficient. Reflects the participating nodes The historical cumulative contribution level; It is a dynamic adjustment factor based on the current communication activity and the severity of gradient changes—when the directional evolution slope... When the gradient is large (rapid changes), the value of this adjustment factor decreases, and the scaling of the desensitized gradient vector increases, thereby reducing the influence weight of this node in the aggregation when the gradient is unstable; when When it is small, the regulating factor tends to be close to Give more aggregation weight to active nodes;

[0141] Multi-party aggregation is performed using the desensitized gradient vector after weighted mapping.

[0142] Specifically, the central server collects the de-identified gradient vectors from each participating node after weighted mapping. ,in, Given the total number of participating nodes in the current communication round, perform an aggregation operation to obtain the global update increment. The aggregation operation involves summing the weighted, de-identified gradient vectors of each participating node and dividing by the total number of participating nodes. The global update increment = Since the desensitized gradient vectors of each participating node have incorporated the influence of personalized weighting coefficients and dynamic adjustment factors in the weighted mapping process, the aggregated global update increment naturally reflects the differentiated contribution weights of each participating node. Nodes with high contribution and active communication have a greater impact on the global update increment, while the impact of nodes with low contribution or drastic gradient changes is appropriately suppressed.

[0143] Following the previous federated learning example, before performing multi-party aggregation, taking node A as an example, assuming that node A has a high cumulative contribution index (its historical de-identification gradient vector has good consistency with the direction of the global update increment), its personalized weighting coefficient is determined. ; Node A's communication activity parameter in the current round directional evolution slope , Then the dynamic adjustment factor is:

[0144] The desensitized gradient vector of node A after mapping is: Similarly, nodes B and C also complete their respective weighted mapping processes. The central server collects the desensitized gradient vectors of the three nodes after mapping and then performs multi-party aggregation processing.

[0145] Example 2:

[0146] This is one embodiment of the present invention, which differs from the previous embodiment in that:

[0147] A differential privacy-based federated learning gradient-protected aggregation system, characterized by comprising:

[0148] Inertia Construction Module: Used to obtain the local gradient vector of the current communication round and retrieve historical gradient sequences to construct an inertia tensor; the inertia tensor is constructed based on the data correlation and time evolution law of the historical gradient sequence, and is used to accurately capture the gradient change inertia;

[0149] Feature deconstruction module: used to deconstruct the local gradient vector into a trend feature flow within the virtual evolutionary hyperplane and a perturbation feature flow perpendicular to the virtual evolutionary hyperplane using the inertia tensor;

[0150] Budget allocation module: used to allocate a first privacy sub-budget to the trend feature stream and a second privacy sub-budget to the perturbation feature stream according to the preset total privacy budget, and establish an asymmetric allocation relationship; wherein, the first privacy sub-budget is higher than the second privacy sub-budget;

[0151] Desensitization and aggregation module: It is used to inject noise into the trend feature stream and the perturbation feature stream using the first privacy sub-budget and the second privacy sub-budget respectively, and to synthesize vectors to obtain desensitized gradient vectors. The desensitized gradient vectors are then aggregated in multiple ways to obtain the global update increment.

[0152] Correction module: Used to obtain the deviation residual vector between the global update increment and the trend feature flow, use the deviation residual vector to numerically correct the weight factors of the historical gradient sequence, and use the corrected weight factors to perform weighted operations with the historical gradient sequence retrieved in the next communication round to generate the inertia tensor of the next communication round.

[0153] Example 3:

[0154] In one embodiment of the present invention, which differs from the previous embodiment, the electronic device includes one or more processors and a memory.

[0155] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0156] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0157] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). In addition, depending on the specific application, the electronic device may include any other suitable components.

[0158] Example 4:

[0159] Embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps described in the "Exemplary Methods" section above according to the various embodiments of this application.

[0160] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0161] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not restrict the application from being implemented using the specific details described above.

[0162] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0163] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0164] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0165] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A federated learning gradient-preserving aggregation method based on differential privacy, characterized in that, include: Obtain the local gradient vector of the current communication round and retrieve the historical gradient sequence to construct an inertia tensor; wherein, the inertia tensor is constructed based on the data correlation and time evolution law of the historical gradient sequence, and is used to accurately capture the gradient change inertia; The local gradient vector is deconstructed using the inertia tensor into a trend feature flow within the virtual evolutionary hyperplane and a perturbation feature flow perpendicular to the virtual evolutionary hyperplane. Based on a preset total privacy budget, a first privacy sub-budget is allocated to the trend feature stream, and a second privacy sub-budget is allocated to the disturbance feature stream, establishing an asymmetric allocation relationship; wherein, the first privacy sub-budget is higher than the second privacy sub-budget; The first privacy sub-budget and the second privacy sub-budget are used to inject noise into the trend feature stream and the perturbation feature stream respectively, and vector synthesis is performed to obtain the desensitized gradient vector. The desensitized gradient vector is then subjected to multi-party aggregation to obtain the global update increment. Obtain the deviation residual vector between the global update increment and the trend feature flow, use the deviation residual vector to numerically correct the weight factor of the historical gradient sequence, and use the corrected weight factor to perform a weighted operation with the historical gradient sequence retrieved in the next communication round to generate the inertia tensor of the next communication round. The generation of the inertial tensor for the next communication round includes: Calculate the rate of change of the angle between adjacent round vectors in the historical gradient sequence, and extract the curvature feature parameters of the gradient evolution trajectory; Establish a numerical mapping relationship between the weighting factor and the curvature feature parameter, and determine the component weight vector for each dimension component in the historical gradient sequence; The component weight vector is used to perform dimensional-level numerical reconstruction on the historical gradient sequence retrieved in the next communication round, and the covariance matrix operation is performed on the reconstructed sequence to generate the inertia tensor of the next communication round.

2. The differential privacy-based federated learning gradient-protected aggregation method according to claim 1, characterized in that, Following the construction of the inertia tensor, the following is also included: Obtain the numerical fluctuation frequency of the historical gradient sequence within a preset time window; The target dimension of the virtual evolutionary hyperplane is determined based on the numerical fluctuation frequency, and the inertial tensor is projected and compressed using the target dimension. Establish a mapping relationship between the numerical fluctuation frequency and the asymmetric allocation relationship. As the numerical fluctuation frequency increases, increase the allocation ratio of the first privacy sub-budget in the total privacy budget, and simultaneously decrease the allocation ratio of the second privacy sub-budget. The noise injection process is performed on the trend feature stream and the disturbance feature stream after projection compression using the adjusted first privacy sub-budget and second privacy sub-budget.

3. The differential privacy-based federated learning gradient-preserving aggregation method according to claim 1, characterized in that, Prior to the multi-party aggregation process, the following is also included: Calculate the directional evolution slope of the historical gradient sequence between consecutive communication rounds, and use the directional evolution slope to establish a privacy budget prediction sequence for the next communication round; Obtain the cumulative contribution index and communication activity parameter of each participating node in the current communication round, and determine the personalized weighting coefficient of each participating node based on the cumulative contribution index and communication activity parameter; The total privacy budget is allocated across rounds using the privacy budget prediction sequence, and the anonymized gradient vector is weighted and mapped according to the personalized weighting coefficients. The multi-party aggregation process is performed using the desensitized gradient vector after weighted mapping.

4. The differential privacy-based federated learning gradient-preserving aggregation method according to claim 1, characterized in that, The establishment of the asymmetric allocation relationship includes: Extract the orientation representation parameters of the trend feature flow in the vector space, and determine the sensitivity feature values ​​of the trend feature flow to external disturbances; Establish a negative correlation mapping relationship between the sensitive feature value and the first privacy sub-budget, and adjust the value of the first privacy sub-budget according to the increase of the sensitive feature value; The mean of the numerical components of the perturbation feature stream is obtained, and the privacy exposure gain of the perturbation feature stream is determined using the mean of the numerical components. Based on the total privacy budget, after deducting the first privacy sub-budget, the remaining budget is allocated to the perturbation feature stream as the second privacy sub-budget, and the second privacy sub-budget is subjected to segmented weight correction according to the privacy exposure gain.

5. The differential privacy-based federated learning gradient-preserving aggregation method according to claim 4, characterized in that, The execution of segmented weight correction includes: Obtain the numerical distribution density of the perturbation feature stream in the current communication round, and determine the cumulative privacy consumption rate of the perturbation feature stream in historical communication rounds; A three-dimensional logical relationship is established between the privacy exposure gain, the numerical distribution density, and the cumulative privacy consumption rate. When the privacy exposure gain exceeds a preset risk threshold, the target correction interval corresponding to the second privacy sub-budget is determined. Within the target correction interval, the second privacy sub-budget is subjected to exponential weight decay using the cumulative privacy consumption rate, and the decayed budget value is compensated in steps according to the density of the numerical distribution.

6. The differential privacy-based federated learning gradient-preserving aggregation method according to claim 1, characterized in that, The numerical reconstruction at the execution dimension level includes: The gain of the numerical magnitude of each dimension component in the historical gradient sequence is adjusted using the component weight vector to obtain a preliminary reconstructed sequence; Calculate the contribution index of the preliminary reconstructed sequence in each dimension, and identify redundant dimensions whose contribution is lower than the dimension contribution threshold; The projection residual of the preliminary reconstructed sequence in a preset orthogonal basis space is obtained. The projection residual is used to zero out the component values ​​of the redundant dimension, and step-by-step energy compensation is performed on the component values ​​of the non-redundant dimension to obtain the reconstructed historical gradient sequence.

7. A differential privacy-based federated learning gradient-protected aggregation system, used to implement the gradient-protected aggregation method according to any one of claims 1-6, characterized in that, include: Inertia Construction Module: Used to obtain the local gradient vector of the current communication round and retrieve historical gradient sequences to construct an inertia tensor; wherein, the inertia tensor is constructed based on the data correlation and time evolution law of historical gradient sequences, and is used to accurately capture gradient change inertia; Feature deconstruction module: used to deconstruct the local gradient vector into a trend feature flow within the virtual evolutionary hyperplane and a perturbation feature flow perpendicular to the virtual evolutionary hyperplane using the inertia tensor; Budget allocation module: used to allocate a first privacy sub-budget to the trend feature stream and a second privacy sub-budget to the disturbance feature stream according to a preset total privacy budget, and establish an asymmetric allocation relationship; wherein, the first privacy sub-budget is higher than the second privacy sub-budget; Desensitization aggregation module: used to inject noise into the trend feature stream and the perturbation feature stream using the first privacy sub-budget and the second privacy sub-budget respectively, and to perform vector synthesis to obtain a desensitized gradient vector, and to perform multi-face aggregation processing on the desensitized gradient vector to obtain a global update increment; Correction module: used to obtain the deviation residual vector between the global update increment and the trend feature flow, use the deviation residual vector to numerically correct the weight factor of the historical gradient sequence, and use the corrected weight factor to perform a weighted operation with the historical gradient sequence retrieved in the next communication round to generate the inertia tensor of the next communication round.

8. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method as described in any one of claims 1 to 6.

9. A computer storage medium storing computer-executable instructions thereon, characterized in that: When the computer-executable instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 6.

Images