A safe and efficient federated learning method and device based on error feedback and norm perception, and a storage medium

Abstract

This invention discloses a secure and efficient federated learning method, device, and storage medium based on error feedback and norm awareness, belonging to the fields of artificial intelligence security and distributed machine learning. This method filters and accumulates important gradient information through gradient compression and error feedback mechanisms, ensuring information integrity and unbiased estimation. It also integrates an adaptive differential privacy mechanism based on gradient norm, dynamically adjusting noise intensity according to importance to optimize the privacy budget. The server aggregates the processed gradients and broadcasts updates to complete iterative training. Theoretically, this invention satisfies (ε,δ)-differential privacy and convergence guarantees, significantly reducing communication overhead, effectively resisting member inference attacks, and maintaining high model accuracy and convergence speed in various learning tasks. It provides a reliable solution for federated learning applications in privacy-sensitive scenarios, achieving a triple balance between communication efficiency, model utility, and privacy security.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence security and distributed machine learning technology, and in particular to a secure and efficient federated learning method, device and storage medium based on error feedback and norm awareness. Background Technology

[0002] In numerous fields such as the Internet of Things, fintech, and healthcare, data, as a core production factor, is increasingly demonstrating its value. Distributed machine learning technology, with its ability to collaboratively utilize data among multiple entities without the need for centralized aggregation of raw data, shows broad application prospects in balancing data utilization and privacy protection. Federated learning is the core application paradigm within this technological framework.

[0003] The core training logic of federated learning is as follows: The central server first distributes the initial AI model to multiple clients (also known as worker nodes). Each client trains its local model based on its own private data. After training, it only sends the model update gradients back to the central server, which then aggregates these gradients to update the global model. Through multiple rounds of model update iterations and exchanges, the training of the entire AI model is finally completed. Depending on the application scenario and deployment architecture, federated learning can be specifically divided into three categories: terminal device federated learning, edge federated learning, and cloud federated learning, each adapting to different device scales, network environments, and business needs.

[0004] However, existing federated learning methods still face many critical technical bottlenecks that urgently need to be addressed, severely restricting their large-scale application in privacy-sensitive scenarios and complex network environments:

[0005] First, privacy protection has shortcomings. While existing methods can avoid the direct exposure of raw data, the iterative exchange of model parameters and gradients during model training becomes a key risk point for privacy leakage. Traditional federated learning systems are threatened by member inference attacks (MIA). Attackers can analyze the transmitted gradient information or model parameters to infer whether a specific sample belongs to the client's local training dataset, leading to the leakage of user privacy data. This poses a serious obstacle to applications in fields with extremely high privacy protection requirements, such as healthcare and finance.

[0006] Secondly, communication overhead remains high. In practical applications such as edge computing, the number of devices involved in training is often extremely large, and network bandwidth is usually limited and network connection stability is insufficient. In such environments, the communication overhead generated by transmitting model gradient updates becomes a core factor restricting training efficiency. Relevant data shows that communication overhead can account for more than 80% of the total training time in federated learning. Excessive communication transmission not only prolongs the model training cycle but may also cause training interruptions due to network fluctuations, seriously affecting the practicality and efficiency of federated learning.

[0007] Third, the privacy budget is allocated unreasonably. In federated learning privacy mechanisms, masking sensitive gradient information by adding noise is a common technique, but most existing methods employ a fixed-intensity noise addition strategy. In reality, the norm of a gradient vector is closely related to its value in model updates: gradients with larger norms typically contain more critical model update information; adding excessive noise to these gradients severely compromises model convergence, leading to a significant drop in final model accuracy. Conversely, gradients with smaller norms carry limited effective information; adding noise of equal intensity to important gradients to them is a waste of the privacy budget, failing to achieve the optimal balance between privacy protection and model utility, and reducing the overall efficiency of privacy protection.

[0008] Therefore, designing a federated learning scheme that can simultaneously achieve a balance between communication efficiency optimization, model accuracy assurance, and privacy security protection has become a key requirement for promoting the large-scale application of federated learning in privacy-sensitive scenarios and complex network environments, and has important theoretical significance and practical application value. Summary of the Invention

[0009] To address the aforementioned shortcomings of existing technologies, this invention provides a secure and efficient federated learning method, device, and storage medium based on error feedback and norm awareness. Based on error feedback gradient compression and norm-aware adaptive differential privacy technology, this invention solves the problems of high communication overhead, vulnerability to member inference attacks, and imbalance between privacy protection and model utility in federated learning. It meets differential privacy requirements, effectively resists privacy attacks, significantly improves communication efficiency, and maintains high model accuracy, achieving a triple balance between communication efficiency, model accuracy, and privacy protection. It is adaptable to practical application carriers such as IoT sensing terminals, medical data processing equipment, and financial risk control terminals, ensuring the security of training and transmission of real and sensitive data such as medical records and financial transactions, and supporting the large-scale deployment of multi-device collaborative training in privacy-sensitive scenarios.

[0010] The specific technical solution of the present invention is as follows:

[0011] This invention provides a secure and efficient federated learning method based on error feedback and norm awareness, which is used for image classification or natural language processing tasks, and includes the following steps:

[0012] Step S1: Server initializes global model parameters Configure hyperparameters; set N to the total number of participating clients. For the first Each client, each participating client Collect raw sensor data and convert it into local training data At the same time, initialize the local cumulative error vectors respectively. and parameter delay counter ;in p This is used to identify whether the counter is for a parameter, that is, to record the number of delay rounds associated with the parameter;

[0013] Step S2, each client From its local training data Sample one batch and calculate the global model parameters for round t. Local stochastic gradients ;

[0014] Step S3, for The gradient compression and privacy-preserving DCPP process is performed, which includes sequentially performing gradient compression and error feedback GCEF and an adaptive differential privacy mechanism, and is implemented through the following steps S4 to S10:

[0015] Step S4: Calculate the cumulative gradient for the current round. ,in The client stored the accumulated error from the previous round. As of the t -1 rounds of all untransmitted gradient information;

[0016] Step S5, Calculation Find the absolute value of all values, sort them in descending order, and take the first few values ​​after sorting. The minimum value corresponding to the gradient of the ratio is used as the threshold α, where s is the gradient compression ratio;

[0017] Step S6: For each dimension of the gradient vector Generate binary mask ,like or Then set the selection mask. Set to 1 if it is not 1, otherwise set to 0; where Indicates the first t In round-robin communication, the client i The d dimensional gradient value, Indicates client i The dThe number of consecutive iterations the gradient has undergone since its last propagation. This represents the final output. A value of 1 indicates that the dimension was selected for upload in this round, while a value of 0 indicates that the dimension was not uploaded in this round. Its information will be retained in the accumulated error, awaiting compensation in subsequent rounds. This represents the upper limit used in the error feedback compression mechanism to control the continuous ignoring of gradient information;

[0018] Step S7: Based on the obtained mask Calculate the compressed gradient = ,in This represents element-wise multiplication; where This represents the gradient after being filtered by the mask, i.e., the gradient information that is ready to be transmitted in this round but has not yet been noise-added;

[0019] Step S8: Record the gradient information that was not transmitted in this round for use in the next round of compensation. And update the parameter delay counter, the calculation formula is: = +1 (1- );

[0020] Step S9: Based on a preset privacy budget gradient Sensitivity and noise scaling constant Based on the local original gradient norm Dynamically calculate the adaptive noise intensity to be added in this round. The calculation formula is as follows: This adaptive differential privacy mechanism, in conjunction with the gradient compression and error feedback processes defined in steps S4 to S8, jointly ensures that the entire federated learning training process meets the requirements. Differential privacy requirements; this requirement is a rigorous mathematical framework that provides strong privacy guarantees by ensuring that the inclusion or exclusion of any single data point in the dataset has a negligible impact on the computational output;

[0021] Step S10: Sample noise from the Gaussian distribution and add it to the compressed gradient, i.e. ,in ;in This indicates that the gradient finally uploaded to the server by the client after complete processing by the DCPP module includes compressed and noisy information. yes d An identity matrix of dimension 1 represents noise that is independent and homoscedastic in each dimension;

[0022] Step S11: The client will process the gradient. Send to the server;

[0023] Step S12: The server collects all clients' [data / information / data]. And calculate the global gradient: ;in This indicates that the server aggregates the amount of security updates uploaded by all clients. The resulting raw aggregation result, where N represents the number of clients participating in each round of federated learning aggregation;

[0024] Step S13: The server will generate the global gradient. Gradient compression and privacy protection are performed to obtain the processed global gradient. and the processed global gradient Broadcast to all clients; among them Indicates server to The global update amount is sent to the client after DCPP processing again;

[0025] Step S14: After receiving the global update, each client updates its local model according to the gradient descent rule: ;

[0026] Step S15: Repeat steps S2 to S14 until the preset total number of training rounds is reached. Or the model converges.

[0027] Preferably, in step S1, the local training data The dataset includes at least one of image classification data and text classification data; wherein the image classification data includes handwritten digit image datasets, natural scene object image datasets, or fine-grained image datasets; the text classification data includes news text datasets, movie review text datasets, or multilingual text datasets; and the local training data is divided among the clients in a non-independent and identically distributed manner.

[0028] The hyperparameters include: the first t Global model parameters of the wheel Indicates the server is in the t The global model state and client index sent at the start of round communication i The first to participate in federal learning i Number of clients, total number of communication rounds This indicates the total number of iterations and learning rate in the predefined federated learning training. This indicates the step size of the model update and the gradient compression ratio. Privacy Budget Differential privacy probability parameters Noise scaling constant Maximum number of delay rounds and the gradient clipping threshold C; wherein, the gradient compression ratio The value range is 80%-99%, and the privacy budget... The value range is 0.1-8.0, and the differential privacy probability parameter... The noise scaling constant is a fixed minimum value much less than 1, typically set to be less than the reciprocal of the size of the client's local dataset. The value range is 0.01-2.0, and the maximum number of delay rounds... The value range is 3-10, and the gradient clipping threshold C is used to limit the gradient. Norm, and used as the gradient in differential privacy noise addition. Sensitivity The value of .

[0029] Preferably, in step S6, the binary mask generation rule combines importance filtering and a delay-forced transmission mechanism to ensure that any gradient component is at most... It must be transferred once within the wheel.

[0030] Preferably, in step S8, the update rule for the parameter delay counter is as follows: for unselected gradient dimensions, the corresponding counter is incremented by 1; for selected gradient dimensions, the counter is reset to 0.

[0031] Preferably, in step S13, when downlink bandwidth is tight and it is necessary to further reduce the amount of transmitted data or when privacy protection needs to be enhanced in the downlink direction, the server will process the global gradient before performing the broadcast. The data is then re-entered into the gradient compression and privacy protection module for further processing to reduce downlink communication overhead or enhance privacy protection in the downlink direction.

[0032] Furthermore, the method is applied to image classification tasks or natural language processing tasks. For image classification tasks, the AlexNet model is trained on the MNIST, CIFAR-10, or CIFAR-100 datasets. For natural language processing tasks, the Transformer model is trained on the AGNews, IMDb, or SWEDISH datasets.

[0033] The present invention also provides a terminal device, comprising:

[0034] One or more processors;

[0035] The memory is communicatively connected to the processor;

[0036] A communication unit connected to the processor;

[0037] The memory stores computer programs;

[0038] When the computer program is executed by the processor, the terminal device is configured as a client or server to implement the federated learning method, wherein:

[0039] When configured as a client, the processor, the corresponding part of the memory, and the communication unit work together as a client processing unit, a client storage unit, and a client communication unit, respectively, to perform local gradient calculation, gradient compression and privacy protection processing based on error feedback and norm awareness, and gradient uploading and update reception with the server.

[0040] When configured as a server, the processor, the corresponding portion of the memory, and the communication unit work together as a server processing unit, a server storage unit, and a server communication unit, respectively, to perform gradient aggregation, global gradient compression and privacy protection processing, and broadcast global updates to clients.

[0041] Preferably, when configured as a client:

[0042] (1) The client storage unit is used to store the local training dataset. Local model parameters Cumulative error vector and gradient delay counter ;

[0043] (2) The client processing unit is used for:

[0044] Perform step S2, from its local training data Mid-sampling and computation of local stochastic gradients ;

[0045] Perform steps S3 to S10 to... Perform gradient compression and privacy protection operations, including: calculating the cumulative gradient. Generate mask And compress and update the accumulated error. With delay counter Adaptive noise is added based on the gradient norm to obtain the protected gradient. ;

[0046] Execute step S14, based on the received global update Update local model parameters ;

[0047] (3) The client communication unit is used for:

[0048] Execute step S11 to process the gradient. Send to the server;

[0049] Receive global updates from the server or original gradient .

[0050] Preferably, when configured as a server:

[0051] (1) The server storage unit is used to store global model parameters. ;

[0052] (2) The server processing unit is used for:

[0053] Perform step S12 to collect gradients from each client. And calculate the global gradient ;

[0054] Execute step S13, for Gradient compression and privacy protection are performed to obtain the processed global gradient. ;

[0055] (3) The server communication unit is used for:

[0056] Receive gradients uploaded by each client ;

[0057] Execute step S13 to process the global gradient. Broadcast to all clients.

[0058] The present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the federated learning method described above. The computer program is configured to, when executed by the processor, perform an iterative training process in the order of steps S2 to S15, including local gradient calculation, gradient compression and error feedback, adaptive differential privacy noise addition, gradient uploading, server-side gradient aggregation and processing, global update reception, and local model update.

[0059] The beneficial technical effects of this invention are as follows:

[0060] 1. This invention employs a joint optimization method of error feedback gradient compression and gradient-sensitive adaptive differential privacy. It achieves "compression-compensation" closed-loop control through a selection mask based on Top-k quantization and an accumulated error feedback mechanism. An adaptive noise mechanism based on the gradient norm dynamically adjusts the differential privacy noise intensity, while strictly satisfying... This method maximizes the preservation of key gradient information under the premise of differential privacy, significantly reduces the negative impact of noise on model convergence, and rationally allocates communication resources, privacy budget, and computational resources of the federated learning system. It effectively resists privacy attacks such as member inference and solves the problem of privacy leakage in the model parameter and gradient iteration exchange process of existing federated learning methods. The two core characteristics of this joint optimization method, low communication overhead and strong privacy protection, are adaptable to the limited bandwidth and low power consumption hardware constraints of practical application devices such as IoT terminals and mobile medical devices. This reduces energy consumption during data transmission and extends the device's battery life. At the same time, it ensures the security of real and sensitive data such as medical records and financial transactions during training and transmission, directly guaranteeing the data compliance of terminal devices in privacy-sensitive scenarios. This significantly enhances the practicality and large-scale deployment capability of federated learning technology in real fields such as healthcare and finance.

[0061] 2. This invention provides a complete solution that combines theoretical convergence and privacy protection. Through rigorous mathematical derivation, it is proven that the gradient update after compression and perturbation satisfies the convergence conditions of unbiasedness and bounded variance, thus theoretically guaranteeing the convergence of the global model. By linking the dynamic allocation of the privacy budget with the importance of the gradient, a strict privacy protection boundary is provided, further enhancing the privacy protection effect and filling the gap in existing technologies where it is difficult to balance privacy protection and model convergence.

[0062] 3. This invention can make full use of the communication resources of the federated learning system. Through high-proportion gradient sparsity and local error accumulation, it reduces the amount of data to be transmitted in each round of communication by about 90% without sacrificing the final accuracy of the model. This significantly reduces the communication overhead of distributed training and is suitable for network environments with limited bandwidth. It effectively solves the problems of long training cycles and easy interruption in scenarios with a large number of devices and unstable networks, and improves the feasibility and efficiency of federated learning in real-world scenarios.

[0063] 4. This invention achieves complementary adaptation of technical characteristics through the collaborative design of gradient compression and error feedback (GCEF) and adaptive differential privacy (ADP) sub-modules. It can maintain excellent performance in various tasks such as computer vision and natural language processing, and solves the problems of insufficient adaptability and large performance fluctuation of existing federated learning methods in different task scenarios. It provides reliable support for the large-scale cross-domain application of federated learning. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the overall architecture of the APFLEF federated learning system based on the error feedback compression module and the adaptive differential privacy module in Embodiment 1 of the present invention;

[0065] Figure 2 This is a schematic diagram comparing the model accuracy of various baseline methods on multiple datasets in Embodiment 3 of the present invention;

[0066] Figure 3 This is a comparison of the convergence curves of the test accuracy of various baseline methods in Embodiment 3 of the present invention on the CIFAR-10 and AGNews datasets as a function of training time;

[0067] Figure 4 This is a comparison of the convergence curves of various baseline methods in Embodiment 3 of the present invention on the CIFAR-10 and AGNews datasets, showing the change in test accuracy with training rounds.

[0068] Figure 5 This is a radar chart for the comprehensive evaluation of the attack and defense effect of the member inference in Embodiment 3 of the present invention, where (a) is CIFAR-10 and (b) is AGNews;

[0069] Figure 6 This is a schematic diagram of the gradient norm evolution of the adaptive noise mechanism and the fixed noise mechanism during the training process in Embodiment 3 of the present invention;

[0070] Figure 7 This is a comparison chart of the communication efficiency and computational overhead balancing ability between the error feedback compression module and Top-k in Embodiment 3 of the present invention;

[0071] Figure 8 This is a comparison of the convergence of the error feedback compression module and the Top-k model under different gradient compression ratios in Embodiment 3 of the present invention, where (a) is the diagram of the error feedback compression module and (b) is the diagram of the Top-k model.

[0072] Figure 9 This is a comparison chart of the final test accuracy and accuracy reduction of the error feedback compression module and Top-k under different gradient compression ratios in Embodiment 3 of the present invention. Detailed Implementation

[0073] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0074] Example 1: Method Example

[0075] This embodiment provides a secure and efficient federated learning method (APFLEF) based on an error feedback compression module and an adaptive differential privacy module. This method can be executed by a terminal device and storage medium. In a specific implementation scenario of this embodiment, taking image classification as an example, the AlexNet model is trained on the CIFAR-10 dataset. The trained model is directly deployed on a surveillance camera terminal, which can classify objects in real time from natural scene images captured by the terminal. When a preset dangerous item, illegal equipment, or other abnormal target is identified, the terminal immediately triggers an audio-visual warning signal and pushes it to the management platform. If this method is applied to the natural language processing task of a medical text analysis terminal, the model can automatically extract key diagnostic and treatment information such as symptom descriptions and medication history from medical record texts, assisting doctors in quickly formulating treatment plans and realizing the direct conversion of model training results into physical device functions. The method specifically includes the following steps:

[0076] Step S1: The central server initializes the global model parameters. Configure hyperparameters, including the total number of communication rounds. Learning rate Gradient compression ratio Privacy Budget Differential privacy probability parameters Noise scaling constant Maximum number of delay rounds Each participating client initializes its local cumulative error vector. and parameter delay counter =0, and the specific explanations of each parameter are as follows:

[0077] (1) Gradient compression ratio The value range is 80%-99%, preferably 90%, which means that the top 10% of the most important gradient elements are retained for transmission;

[0078] (2) Privacy budget in differential privacy The value range is 0.1-8.0, probability parameter Defined as the maximum permissible probability of privacy protection failure, it is a fixed minimum value much less than 1, usually set to be less than the reciprocal of the size of the client's local dataset;

[0079] (3) Noise scaling constant : is an adjustable hyperparameter, with a value range of 0.01-2.0. It can be adjusted... The value can precisely control the trade-off between the overall level of privacy protection and model utility;

[0080] (4) Maximum number of delay rounds The value ranges from 3 to 10, which is used to strike a balance between communication efficiency and gradient information integrity, ensuring that no gradient component is transmitted indefinitely with a delay.

[0081] (5) Gradient clipping threshold C and gradient Sensitivity To meet differential privacy requirements and ensure stable training, local gradients need to be pruned to limit their size. Norm. The gradient clipping threshold C is the preset upper bound for clipping. During training, the local gradient is calculated. All cutting should be performed according to the following rules: If Then Scaling to Otherwise keep This operation ensures that the maximum gradient change caused by a change in a single sample remains unchanged. (Sensitivity) does not exceed C. Therefore, in the noise calculation formula of step S9, the gradient... The sensitivity is set to the clipping threshold C.

[0082] In this embodiment, the key hyperparameters used and their design basis are as follows:

[0083] (1) Total number of communication rounds =200: Based on the complexity of the CIFAR-10 dataset and the training dynamics of the AlexNet model, ensure that the model converges fully;

[0084] (2) Learning rate =0.01: This is a commonly used initial learning rate when training CNN models such as AlexNet, which can achieve a good balance between convergence speed and stability;

[0085] (3) Gradient compression ratio =90%: Setting this value can maintain a high compression ratio to significantly reduce communication overhead, while ensuring the effective preservation of key gradient information through the error feedback mechanism. It is an empirical balance point between communication efficiency and model accuracy.

[0086] (4) Privacy Budget =1.0: This value is commonly used in differential privacy research, and can provide meaningful privacy protection without excessively compromising model utility;

[0087] (5) Noise scaling constant =0.1: According to parameter sensitivity analysis, this value achieves the optimal trade-off between privacy protection (reducing the success rate of MIA attacks) and model utility (maintaining high accuracy);

[0088] (6) Maximum number of delay rounds =5: This setting ensures that any untransmitted gradient components are delayed for a maximum of 5 rounds, striking a trade-off between avoiding permanent loss of information and preventing premature increases in communication overhead.

[0089] Additional notes regarding the training data:

[0090] The local training data The data originates from the client's own business environment and sensing devices. The business environment specifically includes practical application scenarios such as intelligent visual recognition (e.g., handwritten digit recognition, natural scene object classification), news topic classification, movie review sentiment analysis, and multilingual text classification. Sensing devices include image acquisition devices (e.g., surveillance cameras, mobile terminal cameras, scanners) and text input devices (e.g., computers, smart terminals, business system servers). The corresponding local training data types and specific examples are: image classification data (e.g., handwritten digit images from the MNIST dataset, natural scene object images from the CIFAR-10 / CIFAR-100 dataset, and fine-grained item classification images), and text classification data (e.g., news report text from the AGNews dataset, movie review text from the IMDb dataset, and multilingual everyday text from the SWEDISH dataset). Before being used for training, this raw data must undergo preprocessing (e.g., format standardization, noise reduction, word segmentation) on the client side to form data that the model can directly use. All preprocessing is done offline, and the raw data remains on the client's local machine, not participating in network transmission, thus protecting privacy from the source.

[0091] This method follows a clear closed-loop data flow: "local computation → compression and encryption → upload and aggregation → broadcast update". Training begins with server initialization and client local state initialization. In each training round, the client calculates gradients using local data, processes them through the core gradient compression and privacy protection module (DCPP), and uploads the secure, compressed gradients. The server aggregates all client gradients and broadcasts the global update back to the clients, driving local model updates. This closed-loop iterative process ultimately outputs a fully trained global model with differential privacy protection, while all original data remains locally.

[0092] Step S2: In round t, each worker node... From its local data Sample a batch of data and calculate the current model. Local stochastic gradients .

[0093] Step S3: Calculate the results The input gradient compression and privacy protection module (DCPP) is used for processing. This process consists of two sub-steps: gradient compression and error feedback (GCEF) and adaptive differential privacy (ADP). First, the gradient is compressed through the gradient compression and error feedback sub-step. Then, adaptive differential privacy sub-step adds adaptive noise based on the gradient norm to the compressed gradient to obtain the processed gradient. The two sub-steps are tightly coupled and constitute the adaptive core of the APFLEF framework. The processing of this module is specifically implemented through the following steps S4 to S10.

[0094] The DCPP module is the core processor for achieving collaborative optimization in this invention, and its internal data flow is executed strictly in sequence according to the following steps:

[0095] S31, GCEF sub-step: Receive the original gradient First, compare it with the accumulated error from the previous round of storage. The sums form the cumulative gradient to be evaluated. Then, compression based on importance filtering and delay mechanisms is performed, and the compression gradient is output. and update internal status. and ;

[0096] S32, ADP sub-step: Receive the compressed gradient output by GCEF According to the original gradient The noise intensity is dynamically calculated using the norm, and then noise is added to it. The final output is a compressed and privacy-preserving gradient. .

[0097] The two sub-steps are interconnected: GCEF ensures information integrity, while ADP ensures security and privacy budget efficiency.

[0098] The method was implemented on the CIFAR-10 dataset using the aforementioned specific parameters. Experimental results show that the APFLEF framework can reduce the accuracy of member inference attacks from 75.80% to 60.18% while maintaining the model accuracy of 81.19%, reducing the communication volume by 89.83% and increasing the training speed by 4.61 times, which fully verifies the comprehensive effectiveness of the method.

[0099] Step S4: Calculate the cumulative gradient for the current round. ,in This is the accumulated error saved from the previous round. This is the core of the error feedback mechanism, ensuring that untransmitted gradient information is effectively accumulated.

[0100] Step S5, Calculation Find the absolute value of all values, sort them in descending order, and take the first few values ​​after sorting. The minimum value corresponding to the gradient of the proportion is used as the threshold α, so that the proportion of gradient components whose absolute value is greater than α after sorting does not exceed a certain limit. , where s is the gradient compression ratio, that is, the gradient components to be transmitted are determined by Top-k selection.

[0101] Step S6: For each dimension of the gradient vector Perform the following operations to generate a binary mask. :like or Then set the selection mask. It is 1 if it is true, otherwise it is 0. This indicates whether the d-th gradient dimension of the i-th client is selected for transmission during the t-th round of training; The cumulative gradient calculated in step S4 represents the cumulative gradient value of the gradient vector of the i-th client in the d-th dimension during the t-th round of training. It serves as the decision benchmark for gradient compression in this round. It is the cumulative value formed by adding the historical untransmitted error to the current new gradient, ensuring that the gradient information that is temporarily ignored can be compensated and transmitted in subsequent rounds. Record how many consecutive rounds the d-th gradient dimension has not been transmitted in the i-th client; This is the maximum delay epoch, a safety threshold set to ensure that no gradient information is lost. It enforces that any gradient dimension must be within a continuous delay epoch. Each gradient component must be transmitted at least once within a round. This rule combines importance filtering and delayed forced transmission mechanisms to ensure that any gradient component is transmitted at most once. The information is transmitted once within the wheel, thus preventing permanent loss.

[0102] A specific example is as follows: To concisely illustrate the mask generation rules, a 10-dimensional cumulative gradient vector is used as an example (in practical applications, the gradient dimension is usually very high). Assume a client... The cumulative gradient obtained in turn Given a 10-dimensional vector, the absolute values ​​of each dimension are... =[0.1,0.3,2.1,0.05,1.8,0.02,0.5,0.15,3.0,0.4]. Set the gradient compression ratio s=90% and the maximum delay rounds. =5.

[0103] Step S61: First, calculate the threshold. Sort the above absolute value sequence from largest to smallest to get [3.0, 2.1, 1.8, 0.5, 0.4, 0.3, 0.15, 0.1, 0.05, 0.02].

[0104] s=90% means that the gradient dimension needs to be compressed by 90%, retaining only the gradient dimension of 1-s=10% for transmission. 10% of the 10-dimensional gradient corresponds to 1 dimension, meaning that the gradient with the largest absolute value needs to be retained;

[0105] To ensure that only this one maximum gradient is selected, the threshold α needs to be set to the critical value of this maximum gradient, that is, the value immediately following the first maximum gradient (3.0) after sorting, which is the second value after sorting, 2.1. At this time, only gradients with an absolute value greater than 2.1 can be selected, which just satisfies the requirement of retaining one maximum gradient, so α=2.1.

[0106] Step S62: Subsequently, apply the rules to generate a mask:

[0107] For dimensions =3 (value 2.1): because This is not true, and assumes that its delay counter currently has the following value. < Then this round =0;

[0108] For dimensions =9 (value 3.0): because Since the importance condition is met, let's assume... =1;

[0109] For dimensions =1 (value 0.1): If its delay counter current value =4. Because and This round of setting =0. According to the subsequent update rules, its counter will become 5 after this round. In the next round (round t+1), because This dimension will be forcibly selected, that is =1.

[0110] This step integrates the following dual mechanisms, which are key to ensuring model accuracy at high compression ratios:

[0111] (1) Instantaneous importance selection ( ): Filter and transmit the part with the largest magnitude in the current accumulated gradient to ensure that this round of updates targets the most urgent model optimization direction;

[0112] (2) Delayed forced transmission ( This is one of the key designs of this invention. The principle is that the importance of parameters changes dynamically during the training process of a deep neural network. A gradient dimension that seems "minor" in the current iteration may become crucial as the model training enters a new stage. If ignored for a long time, the parameters corresponding to that dimension will not be updated effectively, leading to model bias or convergence stagnation. (This is achieved through a delay counter.) and threshold The mandatory transmission mechanism provides a "safety net" update opportunity for all gradient dimensions, fundamentally ensuring that information in any dimension will not be discarded indefinitely, and maintaining the long-term integrity of gradient information at the system level.

[0113] Step S7: Based on the obtained mask Calculate the compressed gradient = ,in This represents element-wise multiplication;

[0114] Step S8: Record the gradient information that was not transmitted in this round for use in the next round of compensation. And update the cumulative counter: = +1 (1- The update rule is as follows: for unselected gradient dimensions (i.e., masks) For dimensions where 0 is equal to 1, increment the corresponding counter by 1; for the selected dimension (i.e., the dimension where 0 is equal to 1), increment the counter by 1. (dimension = 1), the counter is reset to 0;

[0115] This step is a concrete implementation of the "error feedback" concept and is the core difference between this invention and simple dropout compression (such as Top-k). Its operational logic is as follows: In this round, due to the mask... Gradient information that was applied but not transmitted will be fully saved in the new accumulated error vector. middle.

[0116] The underlying principle is that this untransmitted information is not useless; it reflects the current model's update tendency in secondary directions. By accumulating it... This information will be used in the next round of training along with the new gradient. Merging and re-engaging in the importance competition. This mechanism allows historical gradient information to continuously accumulate in the error vector, and it can take effect once its cumulative effect becomes significant or the forced transmission condition is triggered. This is equivalent to smoothing and integrating gradient information in the time dimension, so that even at a high compression ratio (e.g., 90%), the long-term information throughput still approaches full gradient update, thereby maximizing the model's convergence and final accuracy while ensuring communication efficiency.

[0117] Step S9: Based on the original gradient The norm dynamically adjusts the noise intensity, and the calculation formula is: where e is the natural constant. It's a privacy budget. It is the noise scaling constant. It is gradient Sensitivity is typically taken as the gradient clipping threshold. This formula is the core of this invention for achieving adaptive noise injection: the gradient norm. The larger the value, the more important the information, and the less noise. The smaller the value, the better to protect model performance; a smaller gradient norm adds relatively more noise, thus efficiently utilizing the privacy budget. This adaptive noise mechanism, combining the gradient compression and error feedback processes described above, ensures that the entire federated learning training process meets the requirements. Differential privacy requirements.

[0118] This design embodies the refined management of privacy budgets, and it integrates the ideas of information theory and optimization theory:

[0119] (1) Using the gradient norm as a proxy for information importance: It intuitively reflects the overall strength of this round of local updates and can effectively measure the "task-related information content" it carries. The larger the norm, the more critical the correction of the model direction by this update is usually.

[0120] (2) Design motivation for the inverse exponential relationship: the formula is based on Establish noise intensity The relationship between the gradient norm and the gradient norm is inversely proportional. This nonlinear design enables extremely fine-grained allocation of the privacy budget:

[0121] ① For important gradients: The value is maximized, making This becomes extremely small. This means that only a tiny amount of noise is added, thereby preserving the integrity of key gradient information to the maximum extent, ensuring the model's convergence dynamics and utility.

[0122] ②For minor gradients: The value is small. Relatively large. This means that strong perturbations can be applied to them, thereby efficiently consuming the privacy budget across these dimensions. This achieves a more thorough obfuscation of privacy.

[0123] (3) Advantages compared to fixed-noise schemes: Traditional fixed-noise schemes (such as Fixed-DP) add noise of the same intensity to all gradients. When the gradient norm decays naturally in the later stages of training, fixed noise will severely damage the signal-to-noise ratio, causing the model to fail to converge. The adaptive mechanism of this invention can dynamically adjust the noise and always maintain a better signal-to-noise ratio, thereby significantly reducing the damage to model performance while providing equal or even stronger privacy protection.

[0124] In summary, this mechanism treats the privacy budget as a "limited resource" and allocates it dynamically and non-uniformly based on the amount of gradient information, achieving a Pareto improvement between privacy protection and model utility, that is, improving performance in at least one aspect without harming others.

[0125] Step S10: Sample noise from the Gaussian distribution and add it to the compressed gradient, i.e. ,in .

[0126] Step S11: The worker node processes the gradient. Send to the parameter server.

[0127] Step S12: The server collects all worker nodes'... And calculate the global gradient: ;

[0128] Step S13: The server will generate the global gradient. Gradient compression and privacy protection are performed to obtain the processed global gradient. and the processed global gradient The broadcast is sent to all worker nodes; the server side may choose to re-input the global gradient update into the gradient compression and privacy protection module for further reduction of downlink communication overhead or enhancement of downlink privacy protection.

[0129] Step S14: After receiving the global update, each worker node updates its local model according to the gradient descent rule: .

[0130] Step S15: Repeat steps S2 to S14 until the preset total number of training rounds is reached. Or the model converges.

[0131] This embodiment constructs a complete and implementable closed-loop process from data to model:

[0132] (1) Input and starting point: Local private datasets of each client , as well as the initialized global model and related states;

[0133] (2) Iterative processing flow: In each round of communication In this process, the data sequentially undergoes the following steps: local gradient calculation → DCPP module processing → secure uplink transmission → server-side aggregation → optional downlink processing → downlink broadcast → client-side local model update. Among these, the GCEF and ADP sub-steps within the DCPP module are the core of this flow, respectively addressing the trade-off between communication efficiency and privacy security.

[0134] (3) State update and loop progression: After each round, the client updates its local model parameters. With internal state ( These states serve as inputs for the next iteration, driving the model to optimize progressively.

[0135] (4) Final output: After the iteration is completed, the system outputs the global model parameters after training. .

[0136] Throughout the process, the raw data The data remains locally on the client; only compressed and noise-protected intermediate gradients are transmitted over the network, making it impossible to reverse engineer the original data; all critical states are maintained locally. This design clearly outlines the entire closed-loop process of data generation locally, participation in training, and final formation of the global model.

[0137] The core operating mechanism, cross-task applicability, and theoretical support of the method in this embodiment are as follows:

[0138] (1) Adaptive collaborative logic of the APFLEF framework: Through joint optimization of gradient compression and privacy protection, the communication efficiency of federated learning is significantly improved while protecting data privacy. Small noise is added to important gradients with large norms to maximize their value in model updates; relatively strong noise is added to gradients with small norms to achieve efficient use of the privacy budget;

[0139] (2) Error feedback integrity guarantee of DCPP module: If a certain gradient component is in continuous If a transmission is not selected in round -1, its accumulated error will gradually increase. Wheel time due to satisfaction ≥ Conditions are forcibly transmitted to ensure that important gradient information is not lost;

[0140] (3) Privacy-utility balance mechanism of adaptive differential privacy: For important gradients with large magnitudes, the exponential term The larger size leads to a higher noise standard deviation. Smaller noise is used to protect model performance; for minor gradients with smaller magnitudes, relatively larger noise is added to effectively prevent privacy leaks.

[0141] (4) Cross-task validity verification: In the computer vision task, the AlexNet model was trained on the CIFAR-10 dataset, and the compression ratio was set. =90%, Privacy Budget =1.0, noise scaling constant =0.1, Maximum cumulative rounds =5. Experimental results show that APFLEF, while maintaining the model accuracy of 81.19%, reduces the member inference attack accuracy from 75.80% to 60.18%, reduces communication volume by 89.83%, and improves training speed by 4.61 times. In natural language processing tasks, using the Transformer model trained on the AGNews dataset, APFLEF, while maintaining the model accuracy of 86.04%, reduces the member inference attack accuracy from 90.40% to 72.20%, reduces communication volume by 89.85%, and improves training speed by 2.41 times.

[0142] (5) Theoretical guarantee: The APFLEF framework has been rigorously proven through mathematical derivation to satisfy the following conditions. Differential privacy requirements, and the model can be guaranteed to converge in the gradient compression scenario. The convergence conditions include: (1) the gradient after compression and privacy protection is an unbiased estimate of the original gradient; (2) the variance of the estimation error is bounded.

[0143] Example 2: Terminal Equipment and Storage Medium Example

[0144] This embodiment provides a terminal device and a computer-readable storage medium, which are used together to execute the secure and efficient federated learning method based on error feedback and norm awareness described in Embodiment 1. Its overall architecture can be found in the appendix. Figure 1 The system diagram shown is shown below.

[0145] I. Terminal Equipment

[0146] The terminal device includes: one or more processors; a memory communicatively connected to the processors; a communication unit connected to the processors; and one or more computer programs stored in the memory, which, when executed by one or more processors, enable the processors to implement the various steps of the method described above.

[0147] The terminal device, through the coordinated operation of its processor, memory, and communication unit, performs corresponding functions when configured for different roles, as follows:

[0148] (a) When configured as a client

[0149] When the terminal device is configured as a client (i.e., a worker node) in the federated learning system, its processor, corresponding parts of the memory, and communication unit work together as a client processing unit, a client storage unit, and a client communication unit, respectively.

[0150] 1. Client-side storage unit

[0151] This is used to store the local state data and training-related data for each client, specifically including: the local training dataset. (Original perceptual data, such as visual images, environmental sensors, and entity interaction text, collected by the client, converted from these data) Local model parameters Cumulative error vector and gradient delay counter The cumulative error vector is mentioned above. With gradient delay counter As the data is in a dynamic state, its calling and updating logic during the training process is as follows:

[0152] (1) When to call stored data: The processor calls these state data as needed when performing gradient compression and privacy protection (DCPP) operations in each training round;

[0153] ① During the gradient compression and error feedback (GCEF) step: the processor reads the cumulative error vector updated in the previous round from memory. Used to calculate the cumulative gradient of the current round. ;

[0154] ② When generating the gradient selection mask according to the delay mechanism: the processor reads the gradient delay counter from memory. The current value is used to determine whether each gradient dimension has reached the maximum delay round. However, forced transmission is required.

[0155] (2) Logic for updating stored data: At the end of each round of DCPP operation, the processor synchronously triggers the update of these two state data;

[0156] ① Update of the cumulative error vector: According to the formula The gradient information of those not selected in this round is calculated as the new cumulative error and immediately written back to memory, overwriting the old value. This update must occur in each training round;

[0157] ② Gradient delay counter update: based on the selection mask generated in this round According to the rules = +1 (1- The system updates the counter value. The specific triggering conditions are: for gradient dimensions with a mask of 0 (unselected), the corresponding counter value is incremented by 1; for dimensions with a mask of 1 (selected), the counter is reset to 0. The updated counter value is then written back to memory.

[0158] 2. Client Processing Unit

[0159] The steps for performing the federated learning method described in Example 1 specifically include:

[0160] (1) Execute step S2 to read from the storage unit and Calculate the local stochastic gradient ;

[0161] (2) Perform steps S3-S10 to... Perform gradient compression and privacy protection (DCPP) operations, including: reading and Calculate the cumulative gradient = + According to the preset compression ratio Calculate the selection threshold and combined Generate selection mask The compression gradient is obtained. ;according to Norm calculation of adaptive noise intensity and generate noise. and Adding them together yields the protected gradient. Subsequently, updates were made. and And write it back to the storage unit;

[0162] (3) Execute step S14, based on the global update received from the server. Update local model parameters And store it in the storage unit;

[0163] (4) It is also used to preprocess the raw sensing data (such as format standardization and noise reduction) and convert it into local training data. .

[0164] 3. Client communication unit

[0165] Used to execute step S11, the processing unit generates Send to the server; and execute steps S13-S14 to receive the global update from the server. .

[0166] (ii) When configured as a server

[0167] When configured as a server, the processor, the corresponding portion of the memory, and the communication unit work together as a server processing unit, a server storage unit, and a server communication unit, respectively, to perform gradient aggregation, global gradient compression and privacy protection processing, and broadcast global updates to clients.

[0168] 1. Server storage unit

[0169] Used to store global model parameters ;

[0170] 2. Server processing unit

[0171] Execute step S12, and receive the data uploaded by each client through the communication unit. The global gradient is obtained by performing aggregation calculations. ;

[0172] Execute step S13, for Perform the DCPP operation to obtain the processed global gradient. .

[0173] 3. Server communication unit

[0174] Execute step S12 to receive gradients from each client;

[0175] Execute step S13 and process the data generated by the processing unit. Broadcast to all clients.

[0176] The client and server processing units collaborate through their respective communication units to jointly complete the iterative training process as described in Example 1: the client calculates and uploads the protected gradient locally → the server aggregates the gradients → the server issues a global update → the client updates the local model.

[0177] (III) Multi-processor configuration instance (edge ​​and cloud collaboration)

[0178] In some specific implementations, the plurality of processors may include edge server processors and cloud server processors, forming a hierarchical collaborative computing architecture:

[0179] 1. Edge Server: Used to coordinate multiple worker nodes (clients) in an edge computing environment, performing local model aggregation operations, and acting as a bridge between worker nodes and cloud servers. Its program logic executes the following specific steps:

[0180] (1) Interaction process with worker nodes: In each training round, the edge server is responsible for coordinating the training process of its subordinate worker nodes, specifically including:

[0181] ① Coordinated data acquisition: Trigger or synchronize local data sampling and gradient calculation on each node;

[0182] ② Gradient Receiver: Receives local gradients uploaded by each node and processed by its local DCPP module. ;

[0183] ③ Model distribution: Global model updates received from the cloud server Distribute to all subordinate nodes to drive them to update their local models.

[0184] (2) Specific execution steps of local model aggregation: To reduce uplink communication overhead with the cloud server, the edge server can perform local aggregation. This step is the application of the logic described in step S12 (global gradient aggregation) of Example 1 to the local node set. Specifically, the edge server uses the same federated averaging algorithm as in step S12 to perform local aggregation on the gradients of its subordinate nodes. To perform aggregation, that is, to apply a formula For the subset of nodes under its jurisdiction, the local aggregate gradient is calculated. This calculation result will then be prepared for uploading to the cloud server.

[0185] (3) Data interaction with the cloud server: The edge server acts as a bridge between the local unit and the central cloud, and its data interaction process is as follows:

[0186] ① Upload the local aggregation results: upload the above local aggregation gradients. (Or the original gradients of each node that have not been locally aggregated) are uploaded to the cloud server to participate in global aggregation;

[0187] ② Receive global model: Receive global updates generated and broadcast by the cloud server. Then, the model distribution task in the above-mentioned interaction process with worker nodes is completed.

[0188] 2. Cloud Server: Used as a central server to perform global model aggregation and distribution operations. Its core execution logic strictly follows the process in Example 1, as follows:

[0189] (1) Global gradient aggregation and processing logic: In each training round The cloud server's communication unit receives gradients directly uploaded by all clients and processed by their local DCPP module. Subsequently, the processing unit performs aggregate calculations, that is, according to the formula... The original global gradient is calculated. Then, this original global gradient... The gradient compression and privacy protection (DCPP) module, which is fed into the cloud server, processes the data to obtain the final globally updated update that has undergone consistent compression and privacy protection and is ready for distribution. ;

[0190] (2) Broadcast process for global updates: The output of the DCPP module mentioned above This is the global model update amount for this round. The cloud server's communication unit then... The broadcast is sent to all participating clients, thus completing the server-side work for this round.

[0191] (3) Collaborative closed loop with clients: The cloud server and all clients together form a collaborative training closed loop of "collection-computation-distribution". The specific process is as follows:

[0192] ① Collection: Collect gradient updates calculated and uploaded by all clients based on local data;

[0193] ② Calculate the update: Execute the global gradient aggregation and DCPP processing logic described above to generate a safe global update;

[0194] ③ Distribute the results: Broadcast the global update to all clients to start a new round of training. This closed-loop iteration continues until the model converges.

[0195] II. Computer-readable storage media

[0196] The storage medium is a computer-readable storage medium storing a computer program. When executed by a processor, this computer program can implement the various steps of the federated learning method described in this invention. Specifically, the computer program includes multiple logical modules, which work together to precisely implement the method described in this invention. Its core structure and execution logic include:

[0197] (1) Gradient processing and update control module: This module is the core execution engine of the program, which is based on the total number of training rounds. Iterative control is performed. In each round... In this process, it schedules the following sub-modules in sequence:

[0198] ① Local gradient calculation submodule: This submodule is called to calculate gradients from the local dataset. Sampling data and based on the current model parameters Calculate the local stochastic gradient ;

[0199] ② Error Feedback Compression (GCEF) Submodule: This submodule implements gradient compression and error feedback logic. Its execution steps include: calculating the cumulative gradient. According to the preset compression ratio Determine the threshold Read and determine the gradient delay counter. According to the rules (if) or Generate binary selection mask ; Calculate the compressed gradient = And update the cumulative error. and delay counter For use in the next round;

[0200] ③ Adaptive Noise Addition (ADP) Submodule: This submodule implements gradient-sensitive privacy protection logic. Its execution steps include: reading the original gradient norm. and hyperparameters (privacy budget) Noise scaling constant Gradient sensitivity According to the formula Calculate adaptive noise intensity; generate Gaussian distribution. noise The final output is a privacy-preserving gradient. ;

[0201] ④ Model Update Submodule: This submodule receives a global update broadcast from the server. Then, based on the learning rate According to the rules Update local model parameters.

[0202] (2) Hyperparameter Configuration and Management Module: This module is responsible for persistently storing, initializing, loading, and supplying all hyperparameters required for the training process during runtime. The hyperparameters include the total number of rounds. Learning rate Compression ratio Privacy Budget Noise constant Maximum number of delay rounds and gradient sensitivity It provides a unified parameter access interface for the aforementioned processing submodules.

[0203] (3) Execution Scheduling and Communication Coordination Module: This module defines the timing and triggering rules for program execution. The rules are as follows: Using the training rounds as the basic cycle, within each cycle, the gradient calculation, compression, and noise addition sub-modules on the client side are triggered sequentially, and the results are sent to the server; on the server side, gradient aggregation logic is triggered; after aggregation is completed, broadcast logic to all clients is triggered; after receiving the broadcast, the client triggers its local model update sub-module. This cycle continues until the preset total number of rounds is reached. .

[0204] Example 3: Experimental Verification

[0205] This embodiment conducts experimental verification and result analysis on the secure and efficient federated learning method based on the APFLEF framework described in Embodiment 1, and the terminal device and storage medium described in Embodiment 2, in order to fully demonstrate the technical effectiveness and practicality of the present invention.

[0206] (I) Experimental Platform and Environment

[0207] Hardware environment: Server equipped with Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz, NVIDIA RTX4090 GPU, and Gigabit Ethernet environment;

[0208] Software environment: The operating system is Ubuntu 18.04.6 LTS, the programming language is Python 3.7, and the experiments are based on the PyTorch deep learning framework and the FLGo federated learning framework.

[0209] (II) Experimental Design

[0210] To validate the effectiveness of the APFLEF framework, we conducted experiments on diverse datasets, tasks, and models, and compared it with representative baseline methods on various metrics.

[0211] (1) Experimental objective

[0212] Verify the performance advantages of the APFLEF framework in terms of communication efficiency, model accuracy, privacy protection capabilities, component collaboration, and engineering applicability.

[0213] (2) Experimental tasks, datasets and models

[0214] ①Task types: Covering two typical tasks: computer vision (CV) and natural language processing (NLP);

[0215] ② Datasets: The MNIST, CIFAR-10, and CIFAR-100 datasets were used for CV tasks, while the AGNews, IMDb, and SWEDISH datasets were used for NLP tasks. The selection of these datasets was based on their generally accepted representativeness and difficulty gradient, aiming to systematically verify the effectiveness and generalization ability of the proposed method. Specifically, the MNIST, CIFAR-10, and CIFAR-100 datasets used for computer vision (CV) tasks are academically recognized benchmark datasets, with task complexity increasing sequentially, allowing for a comprehensive evaluation of the method's performance in image classification scenarios ranging from simple to complex. The AGNews, IMDb, and SWEDISH datasets used for natural language processing (NLP) tasks cover diverse classification scenarios such as news topic classification, sentiment analysis, and multilingual text classification, enabling the testing of the method's adaptability to different text task types. By testing on these two classic datasets with different data modalities, complexities, and task types, this experimental design comprehensively evaluates the robustness and universality of the proposed framework.

[0216] ③ Models: AlexNet was used for CV tasks, and Transformer was used for NLP tasks. The selection of these models was based on their representativeness in their respective fields and the evaluation requirements of the experiments: AlexNet, as a classic convolutional neural network (CNN) architecture, is often used as a benchmark model for computer vision tasks. Its clear structure helps to verify the effectiveness of this invention in image classification tasks; Transformer is a cornerstone model in the field of natural language processing. Based on the self-attention mechanism, its parameter scale and gradient characteristics differ significantly from CNNs. Choosing it allows for testing the adaptability of this invention under different model architectures. By testing on these two representative models with significant differences, this experiment aims to comprehensively evaluate the overall performance and generalization of the framework of this invention in diverse federated learning scenarios.

[0217] (3) Baseline method and comparison settings

[0218] Under the condition of non-independent and identically distributed (Non-IID) data partitioning, it is compared with a number of mainstream federated learning methods, including FedAvg (a baseline method without compression / privacy protection), QSGD (an 8-bit quantized stochastic gradient descent method), Top-k (a gradient sparsification method), and FedFDP (a differential privacy method with fixed noise).

[0219] (4) Evaluation indicators

[0220] Communication efficiency: communication time, communication volume, compression ratio, speedup ratio, and computational percentage;

[0221] Model performance: accuracy, convergence speed;

[0222] Privacy protection capabilities: accuracy, precision, recall, F1 score, area under the receiver operating characteristic curve (AUC), and attack surface area for member reasoning attacks (MIA);

[0223] Component synergy and robustness: performance of different module combinations, impact of gradient compression ratio on accuracy, and impact of noise scaling constant c on privacy-utility tradeoff.

[0224] (5) Experimental parameters

[0225] The experiment is configured with 8 worker nodes, 200 communication rounds per experiment, a batch size of 32, and a learning rate of [missing information]. =0.01, gradient compression ratio =0.9, maximum gradient cumulative rounds =5.

[0226] (III) Experimental Results and Analysis

[0227] (1) Communication efficiency verification

[0228] Experiments validated APFLEF's resource-efficient utilization characteristics: In CV tasks, APFLEF achieved a gradient compression rate of 89.83% on CIFAR-10 and CIFAR-100, resulting in training speedups of 4.61 times and 5.08 times respectively compared to FedAvg; on the MNIST dataset, the single-round training time was reduced to 3.48 seconds, with a compression rate comparable to Top-k. As shown in Table 1, APFLEF outperforms FedAvg and QSGD in terms of communication overhead, compression rate, single-round time, and speedup on CV datasets, achieving a compression rate comparable to Top-k while offering shorter single-round times and higher speedups.

[0229] The differences in acceleration effects across different datasets fully demonstrate the working mechanism of the GCEF module in this invention and its adaptability to model complexity. The classification task on the MNIST dataset is relatively simple, using fewer model parameters and having a lower gradient vector dimension. In this case, although the GCEF module can achieve a high proportion of selective compression, the communication time accounts for a small percentage of the total training time due to the relatively small original communication base. Therefore, while a considerable absolute speedup is achieved, the acceleration gains are capped because computation time becomes the primary overhead.

[0230] In contrast, the CIFAR-10 and CIFAR-100 datasets are more complex. The models used for training, such as AlexNet, have a large number of parameters and high gradient vector dimensions, resulting in a huge amount of raw communication in the FedAvg benchmark, making the communication bottleneck extremely prominent. The GCEF module demonstrates its advantages in such high-dimensional gradient scenarios. Its compression operation can eliminate massive amounts of redundant gradient transmission, reducing the communication volume per round by about 90%, thereby shortening the training time per round from tens of seconds to several seconds, achieving a higher absolute speedup.

[0231] This comparison demonstrates that the gradient compression and error feedback mechanism proposed in this invention has a more significant efficiency improvement capability for complex task scenarios with a large number of model parameters and severe communication bottlenecks.

[0232] Table 1. Comparison of communication efficiency of various baseline methods on computer vision datasets.

[0233]

[0234] Note 1: The "×" in the table is a multiplication sign, indicating the corresponding numerical multiple relative to the benchmark method FedAvg.

[0235] Note 2: "-" in the table indicates that the metric is not applicable; the workflow of the baseline algorithm FedAvg does not include a gradient compression step, so its compression ratio is not defined.

[0236] In NLP tasks, APFLEF reduced communication by 89.85% across all datasets, achieving a speedup of 4.97 times on the IMDb dataset and more than 3 times the single-round time on the SWEDISH dataset. It's important to note that this experiment used the same Transformer model with identical structure and parameter count across all NLP tasks. For FedAvg (uncompressed) and QSGD (fixed compression ratio) methods, the theoretical single-round communication volume is determined solely by the model parameter count and compression ratio, resulting in identical communication volume values ​​across the AGNews, IMDb, and SWEDISH datasets. This design aims to control for the communication variable of model size, thereby focusing on comparing the performance differences of each algorithm in core mechanisms such as gradient compression, error feedback, and adaptive noise addition under fair conditions. Table 2 shows that APFLEF also demonstrates outstanding communication efficiency on NLP datasets such as AGNews, IMDb, and SWEDISH, reducing communication by nearly 90% compared to FedAvg, achieving a speedup more than twice that of QSGD, and significantly reducing single-round time compared to other comparative methods.

[0237] Table 2. Comparison of communication efficiency of various baseline methods on natural language processing datasets.

[0238]

[0239] As shown in Table 3, APFLEF effectively saves training time. In CIFAR-10, the communication time decreased from 7.66 seconds in FedAvg to 0.78 seconds, while the computation time percentage increased from 36.74% to 85.95%. As shown in Table 4, APFLEF increased the computation time percentage to 96.13% on the AGNews dataset. These results demonstrate that APFLEF achieves lower single-round training time and higher training throughput while maintaining a high compression ratio, exhibiting excellent communication resource utilization efficiency.

[0240] The following explanation clarifies the relevant concepts related to training time breakdown to facilitate a clear understanding of the data in the subsequent tables: In Tables 3 and 4, "Computation Time" specifically refers to the core computational time spent on forward and backward propagation and gradient calculation within the client's local model; "Communication Time" specifically refers to the network communication time spent transmitting processed gradient data to the server and broadcasting global updates downstream. In Tables 1 and 2, "Single Round Time" represents the total time for a complete training round, encompassing the aforementioned core computation and communication time, as well as other necessary system overhead such as data loading and process synchronization. Therefore, the sum of "Computation Time" and "Communication Time" in Tables 3 and 4 is typically less than the corresponding "Single Round Time" in Tables 1 and 2.

[0241] Table 3. Comparison of training time breakdown for various baseline methods on computer vision datasets.

[0242]

[0243] Table 4. Comparison of training time breakdown for various baseline methods on natural language processing datasets

[0244]

[0245] (2) Model accuracy verification

[0246] Figure 2 This study demonstrates APFLEF's accuracy advantages on multiple CV and NLP datasets: APFLEF consistently outperforms mainstream compression methods such as QSGD and Top-k while maintaining a high gradient compression rate. Specifically, in CV tasks, APFLEF achieves accuracies of 97.96%, 81.44%, and 45.21% on MNIST, CIFAR-10, and CIFAR-100, respectively, closely matching the uncompressed FedAvg benchmark and significantly outperforming other compression methods. In NLP tasks, APFLEF achieves accuracies of 87.91%, 79.98%, and 92.53% on the AGNews, IMDb, and SWEDISH datasets, respectively, essentially on par with FedAvg and outperforming QSGD and Top-k.

[0247] To further evaluate the balance between efficiency and accuracy, Figure 3 The results show the accuracy changes over training time on the CIFAR-10 and AGNews datasets. APFLEF exhibits a faster convergence speed, achieving a higher final accuracy than Top-k within the same time window: On CIFAR-10, APFLEF achieves a 4.77x speedup compared to FedAvg, with an accuracy decrease of only 0.12%, far lower than the decreases in accuracy for QSGD and Top-k; on AGNews, APFLEF achieves a 2.44x speedup while losing only 0.02% of accuracy. Furthermore, Figure 4 The training dynamics curves shown indicate that APFLEF maintains rapid convergence throughout the training process, with the final accuracy consistently approaching the FedAvg benchmark and significantly outperforming QSGD and Top-k.

[0248] (3) Verification of privacy protection effectiveness

[0249] Figure 5 Tables 5 and 6 verify the effectiveness of APFLEF in defending against member inference attacks: The experiment uses a gradient-based white-box attack model, and a binary classifier is trained by analyzing backpropagation gradients to determine the membership of samples. The privacy protection performance of FedAvg, QSGD, FedFDP and APFLEF is compared on six datasets.

[0250] Table 5. Performance Comparison of Multiple Baseline Methods in Defending Against MIA on Computer Vision Datasets

[0251]

[0252] Table 6. Performance Comparison of Multiple Baseline Methods in Defending Against MIA on Natural Language Processing Datasets

[0253]

[0254] Experimental results show that APFLEF can significantly reduce the risk of privacy leakage: Table 5 shows that on the CIFAR-10 dataset, the attack AUC decreased from 0.8591 for FedAvg and 0.7285 for QSGD to 0.6039 for APFLEF. This effect is mainly due to the adaptive noise mechanism of the ADP module in this invention. This mechanism dynamically adjusts the noise intensity according to the gradient norm: relatively strong noise is added to gradient components with smaller norms to effectively blur potentially sensitive patterns; while finely tuned weak noise is added to key gradient components with larger norms to maintain their model update value as much as possible while still effectively interfering with their statistical characteristics. This non-uniform noise injection based on gradient importance systematically destroys the distribution consistency of the original gradient vector, making it difficult for the inference model constructed by the attacker to extract stable discriminative features from the confused gradient updates, thus significantly weakening its member inference ability and causing a sharp drop in the AUC index.

[0255] Similar improvements were observed on the CIFAR-100 and MNIST datasets; Table 6 shows that APFLEF reduced the attack AUC on the AGNews and IMDb datasets to 0.8179 and 0.8263, respectively, both the lowest among all compared methods. While FedFDP also provides privacy protection, it results in a significant loss of model performance, while APFLEF achieves a better balance between privacy protection and model utility. Figure 5 Further confirmation shows that APFLEF's attack surface area on CIFAR-10 and AGNews is 0.2863 and 0.5580, respectively, both significantly smaller than other methods, indicating that it has the lowest risk of privacy leakage.

[0256] (4) Component collaboration verification

[0257] Table 7 and Figure 6 The collaborative performance of APFLEF's core components was verified, achieving a near-optimal dynamic balance between communication efficiency, model accuracy, and privacy protection.

[0258] This ablation study aims to compare the performance of the proposed GCEF and ADP modules with the benchmark and fixed-noise schemes; therefore, a combined experiment of "GCEF and Fixed Privacy (FDP)" was not included. In the CIFAR-10 task, using the GCEF (Gradient Compression and Error Feedback) module alone significantly reduced the single-round training time from 39.06 seconds to 8.35 seconds while maintaining 81.30% model accuracy. Enabling the ADP (Adaptive Differential Privacy) module alone reduced the member inference attack accuracy from 75.80% to 62.92%, significantly enhancing privacy protection. The complete APFLEF framework achieved the best overall performance, maintaining 81.19% accuracy within a single-round training time of 8.48 seconds and further reducing the attack accuracy to 60.18%.

[0259] A consistent trend was observed in the AGNews text classification task: the GCEF module reduced training time by more than 58%, the ADP module reduced attack accuracy from 90.40% to 73.15%, and the complete framework achieved 86.04% accuracy and 72.20% attack accuracy in a single training round of 120.16 seconds.

[0260] Table 7 Comparison of ablation experimental performance between core modules of the APFLEF framework and traditional modules

[0261]

[0262] Note: "-" in the table indicates that the corresponding module was not enabled in this experimental configuration (i.e., no compression or no privacy protection). All configurations completed the testing of all metrics.

[0263] It is worth noting that while fixed-difference privacy schemes can provide some privacy protection, the fixed noise they introduce can severely damage model performance, resulting in an accuracy drop of over 20%. Figure 6 This indicates that as training progresses, the signal-to-noise ratio of fixed noise and decaying gradient continuously deteriorates, ultimately hindering model convergence. This finding mechanistically demonstrates the necessity of the adaptive noise mechanism. Table 7 and Figure 6 The ablation experiments clearly demonstrate that, under the same compression conditions (or no compression conditions), the ADP mechanism of this invention can achieve a privacy-utility tradeoff that is far superior to fixed-noise (Fixed-DP), achieving a significant technical effect of synergistic improvement in communication, accuracy, and privacy. The experimental data directly verifies the core value of the "gradient-sensitive" design—compared to fixed-noise, the ADP mechanism, through its gradient norm sensing capability, retains higher model utility for important gradients and assigns stronger privacy protection to minor gradients, thus plotting a better privacy-utility boundary curve overall, which strongly verifies the technical effectiveness of the core innovation of this invention.

[0264] (5) Engineering applicability and robustness verification

[0265] Figure 7 , Figure 8 and Figure 9 APFLEF's good engineering applicability and robustness have been verified:

[0266] Gradient compression ratio impact analysis: Comparing the gradient accumulation strategy of GCEF with the direct discard strategy of Top-k, Figure 7 The data shows that at a compression ratio of 0.9, both methods can reduce the bandwidth from 584.07 Mb to 59.38 Mb; however, Figure 9 The results show that at a compression ratio of 0.99, GCEF still maintains an accuracy of 76.17%, while the accuracy of the Top-k method drops sharply to 60.08%. Further analysis shows that as the compression ratio increases from 0 to 0.99, the accuracy of GCEF decreases relatively gradually, while Top-k suffers accuracy losses of 6.19% and 21.8% respectively under the same conditions, fully verifying the robustness of the GCEF module to aggressive compression.

[0267] Noise scaling constant c influence analysis: The system verified the influence of different noise intensities on the privacy-utility tradeoff. Table 8 shows that when the noise scaling constant c is in the range of 0.01-0.1, the optimal balance between privacy protection and model utility can be achieved.

[0268] Table 8. Quantitative Analysis of the Impact of Noise Scaling Constant c on the Privacy-Utility Trade-off

[0269]

[0270] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, and for those of ordinary skill in the art, various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. Therefore, the present invention is not limited to the specific details without departing from the general concept defined by the claims and their equivalents.

Claims

1. A secure and efficient federated learning method based on error feedback and norm awareness, characterized in that, The method is used for image classification or natural language processing tasks, and includes the following steps: Step S1: Server initializes global model parameters Configure hyperparameters; set N to the total number of participating clients. For the first Each client, each participating client Collect raw sensor data and convert it into local training data At the same time, initialize the local cumulative error vectors respectively. and parameter delay counter , where p is used to identify that this counter is for the parameter, that is, to record the number of delay rounds related to the parameter; Step S2, each client From its local training data Sample one batch and calculate the global model parameters for round t. Local stochastic gradients ; Step S3, for The gradient compression and privacy-preserving DCPP process is performed, which includes sequentially performing gradient compression and error feedback GCEF and an adaptive differential privacy mechanism, and is implemented through the following steps S4 to S10: Step S4: Calculate the cumulative gradient for the current round. ,in The accumulated error saved in the previous round is stored, along with all gradient information of client i up to round t-1 that has not been transmitted. Step S5, Calculation Find the absolute value of all values, sort them in descending order, and take the first few values ​​after sorting. The minimum value corresponding to the gradient of the ratio is used as the threshold α, where s is the gradient compression ratio; Step S6: For each dimension of the gradient vector Generate binary mask ,like or Then set the selection mask. Set to 1 if it is not 1, otherwise set to 0; where, This represents the maximum number of delay rounds. Step S7: Based on the obtained mask Calculate the compressed gradient = ,in This represents element-wise multiplication; Step S8: Record the gradient information that was not transmitted in this round for use in the next round of compensation. And update the parameter delay counter, the calculation formula is: = +1 (1- ); Step S9: Based on a preset privacy budget gradient Sensitivity and noise scaling constant According to local stochastic gradient norm Dynamically calculate the adaptive noise intensity to be added in this round. The calculation formula is as follows: This adaptive differential privacy mechanism, in conjunction with the gradient compression and error feedback processes defined in steps S4 to S8, jointly ensures that the entire federated learning training process meets the requirements. Differential privacy requirements; Step S10: Sample noise from the Gaussian distribution and add it to the compressed gradient, i.e. ,in ; Step S11: The client will process the gradient. Send to the server; Step S12: The server collects all clients' [data / information / data]. And calculate the global gradient: ; Step S13: The server will generate the global gradient. Gradient compression and privacy protection are performed to obtain the processed global gradient. and the processed global gradient Broadcast to all clients; Step S14: After receiving the global update, each client updates its local model according to the gradient descent rule: ; Step S15: Repeat steps S2 to S14 until the preset total number of training rounds is reached. Or the model converges.

2. The method according to claim 1, characterized in that, In step S1, the local training data The dataset includes at least one of image classification data and text classification data; wherein the image classification data includes handwritten digit image datasets, natural scene object image datasets, or fine-grained image datasets; the text classification data includes news text datasets, movie review text datasets, or multilingual text datasets; and the local training data is divided among the clients in a non-independent and identically distributed manner. The hyperparameters include: the global model parameters in round t. This represents the global model state issued by the server at the start of the t-th round of communication; the client index i represents the i-th client participating in federated learning; and the total number of communication rounds. This indicates the total number of iterations and learning rate in the predefined federated learning training. This indicates the step size of the model update and the gradient compression ratio. Privacy Budget Differential privacy probability parameters Noise scaling constant Maximum number of delay rounds and the gradient clipping threshold C; wherein, the gradient compression ratio The value range is 80%-99%, and the privacy budget... The value range is 0.1-8.0, and the differential privacy probability parameter... The noise scaling constant is a fixed minimum value much less than 1, typically set to be less than the reciprocal of the size of the client's local dataset. The value range is 0.01-2.0, and the maximum number of delay rounds... The value range is 3-10, and the gradient clipping threshold C is used to limit the gradient. Norm, and used as the gradient in differential privacy noise addition. Sensitivity The value of .

3. The method according to claim 1, characterized in that, In step S6, the binary mask generation rule combines importance filtering and delayed forced transmission mechanisms to ensure that any gradient component is at most [a certain value]. It must be transferred once within the wheel.

4. The method according to claim 1, characterized in that, In step S8, the update rule for the parameter delay counter is as follows: for an unselected gradient dimension, the corresponding counter is incremented by 1; for a selected gradient dimension, the counter is reset to 0.

5. The method according to claim 1, characterized in that, In step S13, when downlink bandwidth is tight and it is necessary to further reduce the amount of transmitted data or when privacy protection needs to be enhanced in the downlink direction, the server will process the global gradient before performing the broadcast. The data is then re-entered into the gradient compression and privacy protection module for further processing to reduce downlink communication overhead or enhance privacy protection in the downlink direction.

6. The method according to claim 1, characterized in that, The method can be applied to image classification or natural language processing tasks. For image classification tasks, the AlexNet model is trained on the MNIST, CIFAR-10, or CIFAR-100 datasets. For natural language processing tasks, the Transformer model is trained on the AGNews, IMDb, or SWEDISH datasets.

7. A terminal device, characterized in that, include: One or more processors; The memory is communicatively connected to the processor; A communication unit connected to the processor; The memory stores computer programs; When the computer program is executed by the processor, the terminal device is configured as a client or server to implement the secure and efficient federated learning method based on error feedback and norm awareness as described in any one of claims 1 to 6, wherein: When configured as a client, the processor, the corresponding part of the memory, and the communication unit work together as a client processing unit, a client storage unit, and a client communication unit, respectively, to perform local gradient calculation, gradient compression and privacy protection processing based on error feedback and norm awareness, and gradient uploading and update reception with the server. When configured as a server, the processor, the corresponding portion of the memory, and the communication unit work together as a server processing unit, a server storage unit, and a server communication unit, respectively, to perform gradient aggregation, global gradient compression and privacy protection processing, and broadcast global updates to clients.

8. The terminal device according to claim 7, characterized in that, When configured as a client: (1) The client storage unit is used to store the local training dataset. Local model parameters Cumulative error vector and gradient delay counter ; (2) The client processing unit is used for: Perform step S2, from its local training data Mid-sampling and computation of local stochastic gradients ; Perform steps S3 to S10 to... Perform gradient compression and privacy protection operations, including: calculating the cumulative gradient. Generate mask And compress and update the accumulated error. With delay counter Adaptive noise is added based on the gradient norm to obtain the protected gradient. ; Execute step S14, based on the received global update Update local model parameters ; (3) The client communication unit is used for: Execute step S11 to process the gradient. Send to the server; Receive global updates from the server. or original gradient .

9. The terminal device according to claim 7, characterized in that, When configured as a server: (1) The server storage unit is used to store global model parameters. ; (2) The server processing unit is used for: Perform step S12 to collect gradients from each client. And calculate the global gradient ; Execute step S13, for Gradient compression and privacy protection are performed to obtain the processed global gradient. ; (3) The server communication unit is used for: Receive gradients uploaded by each client ; Execute step S13 to process the global gradient. Broadcast to all clients.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the secure and efficient federated learning method based on error feedback and norm awareness as described in any one of claims 1 to 6; the computer program is configured to, when executed by the processor, perform an iterative training process in the order of steps S2 to S15, including local gradient calculation, gradient compression and error feedback, adaptive differential privacy noise addition, gradient uploading, server-side gradient aggregation and processing, global update reception, and local model update.

Images