A federated learning adaptive weighted aggregation and update compression method

Abstract

The application discloses a federated learning adaptive weighted aggregation and update compression method, and belongs to the technical field of artificial intelligence and distributed machine learning. The method is cooperatively executed by a server end and multiple client ends, under the condition that the original data of the client ends is not uploaded, the client ends complete training based on local data, loss fluctuation, gradient fluctuation and parameter change rate are calculated to form a training stability score, and a dynamic aggregation weight is determined in combination with an update direction similarity. The server end performs effective weight normalization aggregation on sparse updates, keeps the global parameters of the last round without uploading coordinates, and adjusts the retention proportion according to the precision, loss and communication overhead feedback. The method can alleviate the client drift caused by Non-IID data, reduce the model update transmission amount, and improve the convergence speed, communication efficiency and deployment stability of federated learning.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, distributed machine learning, and privacy computing, specifically to a federated learning adaptive weighted aggregation and update compression method. Background Technology

[0002] Federated learning is a distributed machine learning technique for multi-participant, multi-data source, and multi-device environments. Its basic idea is to distribute a global model from the server without directly aggregating the raw data. Each client uses its local data to train the model and uploads its parameters or updates. The server then aggregates the updates from multiple clients to obtain a model with global generalization capabilities. Compared to centralized training, federated learning reduces the privacy and compliance risks associated with the flow of raw data across institutions, regions, or devices, and helps break down data silos.

[0003] Traditional federated averaging algorithms typically weight local model parameters or model updates based on the number of local samples on the client side. This method is simple in structure and easy to implement, but it assumes that clients with larger sample sizes contribute more, and it cannot adequately identify whether the client training process is stable, whether the update direction is consistent with the global optimization direction, or whether the local model is biased due to data heterogeneity.

[0004] On the other hand, federated learning requires repeated uploading and downloading of high-dimensional neural network parameters in multiple rounds of communication. As the parameter size of convolutional neural networks, residual networks, and visual classification models increases, uploading local model updates consumes a significant amount of communication bandwidth. In mobile terminals, edge gateways, vehicle-mounted devices, and industrial field networks, communication overhead often becomes a major bottleneck limiting the deployment of federated learning. Methods such as fixed-ratio Top-K sparsity, quantization, or gradient compression can reduce communication costs, but they often ignore the differences in importance between different parameters, different layers, and different training stages.

[0005] In existing solutions, client-side weight aggregation and compression strategies are typically designed separately: the aggregation phase focuses on "which client contributions are more credible," while the compression phase focuses on "which parameters need to be uploaded." The lack of a coordination mechanism between the two leads to client-side training stability not effectively guiding compression intensity, and the resulting sparse updates after compression failing to adequately match dynamic weight aggregation. Therefore, a federated learning method is needed that can simultaneously consider client-level training stability, parameter-level importance, layer-level differences, and training phase feedback. Summary of the Invention

[0006] The purpose of this invention is to provide an adaptive weighted aggregation and update compression method for federated learning, which solves the technical problems of existing federated learning in Non-IID data environments, such as the difficulty in accurately evaluating client contributions, the impact of local model drift on global convergence, and the excessive consumption of communication resources caused by repeated uploading of high-dimensional model updates.

[0007] To address the aforementioned issues, this invention constructs an integrated technical solution encompassing "training stability assessment—dynamic weighted aggregation—importance-driven compression—feedback adjustment," executed collaboratively by a server and multiple clients. The client side is responsible for local training, stability statistics, updated feature extraction, parameter importance scoring, and sparse uploading; the server side is responsible for model distribution, client selection, dynamic weight calculation, sparse update reconstruction, global model evaluation, and compression ratio adjustment.

[0008] In this scheme, the original training samples are always stored locally on the client side, and the server only receives sparse model updates, mask indexes, and necessary metadata for aggregation and tuning. This design creates a closed loop between aggregation weights, upload parameter selection, and communication budget control, rather than treating aggregation and compression strategies in isolation.

[0009] For ease of explanation, throughout the text Indicates the client number. Represents the coordinates of the model parameters. Indicates the model layer number. Indicates the round of federal communications; superscript Indicates the first Round corresponding variable, subscript , , These represent the client, parameter coordinates, and network layer dimension, respectively. Unless otherwise specified, the vector norm is the L2 norm. To avoid dividing positive numbers by zero.

[0010] Specifically, let the client set be... , No. The set of clients participating in the training round is The server-side global model parameters are Client The local dataset is The number of local samples is The server sends out [something] to the participating clients in each round. and reference direction The client minimizes experience risks locally:

[0011]

[0012] Client completed After one local training round, And form a model update Parameter change rate Used to measure the degree to which local training deviates from the global model.

[0013] in, For the input sample, For input samples The corresponding tags This represents the number of local samples on the client side. For loss function, For model mapping function, For the number of local training rounds, For the rate of change of the parameter, To prevent positive numbers with a denominator of zero.

[0014] To evaluate the reliability of the current client update, this invention is based on a sliding window. Statistical loss sequence Gradient norm sequence and parameter change rate We obtain the loss stability, gradient stability, and parameter change stability, respectively:

[0015]

[0016] For any sub-indicator In the participating client collection Normalization is performed internally:

[0017]

[0018] The overall training stability score is:

[0019]

[0020] in, This represents the operation of calculating the variance of the sequences within the sliding window W.

[0021] For the client In the sliding window The loss sequence within, For gradient norm sequences, This is the gradient variance scaling factor; , , These represent the loss, gradient, and parameter stability sub-indices before normalization, respectively. Represents the normalized i-th Sub-indicators; , , The weights of the three sub-indicators are non-negative and satisfy the following conditions: A higher stability score indicates that the client has less loss fluctuation, more stable gradient changes, and lower parameter deviation in recent training; a lower stability score indicates that the client may be more affected by local class skew, sample noise, or insufficient training.

[0022] The above sub-indicators are in the participating client set After internal normalization, weighted by non-negative weights , and The training stability score is obtained by fusion. The sum of the three weights is 1. In addition to training process stability, this invention also utilizes update direction similarity to evaluate the consistency between client updates and the global optimization trend. (Client complete update) with reference direction The cosine similarity is expressed as:

[0023]

[0024] The server will Normalization And determine the stage mixing coefficient based on the training progress. Then, the client-side aggregate weight is calculated:

[0025]

[0026] In the early stages of training, the influence of stability score can be appropriately increased to reduce the disturbance of the global model by noisy or drifting clients; in the middle and later stages of training, the influence of update direction consistency can be increased to make the aggregation closer to the global optimization direction.

[0027] To reduce uplink traffic, this invention does not require the client to upload the complete model update; instead, it constructs an importance score for each parameter coordinate. For parameter coordinates... The gradient magnitude component, the diagonal Fisher information component, and the historical update component are respectively:

[0028]

[0029] right , and After normalization, we get , and The overall importance score is:

[0030]

[0031] in, and Represents the coefficient of the exponential moving average. The weights are non-negative fusion weights, and the sum of the three is 1. The gradient magnitude reflects the contribution of the current round of optimization, the diagonal Fisher information reflects the sensitivity of the parameters to changes in loss, and the historical update component reflects the importance of continuous parameter changes across rounds.

[0032] The client sets the base retention ratio according to the model layer type. And combined with stability regulators Obtain the actual retention ratio .in, and These represent the lower and upper thresholds for stability, respectively. and These represent the lower and upper limits of the retention ratio, respectively. This represents the interval cutoff function. Then, the layer... intrinsic parameter set according to Select Top-K coordinates in descending order, by Determine the number of parameters to retain and generate a binary mask. And obtain sparse updates. .

[0033] After receiving sparse updates from multiple clients, the server does not treat unuploaded coordinates as zero updates, but instead updates each parameter coordinate. Calculate the effective upload weight sum:

[0034]

[0035] when At that time, the server only uses the client updates that actually upload the coordinates for normalized aggregation; when At this time, the coordinates retain the global parameters from the previous round. This avoids diluting effective updates for locations that have not been uploaded and ensures that the global model after each round of aggregation remains a complete parametric model.

[0036] The server also considers the global precision before and after aggregation. , Global loss , Communication overhead in this round and communications budget Generate feedback quantity ,

[0037]

[0038] server side Update the base retention ratio for the next round of hierarchical updates. Increase the retention ratio when model accuracy decreases or loss increases, and decrease the retention ratio when communication overhead exceeds the budget.

[0039] From the perspective of communication complexity, if the average parameter retention ratio is The total number of parameters in the model is The bit width of a single floating-point parameter is Ignoring mask index overhead for now, the uplink parameter transmission volume can be reduced by... Reduced to Even after incorporating the overhead of a 1-bit mask or index, budget constraints can still be applied using the communication overhead formula.

[0040] When common analytical conditions such as smoothness and bounded variance are met, dynamic weights can be understood as improving the effective contribution of stable clients and reducing drift noise terms. Their error recursion can be expressed as:

[0041]

[0042] In this formula, This represents the target optimal parameters or reference optimal parameters. Indicates the learning rate. The parameter represents a strong convexity or a locally approximate strong convexity. Indicates the first The effective lower bound of the aggregate weight of trusted clients in the round This represents random gradient noise. This represents the drift term caused by data heterogeneity and unstable clients. This expression illustrates that increasing the effective weights of trusted clients and suppressing the drift term is beneficial for improving convergence stability, and does not limit this invention to a specific model or loss function.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] (1) This invention improves the reliability of client contribution evaluation and global aggregation in heterogeneous data environments. By comprehensively considering loss fluctuation, gradient fluctuation, parameter change rate and update direction similarity, and adjusting the evaluation focus according to the training progress, it can suppress noisy updates and local drift, and improve the aggregation reliability and convergence stability in Non-IID scenarios.

[0045] (2) This invention achieves adaptive update compression based on parameter importance and model layer differences. By fusing gradient magnitude, diagonal Fisher information and historical update magnitude, and performing intra-layer normalized sorting; at the same time, different retention ratios are set for different network layers, and the upload scale is adjusted in combination with client stability, so as to reduce communication volume, retain key updates and adapt to different model structures.

[0046] (3) This invention constructs a collaborative closed loop of weighted aggregation, update compression, and feedback regulation. Training stability is used to adjust both the aggregation weights and the actual retention ratio. The server updates the basic retention ratio based on accuracy, loss, and communication overhead. For inconsistent uploaded coordinates, only the actually uploaded updates are aggregated, while the unuploaded coordinates retain their original values, thereby avoiding update dilution and aggregation bias caused by zero padding.

[0047] (4) This invention balances communication efficiency, data protection, and deployment stability. The client only uploads sparse updates, mask indexes, and necessary metadata, while the original samples and labels are kept locally. Rollback and exception handling mechanisms are set up for situations such as insufficient historical data, invalid reference directions, and abnormal weights or uploaded information, so that the system can still operate stably when the client participates dynamically, bandwidth fluctuates, and data distribution changes. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the structure of the federated learning system of the present invention;

[0049] Figure 2 This is a flowchart illustrating the overall process of the federated learning training method of this invention.

[0050] Figure 3 The following are comparison charts of the total communication costs of each algorithm: (a) is a comparison chart of the total communication costs of each algorithm under the CIFAR-10 dataset, and (b) is a comparison chart of the total communication costs of each algorithm under the CIFAR-100 dataset.

[0051] Figure 4 The graphs show the changes in test accuracy with the number of communication rounds. (a) shows the changes in test accuracy of each algorithm with the number of communication rounds under the CIFAR-10 dataset, and (b) shows the changes in test accuracy of each algorithm with the number of communication rounds under the CIFAR-100 dataset.

[0052] Figure 5 The plots show the trade-off between accuracy and communication cost, with (a) showing the Pareto front of accuracy versus communication cost on the CIFAR-10 dataset and (b) showing the Pareto front of accuracy versus communication cost on the CIFAR-100 dataset.

[0053] Figure 6 To compare the convergence trend and fluctuation of the training loss, (a) shows the change of training loss with the number of communication rounds under the CIFAR-10 dataset, and (b) shows the change of training loss with the number of communication rounds under the CIFAR-100 dataset. Detailed Implementation

[0054] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] like Figure 1 As shown, this embodiment uses a server and A federated learning system consisting of several clients. The server stores the global model, training configuration, reference update direction, historical evaluation metrics, and the base retention ratio of each layer; the clients store local data, local model copies, loss sequences, gradient norm sequences, parameter change history, Fisher approximation statistics, and historical update statistics. Clients do not need to directly exchange data.

[0056] In actual operation, the server maintains the status of the global model, client participation records, aggregated weights, communication volume, and retention ratio; the client maintains the local training cache, stability statistics window, and parameter importance records. Both parties interact through delivery messages and update packages with communication rounds and model version identifiers to ensure that uploaded updates correspond to the current global model, and the original training data is always retained locally on the client.

[0057] like Figure 2 As shown, this embodiment uses communication rounds. The process is executed along the main thread, with each round including server preparation, model deployment, local training, stability assessment, importance compression, sparse uploading, dynamic aggregation, feedback adjustment, and convergence judgment.

[0058] Step S1: The server prepares the global model. The server reads the... Round global model parameters And set the maximum number of communication rounds. Local training rounds Learning rate, client participation rate, sliding window Fusion weight, stability threshold, and basic retention ratio of hierarchical levels. Upper and lower limits of retention ratio and And the convergence threshold. If Server-side initialization of global model parameters The update direction will be referenced. Set to an all-zero vector, and use a preset neutral value of 0 when calculating the update direction similarity on the client side; if The server calculates the reference update direction based on the global model parameters of two adjacent rounds: The server will update the reference direction. With the Round global model parameters It will be sent to the participating clients together.

[0059] During parameter initialization, the server can also identify convolutional layers, fully connected layers, normalization layers, and classifier layers based on the model structure, and write default retention ratios for different layers. For normalization parameters, bias parameters, or output classifier parameters that are not suitable for compression, a higher retention ratio can be set to reduce the perturbation of the model output distribution by compression.

[0060] Step S2: The server selects a client and distributes the model. The server can select clients based on random sampling, availability filtering, or a preset participation ratio. Determine the participating set from among the clients and send to clients within the collection. , This includes the local training configuration, stability calculation parameters, and compression / retention ratio configuration for this round. The distributed content does not include raw data, labels, or local intermediate samples from other clients.

[0061] After receiving the model, the client first verifies the version information and confirms its local training conditions. Clients that cannot complete training in this round will not generate updates, and the server will remove clients that failed to upload or pass verification from the server. Removed from the list and based on the updated version. Perform subsequent calculations and aggregations.

[0062] Step S3: The client performs local training. Will Load the local model and the local dataset. Execute in mini-batch training mode There are several local training epochs. In each training batch, the client calculates the predicted output, loss value, and gradient, updates the local model parameters, and records the batch loss and gradient norm. After each local training epoch, the client summarizes the average loss and average gradient norm. After training, the client obtains... ,calculate and .

[0063] In one specific implementation, the client can... The average of the mini-batch losses within each local training epoch is used to obtain the loss for that epoch. The gradient norm for this round is obtained by averaging the gradient norms of each mini-batch. After training is completed, the most recent indivual and Write it into the local sliding window as input for the next stability calculation. This indicates the local training round index. Indicates the first The number of mini-batches contained in each local training epoch. This indicates a mini-batch index. This indicates the corresponding small batch loss. Indicates the model parameters Find the gradient.

[0064]

[0065] Local training can employ stochastic gradient descent or its variants; the specific optimizer is not a limitation of this invention. If non-numerical loss, gradient anomalies, or incomplete training occur during the training process, the client stops generating the current round of updates and returns an abnormal state to prevent invalid results from entering subsequent calculations.

[0066] Step S4: The client calculates training stability and update direction similarity. The client bases the results on the nearest... Wheel or most recently The loss sequence, gradient norm sequence, and parameter change rate at each statistical point are used to calculate three stability sub-indices, which are then fused together to obtain... If the historical window is insufficient, a neutral initial value will be used or calculations will be performed only based on currently available statistics to avoid abnormal amplification of the client during the first round or cold start. The client will also undergo a complete update. Flattened into a vector, relative to the reference direction Calculate cosine similarity ;like If the vector is zero or missing, neutral similarity is used.

[0067] In this step, the loss sequence can be either the local epoch average loss or the batch loss moving average, the gradient norm can be the mean of the gradient 2 norm of each batch, and the parameter change rate can be calculated uniformly after local training is completed. By using the three dimensions of loss, gradient, and parameter change simultaneously, the reliability of the client can be avoided by relying solely on a single loss value.

[0068] For example, when At that time, the client retains the five most recent statistical points and calculates the loss variance for each. and gradient norm variance Then, the volatility is converted into a stability sub-indicator through index mapping. The server or client participates in the set... The same sub-indicator is normalized internally to make the scores of different clients comparable.

[0069] After the stability score and directional similarity are calculated, the client restricts them to a preset valid range and combines them with the local sample size and model version number to form client-level metadata. This metadata only reflects the current training state and does not include original samples, labels, or content that can be directly reconstructed from the original data.

[0070] In step S5, the client calculates parameter importance and generates a hierarchical Top-K mask. The client normalizes the gradient magnitudes accumulated during local training, the diagonal Fisher information EMA, and the historical update EMA, and fuses them to obtain the importance score for each parameter coordinate. For convolutional layers, fully connected layers, and classifier layers, the client employs different base retention ratios; convolutional layers retain the key parameters required for feature extraction, fully connected layers can use stronger compression, and classifier layers use a higher retention ratio or no compression. The client then adjusts the retention ratio accordingly. Select the Top-K coordinates at each layer to form a mask. and will Convert to sparse update .

[0071] When calculating parameter importance, the gradient magnitude, Fisher approximation, and historical update magnitude within the same layer can be normalized within the layer before weighted fusion. Intra-layer normalization can prevent layers with large parameters or large gradient magnitudes from dominating for a long time, allowing each layer to select upload coordinates according to its own importance ranking.

[0072]

[0073]

[0074] After determining the actual retention ratio, the client proceeds according to... Computation layer The number of items retained, and ranked from highest to lowest importance score at that level. One coordinate is set to 1, and the rest are set to 0. If a layer is configured to be uncompressed, then the mask for that layer is all 1s.

[0075] Step S6: The client uploads sparse updates and metadata. The data uploaded by the client includes non-zero update values, corresponding coordinates or binary masks, and training stability scores. Update directional similarity Local sample size The system includes training loss summary, gradient norm summary, and traffic statistics. The client does not upload original images, original labels, complete training samples, or other data not visible to the client.

[0076] The metadata can be represented using a key-value structure, for example, using parameter tensor names to correspond to masks and non-zero update values, and client IDs to correspond to stability scores, sample sizes, and communication volumes. The server can then perform sparse coordinate alignment, effective weight calculation, and communication budget statistics based on this metadata without needing to recover local client data.

[0077] The client encapsulates sparse updates, coordinate information, and training metadata into a single update package. Upon receiving this package, the server verifies the model version, tensor shape, and index range, and does not aggregate updates that are incomplete or do not match the current model.

[0078] Step S7: The server calculates the dynamic aggregation weights. After receiving content uploaded by each client, the server first checks whether the tensor name, tensor shape, and mask length are consistent with the global model; for missing or abnormal metadata, a neutral value can be used, the weight reduced, or the client removed from the update. Subsequently, the server calculates the similarity... exist Internal normalization Determined based on training progress and calculate and If the scores of all clients are too similar or the sum of their weights is close to zero, then uniform weighting or sample size weighting is used as a backoff strategy.

[0079] Stage mixing coefficient The training progress can be segmented: in the early stage of training, increase the proportion of stability score; in the middle stage, maintain a balance between stability and similarity; and in the later stage, appropriately increase the proportion of update direction similarity.

[0080] Step S8: The server performs coordinate-level reconstruction and aggregation. The server calculates the effective upload weights for each model parameter tensor coordinate by coordinate. .for The server uses the coordinates uploaded by the client to perform effective weight normalization through sparse updates, and then adds the result to... Above; for The coordinates are maintained by the server. Unchanged. After aggregation, the server receives the complete result. And based on this, update the reference direction for the next round.

[0081] Since the Top-K coordinates may differ across clients, the server aligns the tensor names and coordinate indices using a mask before aggregation. For the same coordinate, only the client that actually transmitted that coordinate participates in the denominator. The calculation; clients that do not upload the coordinates neither contribute positive updates nor dilute the effective increments of other clients as zero updates.

[0082] For example, for a certain parameter coordinate If the client and Upload the coordinates to the client. If the coordinates are not uploaded, the server will only use... and form And use the sparse update calculations uploaded by both to calculate ; This coordinate is neither included in the numerator nor the denominator. This method differs from the approach of padding missing positions with zeros and then averaging.

[0083] Step S9: The server performs feedback regulation and convergence judgment. The server calculates the global precision and global loss on the validation set, calculates the communication overhead of this round, and determines the result based on the feedback amount. Update the next round Training ends if the maximum number of communication rounds is reached, the accuracy change is less than a threshold for multiple consecutive rounds, the loss change is less than a threshold for multiple consecutive rounds, or the target accuracy is reached; otherwise, let... Then return to step S1.

[0084] After training, the server outputs the final global model, accuracy and loss curves for each round, communication statistics, average stability score, average compression retention ratio, and records of aggregate weight changes for each client. These outputs can be used for pre-deployment evaluation and for subsequent adjustments to stability thresholds, retention ratio limits, and communication budget.

[0085] In the above steps, Used to avoid division by zero; To prevent the retention ratio from exceeding the limit, when a client's local training results in non-numerical loss, gradient explosion, or incomplete uploaded packets, the server can reject the client's update or use a preset low-weight processing method. This ensures that the system can still complete federated training despite changes in the number of clients, differences in data distribution, and changes in communication budget.

[0086] This embodiment illustrates the verification method of the present invention in federated learning for image classification. Experiments can use the CIFAR-10 and CIFAR-100 datasets, and the Non-IID client data partitioning is constructed using the Dirichlet distribution; Dirichlet parameters... Values ​​of 0.1, 0.5, and 1.0 can be used to simulate strong, medium, and weak data heterogeneous scenarios, respectively. The smaller the value, the more unbalanced the class distribution among clients, and the more applicable the present invention is to client drift suppression and communication-constrained training.

[0087] The experimental configuration includes parameters such as the total number of clients, participation ratio per round, number of local training rounds, batch size, learning rate, maximum number of communication rounds, base retention ratio, and communication budget. A reproducible example is: 100 clients, participation ratio per round of 0.1, 5 local training epochs, batch size of 64, initial learning rate of 0.01, and 100 to 200 training rounds; multiple random seeds can be used to run the experiment repeatedly, and the mean and standard deviation can be calculated. Each client uses only local data for training and records metadata such as loss sequence, gradient norm, parameter change rate, sample size, and communication volume, without uploading original images, labels, or intermediate samples.

[0088] To ensure consistent comparison conditions, except for the aggregation or compression mechanism to be verified, all experimental groups used the same data partitioning, model structure, client sampling method, local training parameters, and communication rounds; the communication volume was calculated based on the actual transmitted parameter values, coordinate information, and the amount of data sent by the server-side model.

[0089] The experimental setup included FedAvg, FedProx, SCAFFOLD, FedNova, Top-K, DGC, stability aggregation (SAIC-Agg), importance compression (SAIC-Comp), and the synergistic scheme (SAIC-FL) as controls. FedAvg served as the traditional sample-weighted baseline; FedProx, SCAFFOLD, and FedNova were used to compare the offset suppression effect under Non-IID conditions; Top-K and DGC were used to compare the sparsity compression effect; stability aggregation (SAIC-Agg), importance compression (SAIC-Comp), and the synergistic scheme (SAIC-FL) corresponded to the single-module validation and complete scheme validation of this invention, respectively. The division between the main experiment and the ablation experiment is shown in Table 1 below.

[0090] Table 1. Experimental comparison scheme and verification objectives of the inventive method

[0091]

[0092] In each round of testing, the server side follows... Figure 2 Extract participating clients and distribute the global model; after the clients complete local training, calculate the stability score, update direction similarity and parameter importance score according to steps S3 to S6, generate a hierarchical Top-K mask and upload it for sparse update; the server calculates the dynamic weights according to steps S7 to S9, performs coordinate-level effective weight normalization aggregation, and records the accuracy, loss value, communication cost and compression ratio on the test set.

[0093] Experimental results passed Figures 3 to 6 As shown in Table 2 below. Figure 3Used to compare the total communication cost of each algorithm Figure 4 Used to demonstrate how test accuracy changes with the number of communication rounds. Figure 5 This is used to observe the trade-off between accuracy and communication cost. Figure 6 Used to compare the convergence trend and volatility of training loss. Figures 3 to 6 Table 2 and the results of dynamic aggregation (SAIC-Agg), importance compression (SAIC-Comp), and the combined effect of the two (SAIC-FL) can be verified respectively.

[0094] Table 2 Summary of Algorithm Experiment Results on CIFAR-10 and CIFAR-100 Datasets

[0095]

[0096] The values ​​in Table 2 are example results from the replication experiment in this embodiment. Actual results will vary depending on the model structure, data heterogeneity, random seed, and hyperparameter settings. Figures 3 to 6 It is evident that stability aggregation is primarily used to improve the aggregation reliability and training curve stability in Non-IID scenarios; importance compression and collaborative schemes are mainly used to reduce communication costs and achieve a trade-off between accuracy and communication volume under a given retention ratio and budget.

[0097] In actual deployment, the basic retention ratio, stability threshold, and communication budget can be adjusted according to the task focus: when the accuracy requirements are high, the retention ratio of key layers can be appropriately increased, and the retention ratio of redundant layers can be reduced in bandwidth-limited scenarios; when the client data heterogeneity is high, the influence of training stability on the aggregation weights can be increased; when the model tends to stabilize in the later stages of training, the upload ratio can be gradually reduced to reduce communication overhead.

[0098] In summary, this embodiment describes the present invention from the aspects of system structure, single-round training process, client-side computation, server-side aggregation, feedback control, and experimental verification, enabling those skilled in the art to implement the solution according to the above steps. All equivalent substitutions, improvements, or combinations made based on the technical concept of this invention should fall within the protection scope of this invention.

Claims

1. A federated learning adaptive weighted aggregation and update compression method, characterized in that, The method uses a federated learning system consisting of a server and multiple clients as the execution entity, and performs global model training without uploading the original data from the clients. It includes the following steps: Step 1: Before training begins, the server initializes the global model parameters. Set the training rounds, client participation ratio, stability window, and hierarchical base retention ratio. and the convergence threshold, and in the first Wheel Select the set of clients to participate in training from among the clients. ; Step 2, the server sends a collection to the client. Issued the first Round global model parameters and reference update direction Each client Based on local dataset Perform local training to obtain local model parameters Model update And training process metrics; Step 3: Each client calculates its training stability score based on the training process metrics. And extract local model update features that include at least the parameter change rate and update direction similarity; Step 4: Each client scores the parameters based on their importance. Generate a hierarchical Top-K mask Update the model Selective compression is performed to obtain sparse model updates. The sparse model update, mask index, and training metadata are uploaded to the server. The server verifies the upload results and removes clients that failed to upload or pass verification from the server. Remove from the middle; Step 5: The server calculates the training stability score. Dynamically calculate client-side aggregate weights and update features with local model The system then performs coordinate-level reconstruction and weighted aggregation on multiple sparse model updates to obtain the global model parameters for the next round. ; Step 6: The server generates feedback based on the global model accuracy, loss value, and communication overhead, and adjusts the basic retention ratio for the next layer. Repeat steps 1 to 6 until the preset convergence condition is met.

2. The federated learning adaptive weighted aggregation and update compression method according to claim 1, characterized in that, Local training in step 2 includes: client take over Then, in the local dataset Minimize the local empirical risk function The local experience risk function is: Client passed Each local training round was obtained And calculate the model update and parameter change rate: in, For the input sample, For input samples The corresponding tags This represents the number of local samples on the client side. For loss function, For model mapping function, For the number of local training rounds, For the rate of change of the parameter, To prevent positive numbers with a denominator of zero.

3. The federated learning adaptive weighted aggregation and update compression method according to claim 2, characterized in that, Training stability score in step 3 It is obtained by weighting loss stability, gradient stability, and parameter change stability, specifically including: For any sub-indicator In the participating client collection Normalization is performed internally: The overall training stability score is: in, This represents the operation of calculating the variance of the sequences within the sliding window W. For the client In the sliding window The loss sequence within, For gradient norm sequences, This is the gradient variance scaling factor; , , These represent the loss, gradient, and parameter stability sub-indices before normalization, respectively. Represents the normalized i-th Sub-indicators; , , The weights of the three sub-indicators are non-negative and satisfy the following conditions: , When historical data is insufficient within the sliding window W during the initial training phase, the corresponding stability sub-indicator is set to a neutral initial value, or it is normalized based on the client statistics of the current round of participants, in order to avoid abnormal weights due to missing historical data.

4. The federated learning adaptive weighted aggregation and update compression method according to claim 3, characterized in that, The local model update features in steps 3 and 5 include at least the local training loss sequence, gradient norm sequence, parameter change rate, local sample size, and update direction similarity; when At that time, the server calculates the first round based on the global model parameters of the two adjacent rounds. Reference update direction: Complete client-side model update Reference update direction Similarity of update directions between for: in, This means flattening the model parameter tensor into a vector. To prevent positive numbers with a denominator of zero; when the first round of training or reference direction is missing, Set to a neutral value.

5. The federated learning adaptive weighted aggregation and update compression method according to claim 4, characterized in that, Step 5, dynamically calculating the client-side aggregate weight, includes: the server side will... In the set Internal normalization Let the maximum number of communication rounds be... , No. Normalized training progress of the wheel: Stage mixing coefficient ,in, , and These represent the stability score weights in the early and late stages of training, respectively. The unnormalized weights are: The normalized aggregate weights are: in, Indicates the normalized training progress. Indicates the stage mixing coefficient. This indicates the similarity of the update direction after normalization; when the sum of the unnormalized weights is less than the preset threshold or the difference in client scores is too small, the server-side degradation adopts uniform weight or sample size weight.

6. The federated learning adaptive weighted aggregation and update compression method according to claim 5, characterized in that, Parameter importance scoring in step 4 It is obtained from the following calculations: right , and After normalization, we get , and The overall importance score is: in, When the initialization round is reached, let 0、 0, Indicates client In the Wheelset parameter coordinates The gradient is calculated. This indicates that the local model is updated in coordinates. The component at that location, Represents the gradient magnitude component. This represents the exponential moving average component of the diagonal Fisher information. The exponential moving average component representing the historical update magnitude. , and This represents the result after normalizing the three components within the same model layer. and Represents the coefficient of the exponential moving average. The weights are non-negative fusion weights and the sum of the three is 1; In step 4, the layered Top-K mask sets the base retention ratio according to the model layer type. Convolutional layers use a moderate retention ratio, fully connected layers use a low retention ratio, and classifier layers use a high retention ratio or no compression; for layers... The parameter set inside The client follows Sort in descending order and select the first Generate a mask using one parameter: Where TopK represents the operation of selecting the aforementioned number of parameter coordinates from the parameter set of layer l according to their importance scores from high to low. Indicates the number of layer parameters. Indicates client In the layer The actual retention rate Indicates the number of parameters to retain. This represents the floor function. Represents a binary mask. Represents the compressed coordinates Sparse model update at the location.

7. The federated learning adaptive weighted aggregation and update compression method according to claim 6, characterized in that, Training stability score Used for adjusting the compression retention ratio, set and These represent the lower and upper thresholds for stability, and the stability adjustment factor. for: Current round client In the layer The actual retention rate is: in, and Let represent the adjustment factors corresponding to low stability and high stability, respectively, and satisfy . ; and These represent the lower and upper limits of the retention ratio, respectively. This represents a function that truncates input values ​​to a given interval. Step 5, where the server performs coordinate-level reconstruction and weighted aggregation for the sparse model update, includes: for the first... Each parameter coordinate defines the effective upload weight: when At that time, the server updates the coordinates according to the following formula: when At that time, the server side maintains ;in, and Representing the global model before and after aggregation respectively. One parameter coordinate, This represents the sum of the effective aggregate weights for uploading this coordinate.

8. The federated learning adaptive weighted aggregation and update compression method according to claim 7, characterized in that, In step 6, the feedback quantity is based on the global accuracy before and after aggregation. , Global loss , Communication overhead in this round and communications budget generate: The basic retention ratio will be updated in the next round according to the following formula: in, Non-negative feedback weights Indicates the adjustment step size. This indicates the feedback rate; the retention rate is increased when accuracy decreases or loss increases, and decreased when communication overhead exceeds the budget.

9. The federated learning adaptive weighted aggregation and update compression method according to claim 8, characterized in that, Communication overhead includes uplink and downlink traffic; let the total number of model parameters be... The bit width of a single floating-point parameter is The mask bit width is , No. The number of participating clients in the round is Then the uplink traffic is not compressed. Compressed uplink traffic Uplink communication savings and downlink traffic They are respectively: Preset convergence criteria include reaching the maximum number of communication rounds. The change in global model accuracy across consecutive adjacent rounds is less than a threshold. The change in training loss in consecutive adjacent rounds is less than a threshold. Or achieve at least one of the target accuracy.

10. The federated learning adaptive weighted aggregation and update compression method according to claim 9, characterized in that, Each client encapsulates the sparse model update, mask index, and training metadata including at least the training stability score, update direction similarity, and local sample size into an update package with communication round and model version identifier and sends it to the server. After receiving the update package, the server verifies the model version, parameter tensor name, tensor shape, mask length, and coordinate index range. Only update packages that pass the verification are subjected to dynamic aggregation weight calculation, coordinate-level reconstruction, and weighted aggregation. Update packages that fail the verification are not aggregated.

Images