Federal distillation method based on adaptive bidirectional distillation and soft weighted polymerization

Abstract

The invention relates to the technical field of federated learning, and provides a federated distillation method based on adaptive bidirectional distillation and soft weighted aggregation, which comprises the following steps: a server issues a global model and a selection state identifier of a client to the client; the client constructs a local teacher-student network, dynamically adjusts distillation loss weights of a feature layer and a logic layer according to a received selection state identifier, executes adaptive bidirectional knowledge distillation training, dynamically adjusts an information retention rate to compress local model parameters, and uploads a compressed parameter packet and a training loss score to the server; the server calculates a soft aggregation weight by using a negative exponential function; and the server performs recovery and weighted summation on the local model parameters uploaded by the clients by using the soft aggregation weight to generate a new generation of global model. According to the method, the communication transmission overhead is remarkably reduced, the model convergence precision in a non-independent identically distributed data environment is improved, and meanwhile, the robustness of the system for resisting malicious node poisoning attacks is enhanced.

Description

Technical Field

[0001] This invention relates to the technical field of the intersection of artificial intelligence and distributed computing, and in particular to a federated distillation learning method based on adaptive bidirectional distillation and soft-weighted aggregation, which is a heterogeneous model collaborative training optimization and poisoning-resistant robust aggregation in federated learning. Background Technology

[0002] In the era of "Internet of Everything," where 5G communication, the Internet of Things (IoT), and edge computing technologies are deeply integrated, smart terminal devices are growing exponentially, continuously generating high-value data including images, voice, and behavioral logs. Traditional machine learning relies on a centralized training paradigm of "physical data aggregation," which not only puts unbearable bandwidth and storage pressure on backbone networks but also exposes users' personal privacy, corporate trade secrets, and national data assets to the risk of leakage through centralized storage. In 2016, Google proposed Federated Learning, which uses a distributed collaborative mechanism of "data stays still, model moves," allowing terminal devices to train models locally and only upload encrypted parameter updates, thereby breaking down data silos and significantly reducing the risk of privacy leaks.

[0003] Faced with the challenges of limited computing power and insufficient communication bandwidth on edge devices, federated knowledge distillation has emerged as a lightweight communication and optimization paradigm. It aims to reduce communication overhead and achieve personalized learning by exchanging model outputs or intermediate layer features instead of the massive full set of parameters. However, in practical applications, client data often exhibits severe non-independent and identically distributed characteristics, causing local models on different clients to converge in different directions during training, resulting in a serious "model drift" phenomenon. Most existing federated distillation schemes employ static distillation strategies, forcing client models to mimic the global teacher model with fixed weights regardless of their current performance. This lack of adaptability often backfires: for "weak clients" with poor data quality or in the early stages of training, excessively high distillation intensity can lead to the loss of local features or even training divergence; while for "advantageous clients" with unique data, excessively low weights limit their contribution to the global model.

[0004] Furthermore, model aggregation, as the core hub for coordinating knowledge from multiple parties in federated learning, directly determines the convergence accuracy and robustness of the global model. Traditional federated averaging algorithms primarily rely on simple weighting based on sample size, assuming all participants are honest and the data quality is balanced. However, in open federated networks, malicious participants may launch poisoning attacks, attempting to compromise the usability of the global model or implant backdoors by uploading carefully crafted anomalous parameters or label-flipping data. To defend against such attacks, existing techniques typically employ a threshold-based "hard truncation" strategy, directly discarding client updates with high loss values ​​or significant differences from the global model. While this method filters out some noise, it also crudely discards normal clients in the long-tail distribution (these clients often possess crucial knowledge for identifying difficult samples), leading to a significant decrease in the global model's generalization ability on minority class samples.

[0005] While the invention patent with publication number CN120596951A proposes a method for enhancing the collaborative task of federated small and large models for balanced modalities, by introducing a large model regulator and a local small model for bidirectional distillation, and utilizing server-side data augmentation to alleviate the problem of inconsistent data distribution, its technical solution still has significant limitations: First, this method focuses on balancing modal differences through data augmentation and regulator updates, but ignores the dynamic changes in the "selection state" of the client during training. It lacks a mechanism that can adaptively adjust the distillation weights of the logic layer and feature layer based on the node contribution (core or auxiliary state), which makes weak nodes prone to knowledge forgetting or feature collapse when forcibly aligned. Second, the solution still relies on traditional regulator updates or adapter synchronization in the aggregation stage, and does not introduce a negative exponential soft weighting strategy based on training loss scores, making the system extremely vulnerable to malicious poisoning attacks or low-quality data uploads. In addition, this method does not involve parameter compression technology for communication bottlenecks, and cannot simultaneously achieve high communication efficiency, anti-poisoning robustness, and personalized model accuracy in complex federated network environments. Summary of the Invention

[0006] To address the technical challenge of existing federated learning methods in balancing communication efficiency and model generalization performance in scenarios with heterogeneous data and systems, this invention proposes a federated distillation method based on adaptive bidirectional distillation and soft-weighted aggregation. By constructing a bidirectional knowledge distillation framework, it enables bidirectional knowledge transfer between the global model on the server and the heterogeneous models on the client. Simultaneously, an adaptive selection mechanism based on loss values ​​dynamically weights high-quality knowledge sources, reducing communication overhead while improving the convergence accuracy of heterogeneous models, effectively balancing knowledge transfer efficiency and model heterogeneity. This invention significantly improves model accuracy on non-independent and identically distributed (Non-IID) data while reducing dependence on communication bandwidth, achieving an effective balance between communication efficiency and model utility.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows: a federated distillation method based on adaptive two-way distillation and soft weighted polymerization, the steps of which are as follows:

[0008] S1: Federated learning begins; the server sends the global model and the client's selection status flag to the client.

[0009] S2: The client builds a local teacher-student network, dynamically adjusts the distillation loss weights of the feature layer and logic layer according to the received selection status identifier, performs adaptive bidirectional knowledge distillation training, and dynamically adjusts the information retention rate according to the training rounds to compress the local model parameters, and uploads the compressed parameter package and training loss score to the server.

[0010] S3: The server collects the training loss scores uploaded by each client and uses a negative exponential function to calculate the score-based soft aggregation weights.

[0011] S4: The server uses soft aggregation weights to restore and weight the local model parameters uploaded by each client, generating a new generation of global model;

[0012] S5: Repeat steps S1 to S4 until the given maximum number of communication rounds is reached.

[0013] Preferably, the selection status indicator is based on a preset client filtering ratio set by the server. The first to participate in the training Each client is marked as either core or auxiliary, and a selection status identifier is generated. ,in, This represents the selected core state. This indicates an unselected auxiliary state;

[0014] During the initial training rounds, the server initializes the selection status flags of all participating clients to a uniform value. ;

[0015] The server transmits the global model parameters after dynamic compression via singular value decomposition and simultaneously issues a selection state identifier. For clients in the core state, the global model provides strong guidance through logic layer output and intermediate feature information. For clients in the auxiliary state, the guidance intensity is reduced through an attenuation coefficient.

[0016] Preferably, the method for the client to construct a local teacher-student network is as follows: the client obtains the global model parameters issued by the server and uses the global model parameters as the local teacher network, while using the client's original or initialized local model parameters as the student network; during the training process, a bidirectional distillation strategy is adopted to allow the student network to simulate the prediction logic and intermediate feature representation of the teacher network, while the local teacher network is fine-tuned in reverse to better adapt to the distribution characteristics of the local data.

[0017] The adaptive bidirectional knowledge distillation training is as follows: the client inputs local batch data into the local student network. Global teacher network with frozen parameters Obtain the logical layer output of the student network respectively. Mapping with intermediate feature layers and the logical layer output of the teacher network Mapping with intermediate feature layers ;

[0018] The client selects the training state flag for the t-th round issued by the server. Determine whether the current state is core or auxiliary, and calculate the distillation loss weights of the logic layer in the current training round. Distillation loss weights of feature layers Furthermore, a comprehensive loss function is constructed, which includes classification loss based on cross-entropy, logistic layer distillation loss based on KL divergence, and feature layer alignment loss based on mean square error. Finally, the comprehensive loss function is minimized using gradient descent. To update local model parameters.

[0019] Preferably, the retention rate in the t-th training round is: ;in, For the largest round, The initial retention rate, The final retention rate;

[0020] The training loss score is calculated as follows: the client traverses the private local dataset. For all local samples, calculate their combined loss function value under the current model, and define the arithmetic mean of the combined loss function values ​​of all samples as the training loss score for this round. .

[0021] Preferably, the method for generating the compressed parameter package is as follows:

[0022] The original model parameter matrix is: ;

[0023] The cutoff rank, determined based on the energy threshold, is: ;

[0024] The final compressed parameter package: ;

[0025] in, Represents the model parameter matrix The left singular vector matrix after SVD decomposition, This represents a submatrix consisting of all rows and the first r columns of the left singular vector matrix U. Let r be the vector consisting of the first r elements of the singular values ​​arranged in descending order. Represents the model parameter matrix The submatrix of the first r rows and all columns of the right singular vector matrix V after SVD decomposition; It is a diagonal matrix. Given a diagonal matrix of singular values ​​arranged in descending order, the total number of singular values ​​is... , Let 'a' represent the number of rows and columns of the model parameter matrix W, respectively, and 'a' represent the retention rate that satisfies the condition that the cumulative energy percentage is not less than the target information retention rate. The smallest singular value index, represents the energy of the i-th singular value; T is the transpose of the matrix.

[0026] Preferably, the method for dynamically adjusting the distillation loss weights of the feature layer and the logic layer based on the received selection status identifier is as follows: adjusting the local logic layer base weights... With the basic weights of the feature layer The results were obtained by adaptive dynamic adjustment respectively:

[0027] Distillation loss weights in the logic layer ;

[0028] Distillation loss weights of feature layers ;

[0029] in, The preset attenuation coefficient

[0030] The comprehensive loss function ;

[0031] in, The distillation temperature coefficient and the classification loss of the cross-entropy are given. ;in, One-hot encoding for the real label, The Softmax activation function is used. Total number of categories; This represents the original confidence score of the local student network on class c after forward propagation of the input data, without Softmax activation.

[0032] Logic layer distillation loss ;in, This represents the raw confidence score of the global teacher network on the c-th class of the input data, without Softmax activation.

[0033] Feature layer alignment loss ;in, It is a mapping of intermediate feature layers in the local student network. It is a mapping of intermediate feature layers in the global teacher network;

[0034] The current model is trained locally on the client using adaptive bidirectional distillation, and then processed by a comprehensive loss function. Optimized and updated local student network The teacher network, which has been updated via backpropagation using gradient descent and incorporates local data features with global data, is now available. Knowledge;

[0035] The training loss score ;in, For the first Local datasets for each client The total number of samples; Represents the first in the client's local dataset One input sample, Indicates input sample The corresponding real tags; Represents local dataset The comprehensive loss function value of the i1th input sample;

[0036] After completing the preset number of local training rounds, upload the updated local model parameters to the server. The updated local model parameters are the student network parameters after multiple rounds of iterative optimization by minimizing the comprehensive loss function through gradient descent.

[0037] Preferably, the method for calculating fraction-based soft aggregation weights using a negative exponential function is as follows:

[0038] The server collects the training loss scores uploaded by all participating clients in the current round. And calculate the sum of the negative exponential terms of all training loss scores to obtain the cumulative negative exponential sum. ; Summing by negative exponents As a normalization factor, calculate the first... The soft aggregation weight of each client;

[0039] Introducing a numerical stability protection mechanism to determine the sum of negative exponents. Is it below the preset minimum threshold? To prevent numerical underflow or division by zero errors; if the numerical value is determined to be abnormal, a fallback strategy is automatically triggered, and weights are allocated using an average aggregation method; if the numerical value is determined to be normal, the normalized weights are obtained by dividing the negative exponent of each client by the denominator using the standard Softmax function logic.

[0040] Preferably, the first The soft aggregation weight for each client is

[0041]

[0042] in, For the first Local training loss scores uploaded by each client. The total number of participating clients; The preset numerical stability threshold, This is the sum of the negative exponents of the scores from all participating clients in the current training round, and ;in, The set of clients participating in the aggregation in the current round. For client collection The training loss score for the j1th client.

[0043] Preferably, the method for restoring and weighted summing the local model parameters uploaded by each client is as follows:

[0044] The client packages the compressed parameter package after singular value decomposition and energy truncation along with the local training loss score and uploads it to the server. After receiving the package, the server performs a normalization operation based on the scores uploaded by all participating clients using the negative exponential function Softmax, and calculates the soft aggregate weight for each client.

[0045] The client completes local training and generates a compressed parameter package and training loss score. The data packets are packaged and uploaded to the server; after receiving all the data packets uploaded by the clients participating in the aggregation, the server extracts the training loss score for each client. The soft aggregation weight for each client is calculated using a negative exponential function and Softmax normalization. The compression parameters of each client are restored, and then the soft aggregation weights are used to perform a layer-by-layer weighted summation on the parameter matrix reconstructed by all clients to generate a new generation of global model.

[0046] The server iterates through the local model parameters uploaded by each client. For layers in a compressed state, it uses matrix multiplication to reconstruct the singular value triples into parameter matrices of the original dimensions. For uncompressed layers, it leaves them as is.

[0047] Preferably, the first Compressed data packets uploaded by clients ;in, For the first Training loss score for each client; Indicates the first The first client's The layer model parameter matrix is ​​decomposed by SVD and truncated to rank r, resulting in a left singular vector submatrix. Indicates the first The layer model parameter matrix is ​​truncated to a singular value diagonal matrix of rank r. Indicates the first The layer model parameter matrix is ​​truncated to the right singular vector submatrix of rank r. This represents the collection of all layers in the student network that need to be SVD compressed and uploaded, representing the client's network.

[0048] The server recovered on the [date]. The client trains in round t, and the... The layer parameter matrix is ;

[0049] The aggregated model generates the global model in the (t+1)th round of training. Layer parameters are .

[0050] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention significantly improves communication efficiency, model accuracy, and system robustness within the federated learning framework. Specifically, this invention overcomes the limitations of static weight allocation in traditional federated distillation by introducing a "core-auxiliary" dual-state adaptive adjustment mechanism. It dynamically adjusts the distillation intensity of the logic layer and feature layer based on the selected state of the client, intelligently identifying dominant nodes and maintaining their high-intensity feature alignment, while automatically implementing weight decay for weak nodes. This effectively avoids feature collapse and knowledge forgetting caused by weak nodes forcibly mimicking the global model in heterogeneous data scenarios. Furthermore, this invention innovatively establishes a negative exponential mapping relationship between training loss scores and aggregation weights, replacing traditional average aggregation or hard truncation strategies with score-aware soft aggregation. The invention intelligently assigns high aggregation weights to data nodes with low loss and high data quality, while exponentially suppressing the weights of nodes with high loss and potential malicious poisoning. This maximizes the robustness of the global model to anomaly attacks without directly discarding participants. Furthermore, the invention introduces a dynamic SVD parameter compression mechanism based on energy spectrum, which linearly increases the information retention rate according to the training process. This achieves an adaptive communication strategy of "aggressive compression in the early stage and fine transmission in the later stage." While significantly reducing communication bandwidth consumption, the invention automatically filters noise using singular value truncation, effectively balancing the relationship between communication overhead, privacy and security, and model utility. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of the present invention.

[0053] Figure 2 This is a framework diagram of the present invention.

[0054] Figure 3 This is a graph showing the trend of the accuracy of the present invention versus other methods as a function of training rounds. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] like Figure 1 As shown, this invention provides a federated distillation method based on adaptive bidirectional distillation and soft-weighted aggregation, aiming to significantly reduce communication transmission overhead, improve model convergence accuracy in non-independent and identically distributed data environments, and enhance robustness against malicious node poisoning attacks. The steps are as follows: the server sends a global model and the client's selection status identifier to the client; the client constructs a local teacher-student network, dynamically adjusts the distillation loss weights of the feature layer and logic layer according to the received selection status, performs adaptive bidirectional knowledge distillation training, and dynamically adjusts the information retention rate according to the training rounds to compress the model parameters, uploading the compressed parameters and training loss scores to the server; the server collects the scores uploaded by each client, calculates score-based soft aggregation weights using a negative exponential function, and reduces the weight ratio of high-loss clients; the server uses the calculated soft aggregation weights to restore and weightedly sum the parameters uploaded by each client, generating a new generation global model. This invention significantly reduces communication transmission overhead, improves model convergence accuracy in non-independent and identically distributed data environments, and enhances the system's robustness against malicious node poisoning attacks. The specific implementation steps of this invention are as follows:

[0057] S1: Federated learning begins. The server sends the global model and the client's selection status flag to the client.

[0058] Specifically, the selection status indicator is based on a preset client filtering ratio set by the server. The first to participate in the training Each client is marked as either a core state or an auxiliary state, and a corresponding selection state identifier is generated. ,in, This represents the selected core state. This represents an unselected auxiliary state.

[0059] A cold start strategy is adopted for the selection status flag in the first round: In the initial training round (t=0), since there is a lack of historical training loss scores as a basis for selection, the server initializes the selection status flag of all participating clients to a uniform value. (Core State). This forces all clients to perform high-intensity bidirectional feature alignment using undecayed base weights during the initial stages of global model convergence, thus avoiding training instability or limited convergence speed caused by prematurely suppressing some nodes before model features are fully formed. Base weights include the base weights of the logic layer. With the basic weights of the feature layer .

[0060] The global model distributed by the server plays a core role in the global teacher network, and its internal structure includes an embedding layer, a feature extraction layer, and a classification output layer.

[0061] During the distribution phase, the server not only transmits model parameters dynamically compressed using singular value decomposition to significantly reduce network bandwidth consumption, but also simultaneously distributes a selection state flag. This selection state flag is responsible for classifying different clients into core or auxiliary states, thereby guiding the clients to complete adaptive bidirectional knowledge distillation in conjunction with the global model: for core clients, the global model provides high-intensity guidance through logic layer output and intermediate feature information; while for auxiliary clients, the guidance intensity is reduced through an attenuation coefficient. This design ensures that the system maintains high model convergence accuracy and robustness against attacks even in the face of unevenly distributed data environments.

[0062] As an example, Figure 2 This paper demonstrates the training framework of the proposed method in a real-world application scenario. The dataset used in the experiments is AGNews, a recognized benchmark dataset for short text classification in Natural Language Processing (NLP), constructed by Zhang et al. in 2015 based on the AG's Corpus news corpus. The AGNews dataset covers four categories: "World," "Sports," "Business," and "Sci / Tech," containing 120,000 training samples and 7,600 test samples. The sample distribution across categories is strictly balanced, with 30,000 training data points and 1,900 test data points for each category. The data samples mainly consist of news headlines and brief descriptions, exhibiting semantic conciseness and feature sparsity. In this invention, the AGNews dataset is chosen to verify the generality of the proposed federated distillation learning method on non-image modalities, and to focus on evaluating the convergence performance and communication efficiency of the adaptive bidirectional distillation strategy and the singular value decomposition (SVD) dynamic compression mechanism when processing natural language models (Transformers).

[0063] S2: The client constructs a local teacher-student network, dynamically adjusts the distillation loss weights of the feature layer and logic layer based on the received selection status identifier, performs adaptive bidirectional knowledge distillation training, and dynamically adjusts the information retention rate according to the training rounds to compress the model parameters. The compressed parameter package and training loss score are then uploaded to the server.

[0064] When constructing the local teacher-student network, the client first obtains the global model parameters issued by the server and uses them as the local "teacher model," while using the client's original or initialized parameters as the "student model." During training, the system employs a bidirectional distillation strategy, which means that the student model not only simulates the prediction logic of the teacher model (by calculating the KL divergence of the logic output layer) and intermediate feature representations (by aligning the mean squared error of the feature maps), but also fine-tunes the local teacher model in reverse to better adapt it to the distribution characteristics of the local data.

[0065] This approach firstly, by leveraging the deep knowledge provided by the teacher model, the student model can overcome obstacles posed by different data distributions (non-independent and identically distributed), significantly improving convergence speed and final accuracy in the early stages of training. Secondly, an "adaptive" mechanism is introduced to dynamically adjust the distillation intensity based on the client's assigned selection state flag (core or auxiliary), avoiding model performance collapse caused by forced synchronization when data quality is poor or under attack. Finally, this localized bidirectional distillation eliminates the need for the client to directly transmit complete model gradients. Combined with singular value decomposition (SVD) compression technology, it greatly reduces the communication load between the client and server, achieving high communication efficiency while ensuring global model robustness.

[0066] Specifically, the calculation method for dynamically adjusting the distillation loss weights of the feature layer and the logic layer based on the received selection state identifier is as follows: The client first calculates the selection state identifier of the t-th training round issued by the server. The system determines whether the current state is core or auxiliary, and calculates the distillation loss weights for the logic and feature layers required for this training round accordingly. The client receives the selection state identifier. Then, a preset attenuation coefficient is introduced. The role of the attenuation coefficient is to reduce the coercive force of distillation when the client is not selected, so as to prevent "bad students" from forcibly imitating "good teachers" and thus learning wrong (preventing negative transfer).

[0067] Local logical layer basic weights With the basic weights of the feature layer The specific calculation formula for obtaining the distillation loss weight by performing adaptive dynamic adjustment is as follows:

[0068]

[0069] ;

[0070] in, Distillation loss weights for the logic layer The distillation loss weights for the feature layer.

[0071] This mechanism allows when a status flag is selected. The client maintains the original high-intensity knowledge distillation to maximize global model guidance, while selecting state flags. Automatically decay the weight to This allows for a hierarchical, differentiated training strategy while ensuring model convergence.

[0072] The specific calculation method for adaptive bidirectional knowledge distillation training is as follows: the client first processes the local batch data... Input to local student network Global teacher network with frozen parameters Obtain the logical layer output of the student network respectively. Mapping with intermediate feature layers and the logical layer output of the teacher network Mapping with intermediate feature layers .

[0073] In the specific implementation, the client simultaneously inputs local batch data into both the local student network and the global teacher network with frozen parameters, performing forward propagation computation to obtain the key information needed for distillation. Specifically, as the data flows through each layer of the network, before entering the final classification output layer, the neuron activation vector extracted from the penultimate layer becomes the intermediate feature layer mapping, representing the model's deep semantic understanding of the input data. Subsequently, the data continues to pass through the classification output layer to generate the unnormalized (without Softmax operation) raw score vector, which becomes the logic layer output. In this way, the client obtains the performance of the teacher network and the student network in two dimensions: feature extraction and logic prediction. Then, it uses the mean squared error (MSE) loss function to align the feature layer mappings and uses KL divergence to fit the logic layer output.

[0074] Based on the selection status flag issued by the server Calculate the distillation loss weight of the logic layer in the current round. Distillation loss weights of feature layers A comprehensive loss function is constructed, which includes classification loss, logistic layer distillation loss, and feature layer alignment loss. Finally, the comprehensive loss function is minimized using gradient descent. To update local model parameters.

[0075] During adaptive bidirectional distillation training, the client constructs a composite loss function with multiple dimensions to synchronously update the student and teacher networks. First, the student network update originates from the weighted sum of the local classification loss, the logistic layer distillation loss (KL divergence), and the feature layer distillation loss (MSE). Through backpropagation, gradients are simultaneously propagated back from these three loss functions to the student network, forcing it to align with the global teacher model's prediction logic and high-dimensional features while learning local private data features. Meanwhile, the teacher network is not completely fixed but fine-tuned through a "backdistillation" mechanism: the system calculates the student network's feature feedback loss to the teacher network and updates the teacher network parameters with a small learning rate. In this process, the adaptive dynamic adjustment mechanism plays a crucial control role, dynamically adjusting the distillation weights of the logistic and feature layers based on the selection state flag issued in the current round. This means that when the flag indicates a core state, the network drives deep fusion of the teacher and student models through a high-intensity gradient flow; while when the flag indicates an auxiliary state, the gradient update intensity is weakened through a decay factor. Ultimately, this bidirectional update mechanism enables the student network to quickly absorb global experience, while the teacher network can capture the unique distribution of local data without compromising privacy, thus obtaining an optimized model with improved performance in both directions.

[0076] Among them, classification loss

[0077]

[0078] in, The classification loss is the cross-entropy. One-hot encoding for the real label, The Softmax activation function is used. Total number of categories; This represents the raw confidence score output by the local student network on class c after forward propagation of the input data, without softmax activation. One-hot encoding of the true label. It is obtained by first mapping the original category labels to integer indices, and then generating a binary vector with a length equal to the total number of categories C, where only the corresponding index position is 1 and the rest are 0.

[0079] Logic layer distillation loss

[0080]

[0081] in, This is a logic layer distillation loss based on KL divergence, used to measure the difference in output probability distributions between the student and teacher networks. The distillation temperature coefficient is used to soften the probability distribution in order to extract dark knowledge. This represents the raw confidence score of the global teacher network on the c-th class of the input data, without softmax activation. and These are the dimensions corresponding to the c-th class in the logical layer outputs of the teacher network and the student network, respectively, used to measure the difference in the probability distributions of the two networks' outputs using KL divergence.

[0082] Feature layer alignment loss

[0083]

[0084] in, This is a feature layer alignment loss based on mean squared error, used to force the intermediate feature layer mapping of the student network. Approximating the intermediate feature layer mapping of the teacher network . It is an intermediate feature layer mapping of the local student network. This is the intermediate feature layer mapping of the global teacher network. The loss calculates the difference between the two using mean squared error, thus forcing the intermediate feature layer mapping of the student network. Approximating the intermediate feature layer mapping of the teacher network This enables knowledge alignment at the feature level.

[0085] Therefore, the comprehensive loss function

[0086]

[0087] in, To arrive at the final optimized comprehensive loss function, the client calculates the gradient of the comprehensive loss function. Update the local parameters through backpropagation.

[0088] The specific method for dynamically adjusting the information retention rate based on the training rounds to compress model parameters is as follows:

[0089]

[0090] in, For the current round The target information retention rate at the largest round Internal retention rate (e.g., 0.8) linearly increase to the final retention rate (e.g., 0.95).

[0091] The client first obtains the current global training round. Based on the preset initial retention rate With end retention rate The target information retention rate for the current round is calculated using a linear growth function. This allows the model's information to gradually become more complete as the training process progresses.

[0092] The original model parameter matrix (e.g., the weight matrix of a fully connected layer) is:

[0093] ;

[0094] The cutoff rank, determined based on the energy threshold, is:

[0095] ;

[0096] The final compressed parameter package:

[0097] ;

[0098] in, Describes a left singular vector matrix. This represents a submatrix of the left singular vector matrix U, consisting of all rows and the first r columns. This represents a vector consisting of the first r elements from the singular values ​​arranged in descending order. This represents the transpose of the submatrix of the right singular vector matrix V, taking the first r rows and all columns. It is a diagonal matrix. For the model parameter matrix The right singular vector matrix after SVD decomposition. Given a diagonal matrix of singular values ​​arranged in descending order, the total number of singular values ​​is... , Let 'a' represent the number of rows and columns of the original model parameter matrix W, respectively, and 'a' represent the condition that "the cumulative energy percentage is not less than the target information retention rate". The smallest principal component index of " The energy representing the i-th principal component (singular value) is used in this formula to ensure that the amount of information retained relative to the total energy is at least the target information retention rate. ;i is the sort index for the singular values ​​after SVD decomposition, arranged in descending order. The final generated compressed parameter package It includes the truncated left singular vector, singular value vector, and right singular vector, and its transmission volume is much smaller than the original model parameter matrix. The entire compressed parameter package P is generated through this truncation process. compressed This significantly reduces the transmission volume.

[0099] For the model parameter matrix to be uploaded in the local model Perform Singular Value Decomposition (SVD), obtain the singular value sequence and calculate its cumulative energy spectrum, and select the sequences whose cumulative energy percentage is not less than the target information retention rate. minimum principal component rank Finally, based on the determined rank The decomposed left singular vector matrix, singular value diagonal matrix, and right singular vector matrix are truncated, retaining only the first three elements. Each principal component is uploaded as a compressed parameter package, thereby achieving a dynamic and smooth transition from "coarse-grained high-efficiency transmission" to "fine-grained high-precision transmission".

[0100] Uploaded model parameter matrix After the client completes local adaptive bidirectional distillation training, it accesses the updated local student network. The weight matrix of the layer to be uploaded (such as a fully connected layer) is extracted, which minimizes a comprehensive loss function that includes classification loss, logistic layer distillation loss, and feature layer alignment loss. The data, obtained through backpropagation using gradient descent, is key parameter data that carries the core knowledge of local training and is suitable for compressed transmission.

[0101] The training loss score is calculated as follows: after completing local adaptive bidirectional distillation training, the client uses a private local dataset... A comprehensive assessment of the updated student network was conducted.

[0102] Specifically, the client iterates through the private local dataset. For all local samples, calculate their comprehensive loss function value under the current model. This comprehensive loss function value includes not only the basic cross-entropy classification loss, but also the loss function value based on the current selection state identifier. The dynamically weighted logical layer bidirectional KL divergence loss and the feature layer alignment mean square error are used. Finally, the arithmetic mean of the comprehensive loss function values ​​of all samples is defined as the training loss score for this round. This score quantitatively characterizes the fitting quality and generalization error of the local model under the current heterogeneous data distribution, and is uploaded to the server as a key indicator for subsequent calculation of negative exponential soft aggregation weights to remove potential outliers with high loss. The current model is trained locally on the client side using an adaptive bidirectional distillation method, and then processed by a comprehensive loss function. Optimized and updated local student network Its parameters have been updated via backpropagation using gradient descent, integrating local data features with the global teacher network. The knowledge is used to calculate the comprehensive loss of each sample and ultimately generate the training loss score. The core model.

[0103] single sample The overall loss function value is:

[0104] ;

[0105] No. The training loss scores uploaded by each client are:

[0106]

[0107] Among them, training loss score The lower the value, the higher the model quality or the lower the data noise; For the first Local datasets for each client The total number of samples; Represents the first in the client's local dataset Each input sample (such as text, image, etc.) Indicates input sample The corresponding real tags; Represents local dataset The comprehensive loss function value of the i1th sample. 1 is the sample index, used only to iterate through all samples in the dataset. For the sample The standard cross-entropy loss for supervised learning tasks is used to ensure basic classification accuracy. Represents the bidirectional KL divergence, measuring the logical layer output of the student network. Logical layer output of the teacher network The distribution differences. To compensate for feature layer alignment error, force the intermediate feature layer mapping of the student network. Approximating the intermediate feature layer mapping of the teacher network . and Essentially the same, both are bidirectional KL divergence, which measures the difference in the output distribution of the logical layer of the student network and the teacher network. This represents the intermediate feature layer mapping extracted by the local student network from the input sample x. This represents the intermediate feature layer mapping of the global teacher network to the input sample x.

[0108] Repeat the above steps until the preset number of local training rounds is completed, and then upload the final updated local model parameters as output to the server. The essence of updating the local model parameters is to minimize the comprehensive loss function through gradient descent, completing multiple rounds of iterative optimization of the student network parameters. The final updated local model parameters are obtained by the client within the preset local training rounds, starting from the initial local student network parameters and targeting the local dataset. Each sample in Calculate the comprehensive loss function (The cross-entropy classification loss, dynamically weighted logistic layer KL divergence loss, and feature layer alignment loss are combined), and the loss is backpropagated to the parameters of each layer of the student network through the gradient descent algorithm. The parameters are continuously iterated and updated to minimize the comprehensive loss. After all the preset training rounds are completed, all the trainable parameters of the student network are the "updated local model parameters" that need to be uploaded to the server.

[0109] S3: The server collects the training loss scores uploaded by each client, calculates soft aggregate weights based on the scores using a negative exponential function, and reduces the weight ratio of clients with high loss.

[0110] Specifically, the method for calculating fraction-based soft aggregation weights using the negative exponential function is as follows:

[0111] The server first collects the training loss scores uploaded by all participating clients in the current round. And calculate the sum of the negative exponential terms of all training loss scores. To construct a normalized denominator.

[0112] In this process, a numerical stability protection mechanism is introduced to determine whether the denominator is lower than a preset minimum threshold. To prevent numerical underflow or division by zero errors; if the numerical value is determined to be abnormal (i.e., all client scores are too high, causing the exponent term to underflow), a fallback strategy is automatically triggered, and weights are distributed using an average aggregation method to ensure the system's operational stability in extreme environments; if the numerical value is determined to be normal, the standard Softmax function logic is used to divide the negative exponent term of each client by the denominator to obtain a normalized weight, so that clients with low scores receive a weight close to 1, while the weights of clients with high scores decay exponentially to close to 0, thus achieving defense against poisoning attacks.

[0113] The sum of the negative exponents of all participating client ratings in the current round:

[0114]

[0115] in, The set of clients participating in the aggregation in the current round. For client collection The training loss score for the j1-th client. j1 is the set of clients participating in aggregation in the current round. The client index in the data is only used to iterate through all participating clients in the set and calculate the sum of negative exponential weights. .

[0116] Negative exponential summation As a normalization factor, the final calculated number is... The soft aggregation weight for each client is

[0117]

[0118] in, For the first The local training loss score uploaded by each client; the smaller the value, the better the model quality. The total number of participating clients; A preset numerical stability threshold is used to detect the risk of numerical underflow; soft aggregation weights. It not only achieves adaptive allocation based on data quality, but also effectively suppresses the impact of potential poisoning attacks or poor-quality data through the non-linear characteristics of the exponential function.

[0119] S4: The server uses the calculated soft aggregation weights to restore and weight the parameters uploaded by each client, generating a new generation of global model.

[0120] Specifically, the method for restoring and weighted summing the parameters uploaded by each client is as follows:

[0121] No. Data packets uploaded by each client

[0122]

[0123] No. Data packets uploaded by each client Includes the SVD compression components of each layer and the training loss score for this client. . Indicates the first The first client's The layer model parameter matrix is ​​decomposed by SVD and truncated to rank r, resulting in a left singular vector submatrix. This represents the singular value diagonal matrix truncated to rank r of this layer. This represents the right singular vector submatrix truncated to rank r. Indicates transpose; This represents the set of all layers in the student network that need to be compressed using SVD and uploaded (such as fully connected layers and other layers suitable for matrix factorization, excluding convolutional layers, embedding layers, and other layers that do not require compression).

[0124] The client first packages the compressed parameter package (containing the left singular vector matrix, singular value vector, and right singular vector matrix) after singular value decomposition and energy truncation, together with the local training loss score, and uploads it to the server.

[0125] No. The soft aggregation weight for each client is

[0126]

[0127] Aggregate weight Scoring by training loss After negative exponential normalization, it was found that this mechanism gives greater weight to low-scoring (high-quality) models and suppresses high-scoring (low-quality or poisoned) models. This refers to the set of clients participating in the aggregation in the current round. This represents the set of clients participating in the aggregation in the current round. The training loss score for the j1-th client. Points to the specific k-th client (target client). It is a general index for traversing the collection (all participating clients).

[0128] After receiving the data, the server first performs a normalization operation using the negative exponential function Softmax based on the scores uploaded by all participating clients, and calculates the soft aggregation weight for each client to automatically reduce the contribution of high-loss nodes.

[0129] This part is the first operation performed by the server after receiving the uploaded data. It is the starting point of the server aggregation update process and a prerequisite for subsequent parameter restoration and weighted summation.

[0130] The client completes local training and generates a compressed parameter package (including SVD components) + training loss score. The data is packaged and uploaded to the server; the server receives all clients participating in the aggregation (collection). After uploading the data packet, first extract the training loss score for each client. (or iterate through S corresponding to index j1) j1 The soft aggregate weight for each client is calculated using a negative exponential function plus Softmax normalization. Its core function is to dynamically allocate weights based on scores to suppress high-loss nodes; the next step is based on the calculated soft aggregation weights. The compression parameters of each client are restored (the SVD components are reconstructed into the original dimension parameter matrix), and then a new generation of global model is generated by weighted summation.

[0131] The server iterates through the parameters uploaded by each client. For layers in a compressed state, it uses matrix multiplication to reconstruct the parameter matrix of the original dimension from the singular value triples. For uncompressed layers, the original state is maintained. This significantly reduces client-server communication overhead while ensuring model accuracy. The core logic for identifying compressed layers is based on predefined "compressed layer identifiers" or parameter packet structure features. Only layers suitable for SVD decomposition (such as fully connected layers, with two-dimensional parameter matrices and a large number of parameters) are compressed and transmitted. Uncompressed layers (such as convolutional layers and BatchNorm layers, with few parameters or structures unsuitable for matrix decomposition) are transmitted directly. This avoids meaningless compression computation overhead and significantly reduces the amount of uploaded data through core layer compression (typically reducing transmission volume by 60%-90%).

[0132] The server recovered on the [date]. The client trains in round t, and the... The layer parameter matrix is

[0133]

[0134] Parameter matrix Through the left singular vector Singular value diagonal matrix With right singular vectors Obtained by matrix multiplication and reconstruction. , , It is the core output of the client's SVD decomposition of the target layer parameters plus energy truncation, generated locally and then uploaded. Its core value lies in reducing communication costs, maintaining model accuracy, and dynamically adapting to the training process after parameter reconstruction. It is the first The first client The layer model parameter matrix is ​​decomposed by SVD and truncated to a left singular vector submatrix of rank r. It is the singular value vector truncated to rank r of this layer. It is the right singular vector submatrix truncated to rank r of this layer, and the three are derived from the first... The client in the first After the first round of local training, for the second round The original parameter matrix of the layer is decomposed using SVD, and then filtered for minimum rank based on "cumulative energy percentage ≥ target information retention rate Et". and cut off the front The principal components are obtained and then uploaded to the server; the server performs matrix multiplication. Reconstruct the parameter matrix This method significantly reduces client-server communication overhead, while preserving high-energy principal components to ensure a high degree of approximation between the reconstructed parameters and the original parameters. Furthermore, it allows for dynamic adjustment... It achieves a smooth transition from prioritizing communication efficiency in the early stages of training to prioritizing parameter accuracy in the later stages, while keeping the computational overhead of server reconstruction controllable.

[0135] Finally, the calculated soft aggregation weights are used to perform a layer-by-layer weighted summation on the reconstructed parameter matrices of all clients, thereby generating an updated next-generation global model.

[0136] The aggregated model generates the global model in the (t+1)th round of training. Layer parameters are

[0137]

[0138] parameter It is a weighted sum of all parameters after client recovery, achieving effective knowledge integration.

[0139] S5: Repeat steps S1 to S4 until the given maximum number of communication rounds is reached, which is usually set to 200 rounds.

[0140] like Figure 3 As shown, experimental results indicate that the CASKD method proposed in this invention is effective in extremely non-independent identically distributed systems (ISDs). This invention demonstrates significant advantages in scenarios where federated learning is most complex and heterogeneous. Specifically, the classic federated averaging (FedAvg) algorithm in federated learning is limited by severe data heterogeneity, resulting in slow and fluctuating accuracy. While federated knowledge distillation (FedKD) introduces a distillation mechanism, its performance is bottlenecked due to a lack of targeted weight allocation. In contrast, this invention, with its adaptive bidirectional distillation and score-aware soft aggregation strategy, effectively overcomes model drift, achieving rapid accuracy increases in the early stages of training and ultimately outperforming the comparative algorithms. Furthermore, the CASKD algorithm in this invention maintains extremely high stability and a smooth upward trend in the later stages of training, fully demonstrating that the algorithm achieves strong robustness in complex and heterogeneous scenarios through intelligent knowledge selection and dynamic compression mechanisms, significantly improving the model's generalization ability and training efficiency.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization, characterized in that, The steps are as follows: S1: Federated learning begins; the server sends the global model and the client's selection status flag to the client. S2: The client builds a local teacher-student network, dynamically adjusts the distillation loss weights of the feature layer and logic layer according to the received selection status identifier, performs adaptive bidirectional knowledge distillation training, and dynamically adjusts the information retention rate according to the training rounds to compress the local model parameters, and uploads the compressed parameter package and training loss score to the server. S3: The server collects the training loss scores uploaded by each client and uses a negative exponential function to calculate the score-based soft aggregation weights. S4: The server uses soft aggregation weights to restore and weight the local model parameters uploaded by each client, generating a new generation of global model; S5: Repeat steps S1 to S4 until the given maximum number of communication rounds is reached.

2. The federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization according to claim 1, characterized in that, The selection status indicator is based on a preset client filtering ratio set by the server. The first to participate in the training Each client is marked as either core or auxiliary, and a selection status identifier is generated. ,in, This represents the selected core state. This indicates an unselected auxiliary state; During the initial training rounds, the server initializes the selection status flags of all participating clients to a uniform value. ; The server transmits the global model parameters after dynamic compression via singular value decomposition and simultaneously issues a selection state identifier. For clients in the core state, the global model provides strong guidance through logic layer output and intermediate feature information. For clients in the auxiliary state, the guidance intensity is reduced through an attenuation coefficient.

3. The federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization according to claim 1 or 2, characterized in that, The method for the client to construct a local teacher-student network is as follows: the client obtains the global model parameters issued by the server and uses the global model parameters as the local teacher network, while using the client's original or initialized local model parameters as the student network; during the training process, a bidirectional distillation strategy is adopted to allow the student network to simulate the prediction logic and intermediate feature representation of the teacher network, while the local teacher network is fine-tuned in reverse to better adapt to the distribution characteristics of the local data. The adaptive bidirectional knowledge distillation training is as follows: the client inputs local batch data into the local student network. Global teacher network with frozen parameters Obtain the logical layer output of the student network respectively. Mapping with intermediate feature layers and the logical layer output of the teacher network Mapping with intermediate feature layers ; The client selects the training state flag for the t-th round issued by the server. Determine whether the current state is core or auxiliary, and calculate the distillation loss weights of the logic layer in the current training round. Distillation loss weights of feature layers Furthermore, a comprehensive loss function is constructed that includes classification loss based on cross-entropy, logistic layer distillation loss based on KL divergence, and feature layer alignment loss based on mean square error. Finally, the comprehensive loss function is minimized using gradient descent. To update local model parameters.

4. The federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization according to claim 3, characterized in that, The retention rate in training round t is: ;in, For the largest round, The initial retention rate, The final retention rate; The training loss score is calculated as follows: the client traverses the private local dataset. For all local samples, calculate their combined loss function value under the current model, and define the arithmetic mean of the combined loss function values ​​of all samples as the training loss score for this round. .

5. The federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization according to claim 4, characterized in that, The method for generating the compressed parameter package is as follows: The original model parameter matrix is: ; The cutoff rank, determined based on the energy threshold, is: ; The final compressed parameter package: ; in, Represents the model parameter matrix The left singular vector matrix after SVD decomposition, This represents a submatrix consisting of all rows and the first r columns of the left singular vector matrix U. Let r be the vector consisting of the first r elements of the singular values ​​arranged in descending order. Represents the model parameter matrix The submatrix of the first r rows and all columns of the right singular vector matrix V after SVD decomposition; It is a diagonal matrix. Given a diagonal matrix of singular values ​​arranged in descending order, the total number of singular values ​​is... , Let 'a' represent the number of rows and columns of the model parameter matrix W, respectively, and 'a' represent the retention rate that satisfies the condition that the cumulative energy percentage is not less than the target information retention rate. The smallest singular value index, represents the energy of the i-th singular value; T is the transpose of the matrix.

6. The federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization according to claim 5, characterized in that, The method for dynamically adjusting the distillation loss weights of the feature layer and the logic layer based on the received selection status flag is as follows: Adjust the local logic layer base weights... With feature layer basic weights The results were obtained by adaptive dynamic adjustment respectively: Distillation loss weights in the logic layer ; Distillation loss weights of feature layers ; in, The preset attenuation coefficient The comprehensive loss function ; in, The classification loss is the cross-entropy, which represents the distillation temperature coefficient. ;in, One-hot encoding for the real label. The Softmax activation function is used. Total number of categories; This represents the original confidence score of the local student network on class c after forward propagation of the input data, without Softmax activation. Logic layer distillation loss ;in, This represents the raw confidence score of the global teacher network on the c-th class of the input data, without Softmax activation. Feature layer alignment loss ;in, It is an intermediate feature layer mapping of the local student network. It is a mapping of intermediate feature layers in the global teacher network; The current model is trained locally on the client using adaptive bidirectional distillation, and then processed by a comprehensive loss function. Optimized and updated local student network The teacher network, which has been updated via backpropagation using gradient descent and incorporates local data features with global data, is now available. Knowledge; The training loss score ;in, For the first Local datasets for each client The total number of samples; Represents the first in the client's local dataset One input sample, Indicates input sample The corresponding real tags; Represents local dataset The comprehensive loss function value of the i1th input sample; After completing the preset number of local training rounds, upload the updated local model parameters to the server. The updated local model parameters are the student network parameters after multiple rounds of iterative optimization by minimizing the comprehensive loss function through gradient descent.

7. The federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization according to any one of claims 4-6, characterized in that, The method for calculating fraction-based soft aggregation weights using a negative exponential function is as follows: The server collects the training loss scores uploaded by all participating clients in the current round. And calculate the sum of the negative exponential terms of all training loss scores to obtain the cumulative negative exponential sum. ; Sum of negative exponents As a normalization factor, calculate the first The soft aggregation weight of each client; Introducing a numerical stability protection mechanism to determine the sum of negative exponents. Is it below the preset minimum threshold? To prevent numerical underflow or division by zero errors; If the value is determined to be abnormal, a rollback strategy will be automatically triggered, and the weights will be distributed using an average aggregation method. If the value is determined to be normal, the normalized weight is obtained by dividing the negative exponent of each client by the denominator using the standard Softmax function logic.

8. The federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization according to claim 7, characterized in that, No. The soft aggregation weight for each client is in, For the first Local training loss scores uploaded by each client. The total number of participating clients; The preset numerical stability threshold, This is the sum of the negative exponents of the scores from all participating clients in the current training round, and ;in, The set of clients participating in the aggregation in the current round. For client collection The training loss score for the j1th client.

9. The federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization according to any one of claims 4-6 and 8, characterized in that, The method for restoring and weighted summing the local model parameters uploaded by each client is as follows: The client packages the compressed parameter package after singular value decomposition and energy truncation along with the local training loss score and uploads it to the server. After receiving the data, the server performs a normalization operation using the negative exponential function Softmax based on the scores uploaded by all participating clients, and calculates the soft aggregate weight for each client. The client completes local training and generates a compressed parameter package and training loss score. The data packets are packaged and uploaded to the server; after receiving all the data packets uploaded by the clients participating in the aggregation, the server extracts the training loss score for each client. The soft aggregation weight for each client is calculated using a negative exponential function and Softmax normalization. The compression parameters of each client are restored, and then the soft aggregation weights are used to perform a layer-by-layer weighted summation on the parameter matrix reconstructed by all clients to generate a new generation of global model. The server iterates through the local model parameters uploaded by each client. For layers in a compressed state, it uses matrix multiplication to reconstruct the singular value triples into parameter matrices of the original dimensions. For uncompressed layers, it leaves them as is.

10. The federated distillation method based on adaptive bidirectional distillation and soft-weighted polymerization according to claim 9, characterized in that, No. Compressed data packets uploaded by clients ;in, For the first Training loss score for each client; Indicates the first The first client's The layer model parameter matrix is ​​decomposed by SVD and truncated to rank r, resulting in a left singular vector submatrix. Indicates the first The layer model parameter matrix is ​​truncated to a singular value diagonal matrix of rank r. Indicates the first The layer model parameter matrix is ​​truncated to the right singular vector submatrix of rank r. This represents the collection of all layers in the student network that need to be SVD compressed and uploaded, representing the client's network. The server recovered on the [date]. The client trains in round t, round t... The layer parameter matrix is ; The aggregated model generates the global model in the (t+1)th round of training. Layer parameters are .

Images