A decentralized knowledge distillation method and system oriented to data and model heterogeneity

Abstract

The application discloses a decentralized knowledge distillation method and system for data and model heterogeneity. The method comprises: supervised training on local private data; constructing a regression target based on the outputs of local models and neighbor models on public data batches; training an online graph structure weight learning network to output complementary weights for each neighbor client; using a binary classifier to output a sample-level effectiveness score representing the similarity between public samples and local data distribution; calculating sample-level fusion weights based on the complementary weights and the score, constructing teacher soft labels accordingly, weighting the sample-by-sample distillation loss with the sum of the fusion weights of each sample on all neighbors as the sample importance weight, and updating the local model; training the binary classifier; and iteratively performing the above steps. The application can identify and utilize the complementary knowledge of neighbor clients and suppress sample-level invalid knowledge, thereby improving the convergence performance and model accuracy of decentralized knowledge distillation in non-independent and identically distributed scenarios.

Description

Technical Field

[0001] This application relates to the field of distributed machine learning technology, and in particular to a decentralized knowledge distillation method and system for heterogeneous data and models. Background Technology

[0002] With the development of artificial intelligence technology, machine learning models have achieved remarkable results in fields such as image recognition and natural language processing. However, traditional centralized training requires all data to be aggregated on a central server, which not only incurs significant communication overhead but also poses the risk of data privacy breaches. Federated learning, as a distributed machine learning paradigm, allows multiple clients to collaboratively train models without sharing local data, providing a way to alleviate data silos and privacy protection issues.

[0003] In federated learning, exchanging and aggregating model parameters is a common collaborative approach, such as the FedAvg federated averaging algorithm. However, parameter aggregation requires all clients to use the same model architecture, which is difficult to apply in heterogeneous device scenarios; in addition, directly exchanging model parameters may leak information from the training data, posing a privacy risk.

[0004] Decentralized Knowledge Distillation (DKD) is an emerging federated learning paradigm. Unlike parameter aggregation, DKD achieves knowledge transfer between clients by exchanging the model's output (soft labels / logic values) on a public dataset, rather than the model parameters themselves. This approach has two advantages: (1) it supports heterogeneous models, allowing different clients to use different neural network architectures, such as ResNet18 and ResNet34; (2) it enhances the privacy protection of model parameters by not directly exposing model parameters and raw data. In addition, decentralized architectures, compared to centralized architectures, also have the advantages of no central server communication bottlenecks, resistance to single points of failure, and better scalability; information flows along the graph structure, and different topologies such as circular, random, community, and fully connected determine the information propagation speed, redundancy, and final convergence behavior, making the topology itself a designable degree of freedom.

[0005] In the DKD framework, each client maintains a local model, which is trained under supervision on local private data, while simultaneously performing knowledge distillation with other clients using a public dataset. For each public data sample, the client calculates a weighted average of the soft labels from neighboring clients as the teacher features, and then minimizes the KL divergence between its own output and the teacher features to achieve cross-client knowledge transfer.

[0006] Existing decentralized knowledge distillation solutions mainly improve upon these solutions from two perspectives: client-side weight calculation and knowledge aggregation strategies. (1) Standard DKD Equal-Weight Distillation: The original decentralized knowledge distillation scheme adopts an aggregation strategy that assigns equal weights to all neighbor features. In each round of distillation, the soft labels of all neighbor clients are averaged with equal weights to form teacher features. However, in the case of heterogeneous data (non-independent and identically distributed) scenarios, the data distribution of different clients varies greatly, and the equal-weight averaging strategy cannot distinguish the quality of knowledge of different clients, resulting in low distillation efficiency.

[0007] (2) FedIC (client weights based on mutual information): The weight of each neighbor is obtained by normalizing the inner product of its logical value and the learnable vector by sigmoid and L1.

[0008] (3) WSLD (Weighted Soft Label Distillation): Weights are assigned by evaluating the performance of the client model on public data, with higher-performing clients receiving higher aggregate weights. This method analyzes soft label distillation from the perspective of bias-variance tradeoff. For each training sample, sample-level weights are constructed based on the difference in predictions between the student model and the teacher model on the true category, and these weights are applied to the knowledge distillation loss.

[0009] (4) CAMKD (Category-Aware Multi-Teacher Knowledge Distillation): The distillation process is optimized by calculating weights at the data category level, and data category distribution information is incorporated into the aggregate weights.

[0010] However, the aforementioned existing technologies still have the following shortcomings: First, performance degradation occurs in scenarios with non-independent and identically distributed data. When client data distribution is highly heterogeneous, the performance of the above weight optimization scheme degrades significantly.

[0011] Second, it neglects the role of complementary information in distributed and highly heterogeneous environments. When the local data categories of a client are highly unbalanced, knowledge exchange with complementary neighbors can fill the knowledge gaps in missing categories. Neighbors may possess category knowledge that the local client does not have; effectively identifying such neighbors can accelerate global convergence and improve the final convergence accuracy. Although some methods introduce complementary information through category-level weighting, they lack an evaluation of the effectiveness of neighbor knowledge, which may introduce training noise or unreliable predictions into the distillation process.

[0012] Third, the differences in sample-level knowledge quality are ignored. The reliability of knowledge from the same neighbor client to different public samples varies significantly: for samples fully covered by its local data, the model output is usually reliable; however, for samples with missing local data or large distribution differences, the model prediction may have high uncertainty, or even produce misleading soft labels. Existing solutions treat teacher outputs equally for all samples, and low-quality knowledge can interfere with student model updates. While weighting methods such as WSLD perform weighting at the sample level, their weights characterize the bias-variance tradeoff of a single sample within a centralized teacher-student distillation framework. They rely on publicly labeled datasets to evaluate sample quality, failing to determine whether different neighbor clients possess complementary knowledge, and unable to explicitly model the knowledge effectiveness of neighbor clients on specific public samples in non-independent, identically distributed scenarios. This makes it difficult to address the coupled issues of "which neighbor to learn from" and "whether the current sample is worth learning" in decentralized knowledge distillation. Summary of the Invention

[0013] Based on this, the purpose of this invention is to address the aforementioned problems in the prior art by providing a decentralized knowledge distillation method and system for heterogeneous data and models. This method aims to solve the technical problem of how to identify and utilize complementary knowledge between neighboring clients while suppressing the interference of sample-level invalid knowledge on the distillation process in decentralized scenarios with highly heterogeneous data and models, thereby improving the convergence performance and model accuracy of decentralized knowledge distillation.

[0014] To achieve the aforementioned objectives, this application provides a decentralized knowledge distillation method for heterogeneous data and models. The method is applied to a decentralized communication topology consisting of multiple clients sharing a common dataset, each client possessing a local private dataset and a local model. The method includes: S1. Sample each client's local private dataset for its own training batch, calculate the supervised loss of that training batch and the public dataset to update the local model; S2. Perform forward inference on the current batch of public data for both the local model and the local models of neighboring clients to obtain their respective output logical values; construct a regression target based on the output logical values, and train an online graph structure weight learning network to learn the complementary weights between clients; S3. Use a binary classifier to output a sample-level validity score for each public data sample. The sample-level validity score represents the degree of similarity between the public data sample and the local data distribution of the corresponding neighboring client. S4. For each public data sample, calculate the sample-level fusion weight corresponding to each neighboring client based on the complementarity weight and sample-level validity score; weight and aggregate the outputs of each neighboring client on the public data sample according to the sample-level fusion weight to construct teacher soft labels; calculate the sample-by-sample distillation loss between the output of the local model on the public data sample and the teacher soft labels; use the sum of the sample-level fusion weights of the public data sample on all neighboring clients as the sample importance weight to weight and sum the sample-by-sample distillation losses of each public data sample in the current batch to obtain the weighted distillation loss; and update the local model according to the weighted distillation loss. S5. Train and update the binary classifier based on local private data and public data; S6. Iterate through the above steps until the preset training termination conditions are met.

[0015] Preferably, the graph structure weight learning network includes a first embedding layer, a second embedding layer, a first linear transformation layer, a second linear transformation layer, and an activation function layer; the first embedding layer maps the index of the local client to a first d-dimensional embedding vector, and the second embedding layer maps the index of each neighboring client to a second d-dimensional embedding vector, where d is a positive integer; the embedding vector of the local client in the first d-dimensional embedding vector is transformed by the first linear transformation layer and processed by the activation function to become a query vector, and each of the second d-dimensional embedding vectors is transformed by the second linear transformation layer and processed by the activation function to become a key vector; the graph structure weight learning network obtains the initial weights corresponding to each neighboring client by performing a dot product operation between the query vector and each of the key vectors, and the initial weights are post-processed to obtain the complementary weights.

[0016] Preferably, the complementary weights satisfy nonnegativity constraints and normalization constraints, and the post-processing includes mask self-connection processing, offset processing, and row normalization processing.

[0017] Preferably, the construction of the regression target includes: averaging the output logical values ​​of the local client and the output logical values ​​of each neighboring client on the current public data batch to obtain the local average output vector and the average output vector of each neighbor; calculating the cosine similarity between the local average output vector and each of the neighboring average output vectors, and subtracting the cosine similarity from 1 to obtain the corresponding cosine distance; normalizing each of the cosine distances to obtain the regression target; and the graph structure weight learning network is trained online with the goal of minimizing the mean squared error loss between its output complementary weights and the regression target.

[0018] Preferably, the training of the binary classifier specifically includes: labeling local private data as a first category and public data as a second category, mixing them to form binary classification training data; performing dynamic data augmentation on the binary classification training data; training the binary classifier with binary cross-entropy loss, so that the binary classifier outputs the probability that the input sample belongs to the local data distribution, and the probability is used as the sample-level effectiveness score.

[0019] Preferably, the binary classifier is a convolutional neural network, comprising a feature extractor part and a classifier part connected in sequence; the feature extractor part comprises two levels of convolutional layers, activation layers and max pooling layers connected in sequence, with the last max pooling layer followed by an average pooling layer; the classifier part comprises an unfolded layer, a first fully connected layer, a ReLU activation layer, a second fully connected layer and a Sigmoid activation layer connected in sequence, used to output the probability with a value range of (0,1).

[0020] Preferably, for local client i, neighboring client k of local client i, and public data sample x, the sample-level fusion weights are... satisfy:

[0021] in, The complementary weights corresponding to neighbor client k. The binary classifier of neighbor client k outputs a sample-level validity score for the public data sample x. The importance weight of the samples satisfy:

[0022] in, Let i be the set of neighboring clients.

[0023] Preferably, the construction of the teacher soft label includes: for a public data sample x, using the proportion of the sample-level fusion weight corresponding to each neighbor client to the sum of the sample-level fusion weights corresponding to all neighbor clients as a weighting coefficient, weighting and summing the output logical values ​​of each neighbor client on the public data sample x to obtain an aggregated output; and performing softmax normalization on the aggregated output to obtain the teacher soft label.

[0024] Preferably, the sample-by-sample distillation loss is an introduced temperature coefficient. KL divergence loss ; The weighted distillation loss is the sum of the weighted distillation losses of all common data samples within the current batch, where the weighted distillation loss of common data sample x is... satisfy:

[0025] Where B represents the number of public data samples in the current batch, and the contribution of each public data sample to the local model gradient update is: The importance weight is proportional to the weight of the public data sample.

[0026] To achieve the purpose of the invention, a second aspect of this application provides a decentralized knowledge distillation system for heterogeneous data and models, applying the decentralized knowledge distillation method for heterogeneous data and models described above. The system is deployed on clients within a centralized communication topology composed of several clients. The system includes: The local training module is used to sample training batches from each client's local private dataset and calculate supervised loss to update the local model; The complementary weight learning module is used to perform forward inference on the current batch of public data by the local model and the local models of neighboring clients respectively, and obtain their respective output logical values; a regression target is constructed based on the output logical values, and an online graph structure weight learning network is trained to learn the complementary weights between the clients. The validity assessment module is used to output a sample-level validity score for each public data sample using a binary classifier. The sample-level validity score characterizes the similarity between the public data sample and the local data distribution of the corresponding neighboring clients. The fusion weighted distillation module is used to calculate the sample-level fusion weights corresponding to each neighboring client for each public data sample based on the complementary weights; to perform weighted aggregation of the outputs of each neighboring client on the public data sample according to the sample-level fusion weights to construct teacher soft labels; to calculate the sample-by-sample distillation loss between the output of the local model on the public data sample and the teacher soft labels; to perform weighted summation of the sample-level fusion weights of the public data sample on all neighboring clients as the sample importance weights to obtain the weighted distillation loss; and to update the local model according to the weighted distillation loss. A binary classifier training module is used to train and update the binary classifier based on local private data and public data. The iteration control module is used to control the iterative operation of the above modules until the preset training termination conditions are met.

[0027] Compared with the prior art, the beneficial effects of this invention are: This invention employs a graph-structured weighted learning network to learn complementary weights from neighboring clients, filling knowledge gaps in missing categories, accelerating convergence, and improving final accuracy. By training a binary classifier with local private and public data, it evaluates the effectiveness of neighboring clients' knowledge of current public samples at the sample granularity, suppressing misleading predictions by neighbors for locally uncovered samples without requiring real labels from public data. Sample-level fusion weights are calculated using complementary weights and sample-level effectiveness scores, ensuring that only neighbors with complementary distributions and genuine training experience on the current samples can influence distillation. This also affects the construction of teacher soft labels and the weighting of distillation loss, forming a closed loop of "knowledge quality perception—reliable knowledge fusion—weighted gradient optimization." This extends weight information from forward propagation signal fusion to backward propagation gradient modulation, thereby increasing the contribution of high-value complementary knowledge and suppressing interference from unreliable knowledge. This invention can identify and utilize the complementary knowledge of neighboring clients and suppress sample-level invalid knowledge without sharing original data or relying on real labels from public data, improving the convergence performance and model accuracy of decentralized knowledge distillation in non-independent, identically distributed scenarios. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the decentralized knowledge distillation method provided in the embodiments of this application; Figure 2 A schematic diagram of the overall framework for decentralized knowledge distillation provided in the embodiments of this application; Figure 3 This is a schematic diagram of the graph structure weight learning network architecture provided in the embodiments of this application; Figure 4 This is a schematic diagram of a binary classifier architecture provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. The following embodiments are used to illustrate the invention but are not intended to limit its scope.

[0030] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0031] Example 1 Embodiment 1 of this application provides a decentralized knowledge distillation method for heterogeneous data and models. This method is applied to a communication graph consisting of N clients (N being an integer greater than 1), each client following a preset decentralized communication topology (e.g., ring topology, random topology, community topology, or fully connected topology). Each client communicates directly only with its neighboring clients in the communication graph, without a central server. For client i, its set of neighboring clients is denoted as […]. .

[0032] Each client i maintains: (1) a local private dataset (2) Local private datasets of different clients are allowed to be non-independent and identically distributed; The local model structure of different clients can be the same or different. For example, some clients use ResNet18 and some clients use ResNet34; (3) The graph structure weight learning network is denoted as (4) The binary classifier is denoted as The decentralized communication topology also includes a public dataset accessible to all clients. Public datasets do not require real labels.

[0033] In each round of communication iteration, client i obtains from neighboring client k: the output logical value (soft label) of neighboring client k's local model on the current batch of common data, and the sample-level validity score of neighboring client k's binary classifier for each common sample output. Each client only exchanges the above model outputs and scores, and does not exchange the original data and model parameters.

[0034] like Figure 1 As shown, the method includes: S1. Sample each client's local private dataset for its own training batch, calculate the supervised loss of that training batch and the public dataset to update the local model; S2. Perform forward inference on the current batch of public data for both the local model and the local models of neighboring clients to obtain their respective output logical values; construct a regression target based on the output logical values, and train an online graph structure weight learning network to learn the complementary weights between clients; S3. Use a binary classifier to output a sample-level validity score for each public data sample. The sample-level validity score represents the degree of similarity between the public data sample and the local data distribution of the corresponding neighboring client. S4. For each public data sample, calculate the sample-level fusion weight corresponding to each neighboring client based on the complementarity weight and sample-level validity score; weight and aggregate the outputs of each neighboring client on the public data sample according to the sample-level fusion weight to construct teacher soft labels; calculate the sample-by-sample distillation loss between the output of the local model on the public data sample and the teacher soft labels; use the sum of the sample-level fusion weights of the public data sample on all neighboring clients as the sample importance weight to weight and sum the sample-by-sample distillation losses of each public data sample in the current batch to obtain the weighted distillation loss; and update the local model according to the weighted distillation loss. S5. Train and update the binary classifier based on local private data and public data; S6. Iterate through the above steps until the preset training termination conditions are met.

[0035] like Figure 2 As shown, the decentralized knowledge distillation method (HA-DKD) provided in this application includes a system initialization phase and a main training loop phase.

[0036] System initialization phase: Initialize the local models, graph structure weight learning networks, binary classifiers, and temperature coefficient τ for all clients.

[0037] During the main training loop phase, each round of communication iteration includes the above steps S1-S5, specifically: Step S1 is the local training step. Each client samples a small batch from its local private dataset, calculates the cross-entropy loss between the training batch and the public dataset, and updates the local model parameters through backpropagation.

[0038] Step S2 is the graph weight calculation and update step. Each client performs forward inference on the current batch of public data to obtain the average output logical value vector. After collecting the average output logical values ​​of all neighboring clients, the cosine distance between the local client and each neighboring client is calculated. The cosine distance vector is normalized and used as the regression target of the graph structure weight learning network. The graph structure weight learning network is trained with the neighboring client index as input and the mean squared error loss is used to update its parameters through backpropagation. The output of the graph structure weight learning network is post-processed by masked self-connection, offset, and row normalization to obtain client-level complementary weights. Among them, the post-processing is offset first, then masked self-connection, and finally row normalization. The offset adds a uniform basic weight to each candidate connection to avoid all weights being close to zero and to reduce the extreme differences between different edges. The masked self-connection resets the self-connection weights to zero.

[0039] Step S3 is the binary classifier evaluation step. Each client uses its trained binary classifier to reason about each common sample, outputs a knowledge validity score, and provides the score to its neighboring clients.

[0040] Step S4 is the fusion weighted distillation step. For each common sample, the sample-level fusion weights on all neighbor clients are calculated and summed to obtain the sample importance weights. The fusion weights are used to perform softmax normalization weighted averaging on the neighbor output logistic values ​​to construct teacher soft labels. The sample adaptive weighted KL divergence loss is calculated, and the local model parameters are updated through backpropagation.

[0041] Step S5 is the binary classifier training and update step: Each client mixes local private data (labeled as positive samples) and public data (labeled as negative samples) to train a binary classifier to distinguish between local distributed data and external data.

[0042] Perform iterations on steps S1-S5 above until the preset training termination condition is met, such as reaching the preset number of communication rounds (iterations).

[0043] In decentralized knowledge distillation, each client needs to determine the importance of its neighboring clients in the process. This embodiment employs a graph structure weight learning network to dynamically learn the structural weights among clients. The training objective is to make the network output fit the differences in output distributions between clients. The greater the difference in output distributions, the more likely the neighboring clients are to possess category knowledge that the local clients have not yet fully learned, and the stronger the potential complementarity.

[0044] like Figure 3 As shown, the graph structure weight learning network in step S2 includes an input layer (the input is the local client's index self and the neighbor clients' indices n1-n4), a first embedding layer emb1, a second embedding layer emb2, a first linear transformation layer lin1, a second linear transformation layer lin2, and an activation function layer. The learnable embedding layers emb1 and emb2 map the client indices to d-dimensional embedding vectors, respectively. ~ and ~ lin1 and lin2 perform linear feature transformations on the embedding vectors respectively; the activation function layer provides non-linearity; lin1, lin2, and the activation function layer constitute the hidden layer, and the embeddings of each client are obtained after processing by the first hidden layer (including the first linear transformation layer lin1 and the activation function layer). ~ The embeddings of each client are obtained through processing in the second hidden layer (including the second linear transformation layer lin2 and the activation function layer). ~ The embedding of the local client is processed by the first hidden layer and used as the query vector q. The embeddings of each client are processed by the second hidden layer and used as key vectors k0~k4. The activation function layer in the first hidden layer uses the SiLU activation function, and the activation function layer in the second hidden layer uses the hyperbolic tangent activation function. The network transforms the embedding vector of the central client (local client i) into the query vector (Query), and transforms the embedding vectors of all candidate neighbor clients into key vectors (Keys). The query vector and the key vectors are then compared using the dot product (...). The algorithm dynamically evaluates the importance of each neighboring client based on its score, and outputs an adjacency matrix representing the edge weight vector (adj) of local adjacency relationships. ~ .

[0045] Specifically, for local client i and its neighboring client k, the average output logistic values ​​of the local models of client i and client k are calculated on the current batch of public data to obtain their respective average output vectors; the cosine similarity between them is calculated, and the cosine distance is obtained by subtracting the cosine similarity from 1; the cosine distances are then normalized to obtain the regression target. The cosine distance reflects the degree of difference in the distribution of client outputs on public data. The greater the difference, the more likely the neighboring client is to have category knowledge that the local client has not yet fully learned, and the stronger the potential complementarity.

[0046] The training objective of a graph structure weight learning network is to minimize the mean squared error loss between the predicted adjacency weights and the aforementioned regression objective, i.e., the graph loss.

[0047] in, For the graph loss of local client i, Let i be the number of neighbors of client i. The complementary weights corresponding to neighbor client k in the graph structure weight learning network output are given; g stands for graph, indicating that these weights are graph weights. The normalized cosine distance target is calculated based on the mean of the model outputs of local client i and neighboring client k. Equation (1) drives the graph structure weight learning network to dynamically fit the cosine distance matrix between clients, that is, to capture the complementarity between clients: the larger the cosine distance (the greater the difference in output distribution), the larger the complementary weight is assigned to the neighbor. The network output is processed by masked self-connection (masking the weights corresponding to the client itself), offset and row normalization to make the obtained client-level complementary weights satisfy the non-negativity and normalization constraints.

[0048] Because it employs client-index-based embedding representations, the graph structure weight learning network has a relatively small number of parameters. In one specific embodiment, the network has approximately 5000 parameters in a 10-client scenario, introducing limited additional computational overhead.

[0049] Alternatively, the graph structure weight learning network can be replaced with other network structures such as attention networks that can output normalized weights based on client indices or client features; this application does not limit this.

[0050] In addition to client-level complementary weights, this embodiment further introduces a binary classifier to characterize the knowledge effectiveness of neighboring clients across different common samples. The graph-structured weighted learning network answers the question "Which neighbor should we learn from first?", while the binary classifier answers the question "Is the neighbor's prediction on the current sample reliable?". Even if a neighbor has high overall knowledge complementarity, its prediction may still have high uncertainty if its local data does not sufficiently cover the current sample, and therefore it should not occupy a large weight in the distillation process.

[0051] like Figure 4 As shown, each client maintains a small convolutional neural network as the binary classifier in step S3 (denoted as MiniCNN_Binary). This binary classifier consists of a feature extractor part and a classifier part connected in sequence; the feature extractor part consists of two levels of convolutional layers, activation layers and max pooling layers connected in sequence, with the last max pooling layer followed by an average pooling layer; the classifier part consists of an unfolded layer, a first fully connected layer, a ReLU activation layer, a second fully connected layer and a Sigmoid activation layer connected in sequence. The feature extractor receives 3-channel input, which passes sequentially through a 16-channel convolutional layer 1 (with ReLU activation), a max pooling layer, a 32-channel convolutional layer 2 (with ReLU activation), a max pooling layer, and an adaptive average pooling layer. The feature map is then fed into the classifier and flattened into a 512-dimensional vector. The classifier first passes through a 64-dimensional fully connected layer 1 (with ReLU activation), and then through a 1-dimensional fully connected layer 2 (with Sigmoid activation). The final output is a probability, or confidence value σ, between [0,1]. This value is used to evaluate the effectiveness of the neighboring client's knowledge of the current common sample. The closer the output is to 1, the more similar the sample is to the training data distribution of the client, and the more likely the client's prediction of the sample is based on real training experience rather than random guessing.

[0052] The training data for the binary classifier consists of a mixture of local private data (labeled as the first category, such as positive samples, indicating that the sample belongs to the local data distribution and the model has training experience with this type of sample) and public data (labeled as the second category, such as negative samples, indicating that the sample is an unknown data distribution for the local model). During training, dynamic data augmentation is applied to the training data, enabling the binary classifier to learn to distinguish between data from the local distribution and external data. In one specific embodiment, two types of data augmentation are used in the binary classifier training: 1. Random cropping: First, fill the image with 4 pixels around its edges, then randomly crop it back to its original size.

[0053] 2. Random horizontal flip: Flip the image horizontally with a default probability of 0.5.

[0054] Both types of augmentation are performed on local and public data respectively. Then, the same number of samples are taken from both local and public data. The local samples are labeled as 1 (positive samples), and the public samples are labeled as 0 (negative samples). The two types of samples are concatenated and randomly shuffled before training the binary classifier.

[0055] The binary classifier uses the standard binary cross-entropy loss function:

[0056] in, This indicates the expectation after sampling from the local dataset of client j; This indicates the expectation after sampling from a public dataset; This indicates that the binary classifier classifies the samples. The predicted output is usually the probability of belonging to the "private / positive class". This indicates the probability that a sample belongs to another "common / negative class"; Indicates the first The parameters of the client-side binary classifier; the input data of the binary classifier model are The output is the probability that this data belongs to the first category, i.e., it comes from the local data. This loss function guides the binary classifier in determining the similarity between the input sample and the local data sample.

[0057] During the inference phase, for each public sample For each neighbor client k, a binary classifier for neighbor client k outputs a validity score. It should be noted that this scoring method does not rely on the real labels of public data, overcoming the shortcomings of schemes such as WSLD that depend on public data with real labels. In one specific embodiment, the binary classifier has approximately 20,000 parameters, introducing limited additional computational overhead, making it suitable for resource-constrained edge computing scenarios.

[0058] In step S4, the graph structure weight learning network and the binary classifier evaluate the value of neighbor knowledge from two orthogonal dimensions: graph weights reflect distributional complementarity, i.e., whether the neighbor is likely to possess local missing category knowledge; binary classifier weights reflect knowledge validity, i.e., whether the neighbor has relevant training experience for this specific sample. The product of these two factors ensures that only neighbors that simultaneously satisfy distributional complementarity and possess genuine knowledge of the sample can have a substantial impact on distillation. Step S4 specifically includes the following steps: (1) Fusion weight calculation: For the k-th neighbor of client i and the common sample x of the current batch, the sample-level fusion weight is defined as:

[0059] in, The client-level complementary weights of neighbor client k to current distillation client i are learned by the graph structure weight learning network. Let x be the sample-level validity score (confidence) output by the binary classifier of client k for the common sample. The sample-level fusion weight is the product of the complementarity weight corresponding to the neighboring client and the sample-level validity score output by the binary classifier of the neighboring client for the common data sample. The product of the two reflects both the complementarity relationship between clients and the reliability of sample-level knowledge. Further, the fusion weights of the common sample x on all neighboring clients are summed to obtain the sample importance weight of the sample:

[0060] in, Let i be the set of neighboring clients. The larger the value, the more neighboring clients have high credibility and high complementary correlation with the sample, meaning the teacher signal of the sample is more reliable.

[0061] (2) Construction of teacher soft labels: When calculating the teacher signal, a weighted average based on fusion weights is used. When client i is the student end, the weighted output of neighboring client k on the common sample x is:

[0062] in, Let x represent the output of neighbor client k on the common sample x. For client i, the weighted aggregate output of its neighbors on the common sample x is:

[0063] Weighted aggregation output by temperature coefficient Scaling and softmax normalization yield the teacher's soft label (neighbor output probability distribution). In this embodiment, the temperature coefficient... .

[0064] (3) Sample Adaptive Weighted Loss: In standard knowledge distillation, under equal-weighted KL divergence loss, each sample contributes 1 / B to the gradient, where B is the current training batch size. In contrast, this embodiment uses a sample-by-sample weighted approach. First, the KL divergence (sample-by-sample distillation loss) of each common sample is calculated:

[0065] Where C represents the total number of categories. This is the output of the softmax function for the c-th category. For temperature coefficient, This represents the output logical value of the local model of client i on the public sample x.

[0066] Then use sample importance weights. We weight and normalize the loss values ​​of the common sample x to obtain the weighted distillation loss of the common sample x:

[0067] Where B is the current training batch size, normalized by dividing by the sum of the importance weights of the samples within the batch to ensure that the magnitude of the loss remains stable across different samples. The weighted distillation loss of the current batch is the sum of the weighted distillation losses of all common samples. As shown in equation (8), the contribution of each sample to the gradient is... If a certain sample This sample is much larger than other samples (multiple neighbors are highly familiar with this sample), and it dominates the gradient direction, so the model parameters preferentially adapt to high-quality knowledge with strong determinism; conversely, The influence of smaller samples (those that are generally unfamiliar to neighbors) on the gradient is automatically suppressed, avoiding uncertainty in predictions caused by a lack of relevant training data from interfering with model optimization.

[0068] It should be noted that, unlike schemes such as FedIC which only use adaptive weights to construct teacher signals, this embodiment directly multiplies the fusion weights into the KL divergence loss, extending the weight information from forward propagation signal fusion to backward propagation gradient modulation, so that high-quality samples dominate the optimization process.

[0069] Example 2 Embodiment 2 of this application, based on Embodiment 1, provides a decentralized knowledge distillation system oriented towards heterogeneous data and models. The system is deployed on clients within a centralized communication topology composed of several clients. The system includes: The local training module is used to sample training batches from each client's local private dataset, calculate the supervised loss of the training batch and the public dataset to update the local model; The complementary weight learning module is used to perform forward inference on the current batch of public data by the local model and the local models of neighboring clients respectively, and obtain their respective output logical values; a regression target is constructed based on the output logical values, and an online graph structure weight learning network is trained to learn the complementary weights between the clients. The validity assessment module is used to output a sample-level validity score for each public data sample using a binary classifier. The sample-level validity score characterizes the similarity between the public data sample and the local data distribution of the corresponding neighboring clients. The fusion weighted distillation module is used to calculate the sample-level fusion weights corresponding to each neighboring client for each public data sample based on the complementary weights and sample-level validity scores; to perform weighted aggregation of the outputs of each neighboring client on the public data sample according to the sample-level fusion weights to construct teacher soft labels; to calculate the per-sample distillation loss between the output of the local model on the public data sample and the teacher soft labels; to perform weighted summation of the per-sample distillation losses of each public data sample in the current batch using the sum of the sample-level fusion weights of the public data sample on all neighboring clients as the sample importance weight, to obtain the weighted distillation loss; and to update the local model according to the weighted distillation loss. A binary classifier training module is used to train and update the binary classifier based on local private data and public data. The iteration control module is used to control the iterative operation of the above modules until the preset training termination conditions are met.

[0070] Example 3 To verify the effectiveness of the technical solution of this application, this embodiment 3, based on the method in embodiment 1, conducts comparative experiments on the CIFAR-10 and SVHN datasets. Experimental settings: The number of clients is 10, and the non-independent and identically distributed local data is divided using a Dirichlet distribution (parameter alpha=0.5). The client models use a heterogeneous configuration of ResNet18 and ResNet34, and the communication topologies are community topology, random topology, and full-connected topology, respectively. The comparison algorithms include FedIC, WSLD, CAMKD, and the equal-weighted DKD baseline. Each algorithm is trained to convergence under the same experimental settings, and the test accuracy is used as the evaluation metric.

[0071] Table 1. Comparison experiment results on the SVHN dataset (Dirichlet alpha=0.5, 10 clients, ResNet18 / ResNet34)

[0072] Table 2 Comparison results on the CIFAR-10 dataset (Dirichlet alpha=0.5, 10 clients, ResNet18 / ResNet34)

[0073] As can be seen from Tables 1 and 2: (1) On the CIFAR-10 dataset, the embodiment of this application significantly outperforms all the comparison algorithms under all three topologies. It achieves the highest accuracy of 43.56% under the community topology, which is 15.23 percentage points higher than the equal-weight DKD baseline and 11.50 percentage points higher than the closest existing scheme FedIC. It also maintains above 41% under random topology and fully connected topology. (2) On the SVHN dataset, the embodiment of this application outperforms all the comparison algorithms under both the community topology and random topology. It improves by 19.97 percentage points under the equal-weight DKD baseline under the random topology. (3) Under the fully connected topology of the SVHN dataset, the embodiment of this application (76.28%) is slightly lower than WSLD (78.90%). The reason is that under the fully connected topology, each client has the most neighbors and the highest information redundancy. The space for correction of the distillation direction by sample weighting is relatively limited. However, the embodiment of this application is still significantly better than the equal-weight DKD baseline and FedIC. Overall, the technical solution of this application can effectively improve convergence performance and final model accuracy in decentralized scenarios with highly heterogeneous data and models, especially in scenarios with a limited number of neighbors and information flow constrained by topology.

[0074] Regarding the purpose of this invention, namely, to improve the performance of decentralized knowledge distillation in non-independent and identically distributed scenarios through heterogeneous data perception, the following alternative implementation methods also exist: (1) Using only confidence weights without graph learning: The confidence of the binary classifier is used directly as the sample weights, omitting the graph structure weight learning network. This approach reduces computational complexity, but loses the ability to model the structural relationships between clients, and its performance may degrade in highly heterogeneous scenarios.

[0075] (2) Use attention mechanism to replace graph structure weight learning network: Using multi-head attention network to learn client weights can capture more complex interaction relationships between clients, but the computational complexity increases twice with the number of clients and requires more training data, which limits its applicability in large-scale client scenarios.

[0076] (3) Using hard threshold selection instead of soft weighting: Set a threshold and only distill samples whose validity scores exceed the threshold. This scheme is computationally efficient, but it will discard some valid information, and the threshold needs to be manually adjusted, which is not flexible enough; the soft weighting method adopted in this application retains the information of all samples and automatically adjusts the contribution size through weight.

[0077] (4) Use meta-learning to optimize weights: Meta-learning is used to learn the sample weight allocation strategy in the meta-training stage, which can theoretically obtain better weight allocation, but requires additional meta-training data and computing resources.

[0078] The above alternative implementation shows that the combination of graph structure weight learning network and binary classifier in this application achieves a balance between computational efficiency, model expressive power and training stability. At the same time, the two core modules are organized in a loosely coupled manner, and each module can be replaced or upgraded independently. For example, the graph structure weight learning network can be replaced with an attention network, and the binary classifier can be replaced with other confidence estimation methods without affecting the core working mechanism.

[0079] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A decentralized knowledge distillation method for heterogeneous data and models, characterized in that, The method is applied to a decentralized communication topology consisting of multiple clients with a shared dataset, each client having a local private dataset and a local model. The method includes: S1. Sample each client's local private dataset for its own training batch, calculate the supervised loss of that training batch and the public dataset to update the local model; S2. Perform forward inference on the current batch of public data for both the local model and the local models of neighboring clients to obtain their respective output logical values; construct a regression target based on the output logical values, and train an online graph structure weight learning network to learn the complementary weights between clients; S3. Use a binary classifier to output a sample-level validity score for each public data sample. The sample-level validity score represents the degree of similarity between the public data sample and the local data distribution of the corresponding neighboring client. S4. For each public data sample, calculate the sample-level fusion weight corresponding to each neighboring client based on the complementarity weight and sample-level validity score; weight and aggregate the outputs of each neighboring client on the public data sample according to the sample-level fusion weight to construct teacher soft labels; calculate the sample-by-sample distillation loss between the output of the local model on the public data sample and the teacher soft labels; use the sum of the sample-level fusion weights of the public data sample on all neighboring clients as the sample importance weight to weight and sum the sample-by-sample distillation losses of each public data sample in the current batch to obtain the weighted distillation loss; and update the local model according to the weighted distillation loss. S5. Train and update the binary classifier based on local private data and public data; S6. Iterate through steps S1-S5 until the preset training termination condition is met.

2. The method according to claim 1, characterized in that, The graph structure weight learning network includes a first embedding layer, a second embedding layer, a first linear transformation layer, a second linear transformation layer, and an activation function layer. The first embedding layer maps the index of the local client to a first d-dimensional embedding vector, and the second embedding layer maps the index of each neighboring client to a second d-dimensional embedding vector, where d is a positive integer. The embedding vector of the local client in the first d-dimensional embedding vector is transformed by the first linear transformation layer and processed by the activation function to become the query vector, and each of the second d-dimensional embedding vectors is transformed by the second linear transformation layer and processed by the activation function to become the key vector. The graph structure weight learning network obtains the initial weights corresponding to each neighboring client by performing a dot product operation between the query vector and each of the key vectors, and the initial weights are post-processed to obtain the complementary weights.

3. The method according to claim 2, characterized in that, The complementary weights satisfy non-negativity constraints and normalization constraints, and the post-processing includes mask self-connection processing, offset processing, and row normalization processing.

4. The method according to claim 3, characterized in that, The construction of the regression objective includes: averaging the output logical values ​​of the local client and the output logical values ​​of each neighboring client on the current batch of public data to obtain the local average output vector and the average output vector of each neighboring client; calculating the cosine similarity between the local average output vector and the average output vector of each neighboring client, and subtracting the cosine similarity from 1 to obtain the corresponding cosine distance; normalizing each cosine distance to obtain the regression objective; and training the graph structure weight learning network online with the goal of minimizing the mean squared error loss between its output complementary weights and the regression objective.

5. The method according to claim 1, characterized in that, The training of the binary classifier specifically includes: labeling local private data as the first category and public data as the second category, mixing them to form binary classification training data; performing dynamic data augmentation on the binary classification training data; training the binary classifier with binary cross-entropy loss, so that the binary classifier outputs the probability that the input sample belongs to the local data distribution, and the probability is used as the sample-level effectiveness score.

6. The method according to claim 5, characterized in that, The binary classifier is a convolutional neural network, comprising a feature extractor part and a classifier part connected in sequence. The feature extractor part includes two levels of convolutional layers, activation layers, and max pooling layers connected in sequence, with the last max pooling layer followed by an average pooling layer. The classifier part includes an unfolded layer, a first fully connected layer, a ReLU activation layer, a second fully connected layer, and a Sigmoid activation layer connected in sequence, used to output the probability with a value range of (0,1).

7. The method according to claim 1, characterized in that, For local client i, neighboring client k of local client i, and public data sample x, the sample-level fusion weights satisfy: in, The complementary weights corresponding to neighbor client k. The binary classifier of neighbor client k outputs a sample-level validity score for the public data sample x. The importance weight of the samples satisfy: in, Let i be the set of neighboring clients.

8. The method according to claim 7, characterized in that, The construction of the teacher soft label includes: for a public data sample x, using the proportion of the sample-level fusion weight corresponding to each neighbor client to the sum of the sample-level fusion weights corresponding to all neighbor clients as a weighting coefficient, the output logical values ​​of each neighbor client on the public data sample x are weighted and summed to obtain an aggregated output; the aggregated output is normalized to obtain the teacher soft label.

9. The method according to claim 8, characterized in that, The sample-by-sample distillation loss is introduced by a temperature coefficient. KL divergence loss ; The weighted distillation loss is the sum of the weighted distillation losses of all common data samples within the current batch, where the weighted distillation loss of common data sample x is... satisfy: Where B represents the number of public data samples in the current batch, and the contribution of each public data sample to the local model gradient update is: The importance weight is proportional to the weight of the public data sample.

10. A decentralized knowledge distillation system for heterogeneous data and models, employing the decentralized knowledge distillation method for heterogeneous data and models as described in any one of claims 1-9, characterized in that... The system is deployed on clients within a centralized communication topology comprised of several clients. The system includes: The local training module is used to sample training batches from each client's local private dataset, calculate the supervised loss of the training batch and the public dataset to update the local model; The complementary weight learning module is used to perform forward inference on the current batch of public data by the local model and the local models of neighboring clients respectively, and obtain their respective output logical values; a regression target is constructed based on the output logical values, and an online graph structure weight learning network is trained to learn the complementary weights between the clients. The validity assessment module is used to output a sample-level validity score for each public data sample using a binary classifier. The sample-level validity score characterizes the similarity between the public data sample and the local data distribution of the corresponding neighboring clients. The fusion weighted distillation module is used to calculate the sample-level fusion weights corresponding to each neighboring client for each public data sample based on the complementary weights and sample-level validity scores; to perform weighted aggregation of the outputs of each neighboring client on the public data sample according to the sample-level fusion weights to construct teacher soft labels; to calculate the per-sample distillation loss between the output of the local model on the public data sample and the teacher soft labels; to perform weighted summation of the per-sample distillation losses of each public data sample in the current batch using the sum of the sample-level fusion weights of the public data sample on all neighboring clients as the sample importance weight, to obtain the weighted distillation loss; and to update the local model according to the weighted distillation loss. A binary classifier training module is used to train and update the binary classifier based on local private data and public data. The iteration control module is used to control the iterative operation of the above modules until the preset training termination conditions are met.

Images