Methods of data communication between servers and clients in a federated learning system

By using unlabeled pseudo-datasets for knowledge distillation and sharpness-aware minimization in federated learning, we solve the problems of low communication efficiency and model performance degradation caused by data and model heterogeneity, and achieve efficient model updates and performance improvements.

CN118862948BActive Publication Date: 2026-11-17BEIJING JIAOTONG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410786500.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-18
Publication Date
2026-11-17
Estimated Expiration
2044-06-18

AI Technical Summary

Technical Problem

Existing federated learning methods suffer from problems such as low communication efficiency, degraded model performance, susceptibility to local minima, and low quality of aggregated knowledge when dealing with heterogeneous data and models.

Method used

Knowledge distillation is performed using an unlabeled pseudo dataset. Combined with a sharpness-aware minimization method, the model update process is optimized by generating and weighting local knowledge. The unlabeled pseudo dataset is used as a public dataset for knowledge distillation to generate high-quality global knowledge, and the data is aggregated by assigning weights based on sample quality.

Benefits of technology

It improves the convergence speed and generalization ability of the model under heterogeneous data, reduces communication costs and model transmission requirements, and enhances model performance and training efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118862948B_ABST
    Figure CN118862948B_ABST
Patent Text Reader

Abstract

This invention provides a method for data communication between a server and clients in a federated learning system. The method includes: the server generating a globally shared unlabeled pseudo-dataset and distributing it to each client; the clients optimizing their local models using a sharpness-aware minimization method to obtain optimized local models; the clients generating local knowledge using the local dataset and the unlabeled pseudo-dataset; the server weighting the logits in each client's local knowledge according to weight coefficients to obtain global logits, and distributing the global knowledge composed of the global prototype and global logits to each client; and the clients updating their local models based on their local logits and the global logits in the global knowledge. This invention utilizes the unlabeled pseudo-dataset as a public dataset for knowledge distillation to generate personalized models that fit the local data distribution. The server aggregates local knowledge using weights to obtain high-quality global knowledge, which is used to improve the performance of local models.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and more particularly to a method for data communication between a server and a client in a federated learning system. Background Technology

[0002] Federated learning is a distributed machine learning approach designed to address challenges related to data privacy and security. Traditional centralized machine learning models typically store data centrally, but this can lead to privacy breaches and data security issues. Federated learning trains models on local devices and uploads updated model parameters to a server for aggregation, thus preventing data from leaving the local device and effectively protecting user privacy. Currently, federated learning is widely used in finance, healthcare, and industry.

[0003] However, federated learning faces the challenges of data and model heterogeneity, meaning that data distribution and model architecture may differ significantly across devices. Although federated learning typically aims to train a global model to adapt to all clients, server aggregation of diverse client models can lead to client drift in the aggregated global model, thus affecting the performance and convergence of client models.

[0004] To address this issue, PFL (Personalized Federated Learning) was developed. PFL aims to maintain data privacy while customizing a personalized model for each client to improve the model's performance on specific data distributions.

[0005] Currently, existing PFL (Progressive Functional Model) methods for addressing data heterogeneity can be broadly categorized into two types: global model personalization and client-side local model construction. The global model personalization strategy adjusts the global model through local adaptive steps to meet the specific needs of each client. For example, regularization terms are introduced during local training to reduce the difference between local and shared models, or meta-learning allows the globally shared model to adapt to local private data without exposing private data. The client-side local model construction strategy trains an independent personalized model for each client. For example, client-similarity-based clustering divides clients into multiple groups, and a model better suited to the data distribution of each group is trained separately; or knowledge distillation allows each client to obtain soft value predictions from a globally shared public dataset, and the client model is improved using the soft value predictions aggregated by the server.

[0006] The shortcomings of the PFL methods proposed in the prior art for data heterogeneity include:

[0007] (1) Existing global model personalization methods typically assume that the client and server use the same model structure. However, this assumption may not apply when dealing with heterogeneous models. In addition, the methods in this scheme all require the transmission of model parameters, which leads to communication overhead that is proportional to the model size, thus affecting communication efficiency, and there is also the risk of gradient attacks.

[0008] (2) Existing client-side learning personalized models may be affected by clustering errors and incur significant computational and communication costs. Knowledge distillation-based methods use a simple weighted average when aggregating global knowledge, which reduces the quality of the aggregated knowledge and thus affects model performance.

[0009] (3) Existing technical solutions generally use empirical risk minimization as a local optimizer. However, during the client-side model training process, it is easy to get stuck in steep local minima, resulting in overfitting. This leads to a slower convergence speed and a decrease in generalization ability of the model. Summary of the Invention

[0010] This invention provides a method for data communication between a server and a client in a federated learning system, which improves model performance and convergence while ensuring that sensitive information is not exposed.

[0011] To achieve the above objectives, the present invention adopts the following technical solution.

[0012] A method for data communication between a server and a client in a federated learning system includes:

[0013] The server generates a globally shared unlabeled pseudo dataset and distributes it to each client.

[0014] The client trains a local model using local data and employs a sharpness-aware minimization method to optimize the local model. Optimize to obtain the optimized local model ;

[0015] The client uses the local dataset and the unlabeled pseudo dataset to generate local knowledge, which includes various prototypes native to the client. The predicted values ​​(logits) of the client-side local model for unlabeled pseudo-data samples ;

[0016] The server's local prototypes in the local knowledge of each client Perform a weighted average operation to obtain the global prototype. The global logits are obtained by weighting the logits in the local knowledge of each client according to the weight coefficient. It will be generated by the global prototype. and global logits The resulting global knowledge is distributed to each client.

[0017] The client uses local logits and global logits from the global database. For local models Update.

[0018] Preferably, the server generates a globally shared unlabeled pseudo-dataset and distributes the unlabeled pseudo-dataset to each client, including:

[0019] The server performs weighted calculations on a random subset of local data to obtain a globally shared pseudo-data sample. ,in From size A sample randomly selected from a subset of the data is sent to the pseudo-data sample. Add from the mean Standard deviation is Noise obtained by random sampling from a Gaussian distribution The processed pseudo-data sample is

[0020] The server will be provided with pseudo-data samples. The globally shared dataset constituted Distribute to each client.

[0021] Preferably, the client trains a local model using local data and employs a sharpness-aware minimization method on the local model. Optimize to obtain the optimized local model ,include:

[0022] Utilizing local data to create a local model on the client side. During training, the objective function of the sharpness-perceived minimization method is:

[0023]

[0024] in Represents sharpness, in formula (1) ,in For loss function, To have a disturbance radius of The maximum loss within the space, Let be the perturbation vector, and the calculation formula is: , The loss function with respect to the model parameters gradient, It is an L2 norm;

[0025] The formula for updating the local model is: ,in For learning rate, For the first Rotate local model parameters;

[0026] The objective function is then weighted and adjusted, resulting in the following objective function:

[0027]

[0028] in The weighting coefficients used to control the objective function in finding local minima and flat regions are adjusted. The calculation formula for model updates is as follows: ;

[0029] When global knowledge has not been generated in the initial round, the objective function for training the local model on the participating client is:

[0030] and For local data samples The predicted and actual values, and the objective function for training the participant model in subsequent communication rounds, are: ,in Let the mean squared error loss function be . and Clients respectively local samples Tag is Feature representation and global prototype For hyperparameters, Local data sample for the client Through feature extractor The acquired feature representation;

[0031] After training, the optimized local model is obtained. .

[0032] Preferably, the client generates local knowledge using the local dataset and the unlabeled pseudo dataset, the local knowledge including various prototypes native to the client. The predicted values ​​(logits) of the client-side local model for unlabeled pseudo-data samples ,include:

[0033] The client obtains feature representations of the local training data through the model feature extractor. ,in For the client The training data, for Sample labels These are model parameters; this is the local model. Feature extractor, through formula Calculate the average representation of each local class. That is, the local class prototype, where For the client The data set labels are Quantity;

[0034] The client outputs logits on a public dataset through a model classifier. ,in For the client The parameters are The classifier, logits, is a public dataset sample. The local model classifier outputs predicted values ​​for each category of the sample. Samples for public datasets Feature representation;

[0035] The client assigns weights based on how closely the samples in the public dataset approximate the global prototype. ,in Samples for public datasets The weight, To calculate Euclidean distance. Pseudo-labels generated for samples in a public dataset. Obtained through the local model classifier , Samples for public datasets The classification prediction probability value, through The category with the highest probability is used as a sample in the public dataset. Class tags, The pseudo-labels in the public dataset are samples Feature representation, For global knowledge, the tag is The prototype;

[0036] Average representation of each class on the client's local machine local logits The client's local knowledge constitutes the client's local knowledge, and the client will store the local knowledge and its corresponding weights. Send them to the server together;

[0037] Preferably, the server uses local prototypes from the local knowledge of each client. Perform a weighted average operation to obtain the global prototype. The global logits are obtained by weighting the logits in the local knowledge of each client according to the weight coefficient. It will be generated by the global prototype. and global logits The resulting global knowledge is distributed to each client, including:

[0038] The server receives local class prototypes from the local knowledge uploaded by all participating clients. Then, for the local prototype Perform weighted average operation , obtain the global prototype ,in To have sample labels The client, For the client The middle category is The prototype, The number of class tags;

[0039] The server performs a weighted calculation on the soft value prediction logits output by the local model based on the weight coefficients. Get global logits ,in Weight Calculated based on Euclidean distance, the smaller the distance, the higher the weight. For traversing the client weight The weights are obtained by recalculation. and They are respectively Public dataset samples In the client The weighting coefficients and logits, Size of the public dataset;

[0040] Global prototype obtained from server aggregation and global logits This constitutes global knowledge, which is then distributed to various clients.

[0041] Preferably, the client uses local logits and global logits from global knowledge. For local models Updates will be made, including:

[0042] Each client utilizes a public dataset and local logits and global logits for local models Training and updates are performed, with the objective function being: ,in The Kullback-Leibler divergence loss function is used. and Samples from public datasets Local logits and global logits, Let be the cross-entropy loss function, where and Samples from public datasets Predicted values ​​and pseudo-labels;

[0043] In the local model During the training and updating process, the parameters Perform dynamic control. The calculation formula is ,in For the current round, For the total number of rounds, It increases with the number of training rounds.

[0044] As can be seen from the technical solutions provided by the embodiments of the present invention described above, the embodiments of the present invention utilize unlabeled pseudo-datasets as a public dataset for knowledge distillation. These unlabeled data provide additional auxiliary information. Good performance can be achieved by using only a small number of data samples. The present invention evaluates the sample quality based on the samples possessed by each client and generates corresponding weights accordingly. The server uses these weights to aggregate local knowledge to obtain high-quality global knowledge, which is used to improve the performance of the local model.

[0045] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a personalized federated learning method based on knowledge distillation and sharpness perception minimization, provided for an embodiment of the present invention. Detailed Implementation

[0048] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0049] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0050] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0051] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0052] The flowchart of a method for data communication between a server and a client in a federated learning system provided by this invention is as follows: Figure 1 As shown, the specific steps are as follows:

[0053] Step S1: The server generates a globally shared unlabeled pseudo dataset and distributes the unlabeled pseudo dataset to each client.

[0054] The server performs weighted calculations on a random subset of local data to obtain a globally shared pseudo-data sample. ,in From size A sample is randomly selected from a subset of the local data. To ensure that the generated pseudo-data samples maintain a certain distance from the local data, the pseudo-data samples are further processed by adding a value from the mean of the local data. Standard deviation is Noise obtained by random sampling from a Gaussian distribution The processed pseudo-data sample is .

[0055] The server will be provided with pseudo-data samples. The globally shared dataset constituted Distribute to each client.

[0056] Step S2: The client trains a local model using local data and optimizes it using a sharpness-perceived minimization method.

[0057] Utilizing local data to create a local model on the client side. During training, the optimization objective of the sharpness-perception minimization method is:

[0058]

[0059] in Represents sharpness, in formula (1) ,in For loss function, To have a disturbance radius of The maximum loss within the space, Let be the perturbation vector, and the calculation formula is: , The loss function with respect to the model parameters gradient, It is an L2 norm.

[0060] The formula for updating the local model is: ,in For learning rate, For the first Rotate local model parameters.

[0061] As can be seen from the model update calculation formula of the local model above, the local model update is mainly guided by the perturbation vector, causing the local model to update towards a region where the loss function is flatter. Under heterogeneous data, the updated local minimum may converge to a point with a large loss value, and the convergence speed will decrease. In order to balance the magnitude of the loss value and the flatness of the region when searching for local minima, the objective function of optimization is weighted and adjusted. The adjusted objective function is:

[0062]

[0063] in The weighting coefficients used to control the objective function in finding local minima and flat regions are adjusted. The calculation formula for model updates is as follows: .

[0064] When global knowledge has not been generated in the initial round, the objective function for training the local model on the participating client is:

[0065] and For local data samples The predicted and actual values. In subsequent communication rounds, the objective function for training the participant model is: ,in Let the mean squared error loss function be . and Clients respectively local samples Tag is Feature representation and global prototype This is a hyperparameter. Local data sample for the client Through feature extractor The obtained feature representation.

[0066] Step S3: The client generates local knowledge and uploads it to the server.

[0067] The input data for this step consists of the client's local dataset and the global pseudo-data sample obtained in step S1. The output data is the local class prototype. Local logits Weights of samples in the client's public dataset .

[0068] The client obtains feature representations of local training data samples through the model feature extractor. ,in For the client The training data, for Sample labels For model parameters The feature extractor. Since each category contains multiple samples in the client's local dataset, to capture the main features of all samples in each category, a formula is used. Calculate the average representation of each local class. That is, the local class prototype, where For the client The data set labels are The quantity.

[0069] Meanwhile, the client outputs soft-value prediction logits on the public dataset through the model classifier. ,in For the client The parameters are The classifier is called `logits`. `logits` is the probability value predicted for each class of a sample obtained from the local model classifier on the client side, based on samples from the public dataset. Samples for public datasets The characteristics are represented.

[0070] Existing methods typically employ a simple weighted average approach when aggregating local logits uploaded by clients on the server side. This involves assigning equal weights to each client's logits to calculate the global logits. However, due to the uneven distribution of data among clients, some clients may have very few or no samples in a particular class, resulting in lower quality logits from these clients. When a client has a low percentage or no samples of a certain class in its local data, the local model struggles to fully learn the features of that class, making it difficult to accurately predict rare or unseen samples. Furthermore, if the server assigns the same weights to logits from different clients during aggregation, the quality of the aggregated logits may degrade, potentially even misleading model updates.

[0071] To address this issue, we propose a weighting method based on sample quality. Specifically, the client evaluates sample quality based on how closely samples in the public dataset approximate the global prototype and assigns corresponding weights. ,in Samples for public datasets The weight, To calculate Euclidean distance. Pseudo-labels generated for samples in a public dataset. Obtained through the local model classifier , Samples from public datasets The classification results predict probability values, through The category with the highest probability is used as a sample in the public dataset. The class tag. The pseudo-labels in the public dataset are samples Feature representation, For global knowledge, the tag is The prototype.

[0072] Client-side local class prototype local logits The client's local knowledge constitutes the client's local knowledge, and the client will store the local knowledge and its corresponding weights. Send them to the server together.

[0073] Step S4: The server aggregates the local knowledge of each client to obtain global knowledge, and then distributes the global knowledge to each client.

[0074] The server receives local class prototypes from the local knowledge uploaded by all participating clients. Then, for the local prototype Perform weighted average operation , obtain the global prototype ,in To have sample labels The client, For the client The middle category is The prototype, This represents the number of class tags.

[0075] The server also performs a weighted calculation on the soft value prediction logits output by the local model based on the weight coefficients. Get global logits ,in Due to weight Calculated based on Euclidean distance, the smaller the distance, the higher the weight. For traversing the client weight The weights are then calculated again. and They are respectively Public dataset samples In the client The weighting coefficients and logits, Size of the public dataset.

[0076] Global prototype obtained from server aggregation and global logits This constitutes global knowledge. The server distributes this global knowledge to each client for local model updates.

[0077] Step S5: The client updates the local model based on the received global knowledge.

[0078] The input data for this step is the client's local model. Local logits Global logits obtained by aggregation with the server The output is the updated local model.

[0079] Each client updates its local model using the public dataset and global logits. The objective function is: ,in The Kullback-Leibler divergence loss function is used. and Samples from public datasets Local logits and global logits, Let be the cross-entropy loss function, where and Samples The predicted values ​​and pseudo-labels.

[0080] Because local models perform poorly in the early stages of training, classification accuracy improves with each training epoch. In the early stages, knowledge generated from unlabeled shared data may provide incorrect guidance for model updates. To mitigate this impact, parameter optimization is used... Perform dynamic control. The calculation formula is ,in For the current round, For the total number of rounds, It increases with the number of training rounds.

[0081] In summary, this invention utilizes unlabeled pseudo-datasets as a public dataset for knowledge distillation, providing additional auxiliary information. Good performance can be achieved using only a small number of data samples.

[0082] This invention evaluates the quality of samples based on the samples possessed by each client and generates corresponding weights accordingly. The server uses these weights to aggregate local knowledge to obtain high-quality global knowledge, which is then used to improve the performance of the local model.

[0083] This invention utilizes a weighted sharpness-aware minimization method to balance the magnitude of the loss value and the flatness of the region when searching for local minima. This allows model updates to better satisfy the condition of smaller loss values ​​located in the flat region of the loss function, thereby improving the convergence speed and generalization of model training on heterogeneous data.

[0084] Compared with existing methods, this invention has advantages in both model communication efficiency and performance. First, it uses an unlabeled pseudo-dataset as a public dataset for knowledge distillation, achieving similar performance levels with fewer data samples compared to current methods based on randomly acquired public datasets. Throughout the process, no model parameters need to be transmitted; only soft value predictions and prototypes are transmitted, making communication costs independent of model size, thus reducing communication costs while maintaining performance. Second, it generates corresponding weight coefficients based on the quality of client samples to aggregate local knowledge on the client side. High-quality global knowledge is obtained to improve local model performance while avoiding the exposure of sensitive information. During model training, a weight adjustment sharpness-aware minimization method is used to balance the magnitude of the loss value and the flatness of the region when searching for local minima. This allows model updates to better satisfy the condition of smaller loss values ​​located in the flat region of the loss function, improving the convergence speed and generalization of model training under heterogeneous data.

[0085] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0086] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0087] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0088] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for data communication between a server and a client in a federated learning system, characterized in that, include: The server generates a globally shared unlabeled pseudo dataset and distributes it to each client. The client trains a local model using local data. The local model is optimized using a sharpness-aware minimization method, resulting in an optimized local model. ; The client uses the local dataset and the unlabeled pseudo-dataset to generate local knowledge, which includes various prototypes native to the client. The predicted values ​​(logits) of the client-side local model for unlabeled pseudo-data samples ; The server's local prototypes in the local knowledge of each client Perform a weighted average operation to obtain the global prototype. The global logits are obtained by weighting the logits in the local knowledge of each client according to the weight coefficient. It will be generated by the global prototype. and global logits The resulting global knowledge is distributed to each client. The client uses local logits and global logits from the global database. For local models Update; The client uses local data to train a local model, and then optimizes the local model using a sharpness-aware minimization method to obtain the optimized local model. ,include: Utilizing local data to create a local model on the client side. During training, the objective function of the sharpness-aware minimization method is: in Represents sharpness, in formula (1) ,in For loss function, To have a disturbance radius of The maximum loss within the space, Let be the perturbation vector, and the calculation formula is: , The loss function with respect to the model parameters gradient, It is an L2 norm; The formula for updating the local model is: ,in For learning rate, For the first Rotate local model parameters; The objective function is then weighted and adjusted, resulting in the following objective function: in To control the weighting coefficients used for finding local minima and flat regions in the objective function, the calculation formula for model updates is adjusted as follows: ; When global knowledge has not been generated in the initial round, the objective function for training the local model on the participating client is: ; and For local data samples The predicted and actual values, and the objective function for training the participant model in subsequent communication rounds, are: ,in Let the mean squared error loss function be . and Clients respectively local samples Tag is Feature representation and global prototype For hyperparameters, Local data sample for the client Through feature extractor The acquired feature representation; After training, the optimized local model is obtained. .

2. The method according to claim 1, characterized in that, The server generates a globally shared unlabeled pseudo-dataset and distributes it to each client, including: The server performs weighted calculations on a random subset of local data to obtain a globally shared pseudo-data sample. ,in From size A sample randomly selected from a subset of the data is used to send to the pseudo-data sample. Add from the mean Standard deviation is Noise obtained by random sampling from a Gaussian distribution The processed pseudo-data sample is The server will be provided with pseudo-data samples. The globally shared dataset constituted Distribute to each client.

3. The method according to claim 2, characterized in that, The client generates local knowledge using the local dataset and the unlabeled pseudo dataset, the local knowledge including various prototypes native to the client. The predicted values ​​(logits) of the client-side local model for unlabeled pseudo-data samples ,include: The client obtains feature representations of the local training data through the model feature extractor. ,in For the client The training data, for Sample labels These are model parameters; this is the local model. Feature extractor, through formula Calculate the average representation of each local class. That is, the local class prototype, where For the client The data set labels are Quantity; The client outputs logits on a public dataset through a model classifier. ,in For the client The parameters are The classifier, logits, is a public dataset sample. The local model classifier outputs predicted values ​​for each category of the sample. Samples for public datasets Feature representation; The client assigns weights based on the similarity between samples in the public dataset and the global prototype. ,in Samples for public datasets The weight, To calculate Euclidean distance, Pseudo-labels generated for samples in a public dataset. Obtained through the local model classifier , Samples for public datasets The classification prediction probability value, through The category with the highest probability is used as a sample in the public dataset. Class tags, The pseudo-labels in the public dataset are samples Feature representation, For global knowledge, the tag is The prototype; Average representation of each class on the client's local machine local logits The client's local knowledge constitutes the client's local knowledge, and the client will store the local knowledge and its corresponding weights. Send them to the server together.

4. The method according to claim 3, characterized in that, The server provides local prototypes in the local knowledge of each client. Perform a weighted average operation to obtain the global prototype. The global logits are obtained by weighting the logits in the local knowledge of each client according to the weight coefficient. It will be generated by the global prototype. and global logits The resulting global knowledge is distributed to each client, including: The server receives local class prototypes from the local knowledge uploaded by all participating clients. Then, for the local prototype Perform weighted average operation To obtain the global prototype ,in To have sample labels The client, For the client The middle category is The prototype, The number of class tags; The server performs a weighted calculation on the soft value prediction logits output by the local model based on the weight coefficients. Get global logits ,in Weight Calculated based on Euclidean distance, the smaller the distance, the higher the weight. For traversing the client weight The weights are obtained by recalculation. and They are respectively Public dataset samples In the client The weighting coefficients and logits, Size of the public dataset; Global prototype obtained from server aggregation and global logits This constitutes global knowledge, which is then distributed to various clients.

5. The method according to claim 4, characterized in that, The client uses local logits and global logits from global knowledge. For local models Updates will be made, including: Each client utilizes a public dataset and local logits and global logits for local models Training and updates are performed, with the objective function being: ,in The Kullback-Leibler divergence loss function is used. and Samples from public datasets Local logits and global logits, Let be the cross-entropy loss function, where and Samples from public datasets Predicted values ​​and pseudo-labels; In the local model During the training and updating process, the parameters Perform dynamic control. The calculation formula is ,in For the current round, For the total number of rounds, It increases with the number of training rounds.

Citation Information

Patent Citations

  • Deep learning edge computing system framework based on industrial Internet of Things platform

    CN114169506A

  • Open-end fund reserved cash proportion prediction method, device and system

    CN114757781A