Distance correction and aggregation method for heterogeneous models based on prototype federated learning

By adopting the heterogeneity model distance correction and aggregation method of prototype federated learning in federated learning, processing data and device heterogeneity, the problems of overfitting and underfitting in model training in federated learning are solved, and heterogeneity is improved Classification accuracy and recognition accuracy of qualitative image datasets.

CN118152802BActive Publication Date: 2024-11-05SHAANXI UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410258285.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-07
Publication Date
2024-11-05
Estimated Expiration
2044-03-07

AI Technical Summary

Technical Problem

The existing federated learning algorithm fails to effectively deal with data heterogeneity and device heterogeneity, resulting in overfitting and underfitting problems during model training, and the classification accuracy and recognition accuracy of heterogeneous image data sets are not high.

Method used

Using the distance correction and aggregation method of heterogeneity model based on prototype federated learning, the deep features of heterogeneous images are extracted by setting the embedding layer network on the client, the embedding vector is obtained and average clustered, and the local prototype is formed, and the server is performed. Weighted aggregation, add contrast loss correction terms to optimize local objective functions, and dynamically adjust the number of training epochs to adapt to the differences in equipment computing power.

Benefits of technology

It improves the accuracy of the federated learning model, reduces the negative impact of heterogeneity of data samples and training equipment on the model, enhances the accuracy of classification and testing, avoids overfitting and underfitting, and stabilizes the model training process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118152802B_ABST
    Figure CN118152802B_ABST
Patent Text Reader

Abstract

The present invention discloses a heterogeneous model distance correction and aggregation method based on prototype federated learning. On the client side, an embedding layer network for extracting deep features of heterogeneous images is set up, and the network is used to extract features of the input heterogeneous images and output them to a low-dimensional space to obtain an embedding vector, and then the embedding vectors in the same space are clustered on average to obtain a local prototype; on the server side, the local prototype uploaded by the client is received, and the paradigm distance between the local prototype and the global prototype of the previous round is used as the aggregation weight of the local prototype of this round, and the local prototype is weighted and aggregated to form a new global prototype, and sent to the client participating in the training for the next round of learning. The present invention can improve the accuracy of the horizontal federated training model and reduce the negative impact of the heterogeneity of data samples and training equipment on the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, relates to horizontal federated learning, and particularly to a heterogeneous model distance correction and aggregation method based on prototype federated learning. Background Art

[0002] "Data silos" reduce the availability of data in the field of machine learning. Federated Learning (FL) is a distributed machine learning training framework. During the FL training process, clients distributed in different geographic spaces can perform model training based on local data sets, and send the trained local model gradient parameters to the central server for average aggregation into global model parameters. The server then sends the global model parameters to the next round of selected clients for FL training. This distributed framework provides a feasible idea for breaking the "data silos", but it faces two major challenges, namely data heterogeneity and device heterogeneity. Based on previous studies on FL heterogeneity, this paper summarizes it in the following two points:

[0003] (1) Data heterogeneity: The data held by different clients differ in terms of data distribution, number of data samples, and quality of data samples (for example, features do not correspond to labels).

[0004] (2) Device heterogeneity: In actual applications, the computing power of the devices used by the client varies. For example, a device with less computing power requires more training epochs to achieve the same model effect as a device with better computing power.

[0005] Therefore, although the existing FL algorithms have certain advantages, they do not fully consider the negative impact of data sample and training equipment heterogeneity on the model. How to effectively process heterogeneous data, use prototypes to transfer heterogeneous model parameters, and efficiently aggregate local models are still challenges in the FL field. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a heterogeneous model distance correction and aggregation method based on prototype federated learning, in order to solve the problem of data heterogeneity in federated learning and improve the classification accuracy and recognition accuracy of heterogeneous image data sets in the federated learning process.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A distance correction and aggregation method for heterogeneous models based on prototype federated learning, including:

[0009] On the client side, an embedding layer network is set up to extract deep features of heterogeneous images. The network is used to extract features of the input heterogeneous images and output them to a low-dimensional space to obtain an embedding vector. Subsequently, the embedding vectors in the same space are clustered on average to obtain a local prototype.

[0010] On the server side, the local prototype uploaded by the client is received, and the paradigm distance between the local prototype and the global prototype of the previous round is used as the aggregation weight of the local prototype of this round. The local prototype is weightedly aggregated to form a new global prototype, which is sent to the clients participating in the training for the next round of learning.

[0011] The improvement of the client of the present invention can be further described as follows: first, an embedding layer network for extracting deep features of heterogeneous images is designed, which extracts features of the input heterogeneous image and outputs it to the low-dimensional OUTDIM space to obtain an embedding vector, and averages the embedding vectors in the same OUTDIM space to obtain a local prototype. At the same time, a correction term is added to the local objective function of FedAvg, and the correction term is changed to a local and global contrast loss function. The client performs gradient descent optimization on the improved loss function to update the local model, and retains the updated model locally, and then uploads the local prototype to the server.

[0012] Compared with the prior art, the present invention can improve the accuracy of the horizontal federated training model and reduce the negative impact of the heterogeneity of data samples and training equipment on the model. More specifically, the beneficial effects are as follows:

[0013] (1) The present invention can effectively alleviate the overfitting problem caused by device and data heterogeneity in the model training process of federated learning by adding correction terms to local objectives and functions, so that the model training loss is within a certain range, thereby making the model convergence more stable.

[0014] (2) When the degree of device and data heterogeneity is extreme, using the prototype to update the local model can improve the classification and testing accuracy of the model, especially the accuracy of heterogeneous image classification.

[0015] (3) In a multi-classification dataset, a certain correction factor can be used to correct the client's prototype, thereby stabilizing the model training and avoiding extreme values ​​during the model training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the framework of the heterogeneous model distance correction and aggregation method based on prototype federated learning of the present invention.

[0017] Figure 2 It is an embedded layer network design. DETAILED DESCRIPTION

[0018] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0019] First, the preliminary knowledge required to understand the present invention is introduced:

[0020] 1. Local Update of Federated Learning under Heterogeneous Conditions

[0021] Under the heterogeneous conditions of federated learning, the optimization objective of the FedAvg algorithm is That is, use the model parameter ω to predict the instance x. Assume that there are n clients participating in federated learning, and each client has a Local dataset Therefore, the local optimization objective of client i can be expressed as:

[0022]

[0023] Among them, ω i represents the model parameters of client i, is the loss function under supervised learning tasks (such as cross-loss entropy function, etc.), n represents the number of clients participating in federated learning, and N is the sum of all n client data instances. FedAvg is a classic federated model aggregation algorithm. The gradient transmission parameter is ω i The local model can integrate features from multiple data sources to some extent, alleviating communication pressure. However, the data distribution and quality of different users vary greatly, and each user's model parameters are only applicable to their own data. The transfer of gradient parameters and simple linear local model aggregation methods will aggravate the "drift" between the local model and the global model, ultimately affecting the accuracy of the prediction.

[0024] 2. Global Update of Federated Learning under Heterogeneous Conditions

[0025] In the traditional federated learning process, the local gradient parameters are set on the server side. Aggregate into the global gradient parameter ω t+1 ,Right now

[0026] 3. Prototype Federated Learning

[0027] Compared with the "data-hungry" mode in which computers need to learn features from a large amount of data, research shows that humans can quickly abstract categories and classify new unlabeled samples based on a small number of samples with known labels. This way of imitating human abstract category learning can be defined as prototype learning. The FedProto algorithm borrows the concept of prototype to represent a class, and applies the method of updating the local model with the prototype to the scenario of heterogeneous federated learning. When the client has fewer training data samples and a high degree of heterogeneity, the FedProto algorithm uses embedding vectors to obtain local prototypes to achieve collaborative updates of federated learning. In addition, the algorithm proposes to use prototype parameters to update the local model. On the basis of the local objective function of the FedAvg algorithm, the paradigm distance between the local and global prototypes is added to allow the local prototype to align with its local model, and the above heterogeneity problem is solved by minimizing the sum of the losses of all client local learning tasks. For client i, the optimization objective in the FedProto algorithm is as follows:

[0028]

[0029] The character c represents the prototype, and k represents the prototype category. Assuming that there are K types of client prototype categories participating in federated learning (k∈[1,K]), then represents the kth local prototype of client i in the tth round of federated learning. is the loss function of the local and global prototypes. The FedProto algorithm replaces the gradient transfer method with prototype transfer, that is, client i transfers the local prototype Upload to the server, the server collects and averages the local prototypes to get the global prototype c (t+1,k) , and send it to client i for update. Its prototype aggregation formula is:

[0030]

[0031] Among them, N k is the number of clients with the k-th prototype, |D i,k | is the number of samples of client i about the k-th prototype, |S k | represents the size of the sample set with the k-th prototype.

[0032] Based on the above preliminary knowledge, refer to Figure 1 The present invention is a distance correction and aggregation method for heterogeneous models based on prototype federated learning, including improvements in processing methods on the client and the server, specifically:

[0033] On the client side, an embedding layer network is set up to extract deep features of heterogeneous images. Local heterogeneous image samples are input into the embedding layer network, features are extracted and output to a low-dimensional space to obtain an embedding vector. Subsequently, in order to solve data heterogeneity, the embedding vectors in the same space are clustered on average to obtain local prototypes.

[0034] On the server side, the local prototype uploaded by the client is received, and the paradigm distance between the local prototype and the global prototype of the previous round is used as the aggregation weight of the local prototype of this round. The local prototype is weightedly aggregated to form a new global prototype, which is sent to the clients participating in the training for the next round of learning and training.

[0035] In an embodiment of the present invention, the above embedding vector is obtained as follows: the local prototype is the mean of the embedding vector, which can be considered as the center point of the embedding vector. In particular, due to the heterogeneity of client data samples, for client i, the local prototype is not necessarily the only center point in the embedding vector space. The same client has multiple local prototypes to represent the categories of different embedded samples. In the tth round of federated learning, the client inputs the sample to the embedding layer to extract the image features, obtains the embedding vector, and then takes the mean of the embedding vector to obtain the local prototype. The obtained local prototype can be expressed as:

[0036]

[0037] in, represents the local prototype of the kth class of client i in the tth round of global communication of federated learning. i represents the heterogeneous image data sample set owned by client i, y i It means that client i has a data sample set x l The corresponding data label, S i,k represents the heterogeneous image sample set representing the local prototype of the k-th class of client i, |S i,k | indicates S i,k Size, g φ (x i ) represents x i , i∈[1,n], n is the number of clients participating in the training, k∈[1,K], K is the number of local prototype categories sent by each client.

[0038] In an embodiment of the present invention, a structure of an embedded layer network is as follows: Figure 2As shown, the embedding layer network is responsible for extracting the deep features of the client image data and obtaining the corresponding embedding vector. In the present invention, the embedding layer network design includes 5 convolutional layers, 3 pooling layers and 2 fully connected layers. The convolutional layer is used to compress high-dimensional images and extract input image features. The pooling layer is used to reduce redundant features and improve convolution efficiency. The fully connected layer is used to flatten the extracted features and classify the output. Since the pixel of the omniglot grayscale image of the data set in this experiment is 105×105, the first layer of convolution kernels is set to 16, so the number of input channels of convolution layer 1 is set to 1 and the number of output channels is set to 16. The size of the feature map obtained after the convolution layer is 105×105. Convolution layer 2 has 16 input channels and an output of 32. Since a pooling layer is also defined after the convolution layer, the size of the feature map is 52×52. After five convolutions and three poolings, the original image is directly input into the fully connected layer. At this time, the convolution layer needs to be expanded into a 1-dimensional input to the fully connected layer 1. The fully connected layer 1 requires 13×13×64=10816 neurons to receive the expanded feature map, that is, the input of the fully connected layer 1 is 10816. The fully connected layer 2 is responsible for outputting the activated information from the fully connected layer 1 into the embedding space of the OUTDIM dimension to form an embedded vector.

[0039] The above processing on the client side can be implemented based on the framework of the FedAvg algorithm, and some improvements are made to it. Specifically, the improvement mainly adds a correction term to the local objective function of the existing FedAvg algorithm. The correction term is the comparison loss function between the local prototype and the issued global prototype. The local model is updated by gradient descent optimization based on the loss function of the improved FedAvg algorithm, and the updated model is retained locally, and then the local prototype is uploaded to the server. In the embodiment of the present invention, the above comparison loss function is expressed as follows:

[0040]

[0041] The client completes the local update by setting optimization constraints on the local objective function.

[0042] In an embodiment of the present invention, the method for establishing the local objective function is as follows: the optimization objective of the FedMPD algorithm is It is defined as two parts, one of which is the supervised learning task loss Part of it is the contrast loss correction term Therefore, on the client side, the local optimization objective based on the prototype federated learning FedMPD algorithm can be defined as follows:

[0043]

[0044] Among them, ω i represents the model parameters of client i, Represents ω i The optimization goal of , which can also be called the loss function, Indicates that x i Input model ω i The prediction results obtained, N k represents the total number of k-th category data, represents the k-th data set of client i, express The size of c (t,k) represents the global prototype of the kth class in the tth round of communication. represents the supervised learning task loss, represents the correction term of the k-th prototype (i.e., the aforementioned contrast loss function), is a local heterogeneous image dataset, N is the total number of heterogeneous data images of all n clients, λ is a correction term parameter, and is a predefined constant. The present invention introduces a correction term parameter λ into the loss function to control the degree of influence of the global correction term on the training model. The FedMPD algorithm considers alleviating the deviation between local and global prototypes, while making prototypes that are close but belong to different categories distinguishable. Therefore, in addition to the supervised learning loss function task, a correction term for contrast loss is set, so that local prototypes belonging to the same category approach the global prototype, and local prototypes of different categories are far away from the global prototype, and the distance away is not less than r, where the distance r can be set as the distance between the kth global prototype and its nearest (k+1th category) global prototype. The mathematical expression for defining r is: r=mind(c (t,k) ,c (t,k+1) ). The present invention uses the current k-th local prototype as the positive label prototype, and uses the global prototype with a distance r from it as the easily confused negative label prototype to define the contrast loss function of the client correction term, that is:

[0045]

[0046] In the above, the first term is to encourage the local prototype of the kth class to be close to the global prototype, and c (t,l) It is represented as other global prototypes that are not of the k-th class. When the local prototype is not of the k-th class, the latter term aims to limit the distance between the prototype of the non-k-th class and the global prototype of the nearest (for example, k+1) class. d() is the function for solving the paradigm distance. The contrast loss is used to measure the difference between local prototypes and global prototypes in multi-classification problems, which facilitates embedding vector clustering and improves the accuracy of local prototypes. However, due to the heterogeneity of the underlying samples, for data of a small number of heterogeneous samples, too many local class prototype updates will still cause differences between local and global prototypes that originally belonged to the same class. In order to limit this difference, the present invention proposes constraints to deeply solve the problem of data heterogeneity.

[0047] In an embodiment of the present invention, the method for establishing the constraint conditions for optimizing the local objective function is as follows: in the scenario of federated learning of heterogeneous data, the difference in computing power of the devices will increase the deviation between the local and global models to a certain extent. In addition, the number of local training epochs of the device in the FedProto algorithm is fixed. For local devices with uneven computing power, the fixed number of epochs will cause two situations: (1) Devices with low computing power will experience underfitting after a fixed number of epochs. (2) Devices with better computing power will experience overfitting after a fixed number of epochs. In response to these two problems, on the client side, the present invention sets the optimization constraint conditions for the local objective function as follows:

[0048]

[0049] Among them, α is the marginal function of the embedding vector, and the specific expression is α = d(g φ (x i ),g φ (x j )).

[0050] In an embodiment of the present invention, the global update method is as follows: in the server prototype aggregation stage, the present invention uses the distance between the local prototype and the global prototype to set the aggregation weight, and the local prototype closer to the global prototype has a greater aggregation weight. This method can ensure that the local prototype closer to the global prototype has a greater influence in the aggregation process. Specifically, on the server side, high-quality local prototypes are screened out by setting a threshold r and are given a greater weight for aggregation:

[0051]

[0052] The process of the present invention can be described as follows:

[0053] For round t of federated learning on client i:

[0054] Input: Client local dataset Global prototype c (t,k) , the number of clients i=1,…,n, the contrast loss correction parameter λ, the local model of the previous round

[0055] Output: Client's local model for round t

[0056] 1. The server randomly selects n clients to participate in training.

[0057] 2. The server sends the global prototype c (t,k) to the selected client.

[0058] 3. The client accepts the global prototype.

[0059] 4. Clients run in parallel:

[0060] 5. Update local prototype

[0061] 6.

[0062] 7. Calculate the loss

[0063]

[0064] 8. Gradient descent updates the local model

[0065]

[0066] 9. The local model of the t+1th round of global communication Reserved to client i.

[0067] 10. Local prototype Upload to server

[0068] 11. The server performs weighted aggregation on the local prototypes to obtain the global prototype c (t+1,k)

[0069]

[0070] 12. The server sends the local prototype c of round t+1 (t+1,k)

[0071] The difference between the local prototype and the local model mentioned above is that the local prototype is uploaded to the server to obtain the global prototype, and the global prototype is sent to the client to compare the loss with the local prototype to obtain the correction term, which is used to correct the local objective function to obtain the local model. The local model is saved locally and will not be uploaded to the server.

[0072] By improving and updating the local local model and the global model on the client side, an optimization method for the FedMPD algorithm is proposed to deal with the inherent device and data heterogeneity in federated learning by measuring the prototype distance. The algorithm allows local execution of variable workload to alleviate data and device heterogeneity, and relies on contrast loss correction terms to prevent the "drift" of local prototypes. On the server side, in the process of local prototype aggregation, a distance-based weighted aggregation method is added to improve the traditional average aggregation method, so that under the assumption of device heterogeneity, the method of allowing the number of local epochs to be variable can help the model converge more stably.

[0073] The local model learned by the present invention can be used for image recognition. Figure 1In the tth round of federated learning, assume that there are n clients participating in the training. First, client i sends the local dataset Sample x in i , input to the embedding layer to get the embedding vector g φ (x i ). 0 (x i ) After mean processing, the local k-th prototype is obtained Secondly, the client compares the local prototype with the global prototype and obtains the correction term of the optimization target to update the local model. Finally, the client uploads the local prototype to the server for weighted aggregation. Assume that there are K types of local prototype categories sent by each client, that is, (where i∈[1,n]k∈[1,K]), the server performs weighted aggregation based on the distance between the uploaded local prototype and the global prototype. The closer the local prototype is to the global prototype, the greater the aggregation weight. Figure 1 In the local data set of client 1, the prototype categories obtained are tabby cat, blue cat (i.e. Figure 1 The original input of client 1 in the figure, the left and right pictures are respectively a tabby cat and a blue cat), client n has two prototypes, a tabby cat and a Shiba Inu, and the local prototype categories of these two clients may overlap according to the similarities and differences of the sample data (i.e. Figure 1 In the example, client 1 and client n have different numbers of data of the raccoon cat category. After these raccoon cat data of the same type are passed through the network model of the present invention on their respective devices, they can all have local prototypes of the raccoon cat category. The difference is that the client with more raccoon cat samples has a more accurate local prototype. ), the client uploads these local prototypes to the server. The server performs weighted aggregation on the local prototypes according to the global prototype distance to form a new round of global prototypes, and sends it to the clients participating in the training for the next round of learning, Figure 1 The picture in shows the aggregation process of the client local prototype to the server global prototype, but the global prototype obtained by aggregation is essentially a high-dimensional matrix. Figure 1 Formula c (t+1,k) To indicate that c (t+1,k) Represents the truly aggregated global prototype, which will be sent to the client to update the client's local model.

[0074] The present invention can aggregate the features of heterogeneous data images of each client through local prototypes uploaded to the server to obtain a global prototype without uploading the local model. The global prototype can be sent to the client to compare the loss with the local prototype and update the local model. The local model after learning is continuously updated, which has the following advantages in the application process:

[0075] (1) The present invention protects the image data information of the client segment. After a limited number of global communications, for client i, it has both image features from other clients and does not expose the gradient features of the local model of the client, thereby protecting the image data privacy of the user client (the traditional algorithm needs to upload the local model to the server, and the gradient contained in the local model can be reversed by the attacker to restore the client's data information, thereby causing the user client data to be leaked)

[0076] (2) The present invention reduces the time and network resource consumption in the federated learning process. In the traditional federated learning training process, uploading the local model of the client requires more network resources and time, because the local model contains some redundant image features that do not contribute much to the recognition task. In the present invention, redundant image features are filtered out by uploading the local prototype, so as to reduce the network resources and time spent by the client to upload to the server.

[0077] (3) The present invention can save the computing power of client devices and improve the recognition efficiency and accuracy of the model. In traditional federated learning, the computing power of client devices varies (device heterogeneity). A fixed number of training epochs may cause the model trained by devices with good computing power to enter overfitting in advance, while devices with poor computing power are still in an underfitting state but are forced to enter an aggregated state. In the present invention, the number of training epochs of the client is dynamically adjusted through constraints, which not only saves device computing power, but also prevents the local model from entering an over (under) fitting state, thereby improving recognition efficiency and accuracy.

Claims

1. A distance correction and aggregation method for heterogeneous models based on prototype federated learning, It is characterized in that include: On the client side, an embedding layer network is set up to extract deep features of heterogeneous images. The network is used to extract features of the input heterogeneous images and output them to a low-dimensional space to obtain an embedding vector. Then, the embedding vectors in the same space are clustered on average to obtain a local prototype. A correction term is added to the local objective function of the FedAvg algorithm to improve it. The correction term is the comparison loss function between the local prototype and the issued global prototype. Based on the loss function of the improved FedAvg algorithm, a gradient descent optimization is performed to update the local model, and the updated model is retained locally. Then, the local prototype is uploaded to the server. On the server side, the local prototype uploaded by the client is received, and the paradigm distance between the local prototype and the global prototype of the previous round is used as the aggregation weight of the local prototype of this round. The local prototype is weighted and aggregated to form a new global prototype, which is sent to the clients participating in the training for the next round of learning. in: On the client side, the obtained local prototype is represented as: Among them, x i represents the heterogeneous image data sample set owned by client i, y i Then it means x i The corresponding data labels, represents the local prototype of the kth class of client i in the tth round of global communication of federated learning, S i,k represents the heterogeneous image sample set representing the local prototype of the k-th class of client i, |S i,k | indicates S i,k Size, g φ (x i ) represents x i Embedding vector, i∈[1,n], n is the number of clients participating in the training, k∈[1,K], K is the number of local prototype categories sent by each client; On the client side, define the local optimization objective based on the prototype federated learning FedMPD algorithm: Among them, ω i represents the model parameters of client i, Represents ω i The optimization goal is Indicates that x i Input model ω i The prediction results obtained, N k represents the total number of k-th category data, represents the k-th data set of client i, express The size of c (t,k) represents the global prototype of the kth class in the tth round of communication; represents the supervised learning task loss, represents the correction term of the k-th prototype, is the local heterogeneous image dataset, λ is the correction parameter, and N is the total number of heterogeneous data images of all n clients; Among them, d() is the function for solving the paradigm distance, c (t,l) It is represented by other global prototypes other than the kth class, r is the distance between the kth class global prototype and its nearest (k+1 class) global prototype, and the mathematical expression of r is defined as: r = min d (c (t,k) ,c (t ,k+1) ); On the client side, the optimization constraints are set for the local objective function as follows: Among them, α is the marginal function of the embedding vector, α = d(g φ (x i ),g φ (x j )).

2. According to the heterogeneous model distance correction and aggregation method based on prototype federated learning according to claim 1, It is characterized in that On the server side, the closer the local prototype is to the global prototype, the greater the aggregation weight.

3. According to the heterogeneous model distance correction and aggregation method based on prototype federated learning according to claim 1, It is characterized in that On the server side, high-quality local prototypes are screened out by setting a threshold r, and are given greater weight for aggregation: