A federated learning heterogeneous model aggregation method based on fairness feature representation

Abstract

The application provides a federated learning heterogeneous model aggregation method based on fairness feature representation, and belongs to the technical field of machine learning. The method mainly comprises the following steps: feature matching degree measurement; communication parameter dynamic selection; and heterogeneous model aggregation. The application quantifies feature mismatch caused by different training processes due to heterogeneous data and heterogeneous computing resources, uploads local parameters of different feature representations in a fair manner, and uses a model aggregation method with dynamic matching features to greatly reduce the influence of unordered feature extraction between different devices and avoid overfitting of the global model in collaborative aggregation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine learning, and particularly relates to a federated learning heterogeneous model aggregation method based on fairness feature representation. BACKGROUND

[0002] Federated learning is a new distributed machine learning algorithm, model parameters are locally trained on each edge device, then transmitted to the central server for aggregation, and finally the aggregated global model parameters are returned to each device for the next training process. Sharing training parameters instead of local data plays a role in data privacy protection, and also makes effective use of the computing resources of edge devices.

[0003] However, when different users use mobile edge devices in different scenarios, the data collected by the devices has statistical heterogeneity and is not independent and identically distributed data. In addition, there are more than 2000 different system-on-chips for mobile edge devices, and the available computing resources differ greatly. The mobility of the device also causes the communication bandwidth to fluctuate. The heterogeneity of the data and the heterogeneity of the devices hinder the generalization of the global model and pose challenges to the aggregation of heterogeneous models of federated learning nodes.

[0004] Therefore, a federated learning heterogeneous model aggregation method based on fairness feature representation is needed to overcome the above shortcomings. SUMMARY

[0005] In order to overcome the problem that the heterogeneity of multiple data and the heterogeneity of devices hinder the generalization of the global model, the application provides a federated learning heterogeneous model aggregation method based on fairness feature representation.

[0006] In order to achieve the above purpose, the application provides the following technical scheme:

[0007] A federated learning heterogeneous model aggregation method based on fairness feature representation, comprising the following steps:

[0008] The federated learning heterogeneous model aggregation method based on fairness feature representation provided by the application has the following

[0009] Beneficial effects:

[0010] The application quantifies the feature mismatch caused by different training processes due to heterogeneous data and heterogeneous computing resources, uploads local parameters of different feature representations based on fairness, and uses a model aggregation method with dynamic matching features to greatly reduce the influence of unordered feature extraction between different devices and avoid overfitting of the global model in collaborative aggregation. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application and the design thereof, the following will briefly introduce the drawings required by the present embodiment. The drawings in the following description are only part of the embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0012] Figure 1 The principle block diagram of the federated learning heterogeneous model aggregation method based on fairness feature representation of the present application 1;

[0013] Figure 2 The flow chart of the federated learning heterogeneous model aggregation method based on fairness feature representation of the present application 1. DETAILED DESCRIPTION

[0014] In order for those skilled in the art to better understand the technical solutions of the present application and to implement them, the present application will be described in detail below in conjunction with the drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0015] In order to facilitate understanding, the following explains the terms involved in the present application:

[0016] Communication overhead: In the process of network transmission, due to the transmission of signals, it is necessary to transform the data format, and some redundant data will inevitably be added, which is necessary for transmission, and the proportion of these redundant data in the source data is called overhead.

[0017] Embodiment 1

[0018] The present application solves the problem of feature mismatch and feature representation range mismatch caused by the heterogeneity of data collected by mobile edge devices and the heterogeneity of computing resources and communication capabilities, builds a local machine learning model of each mobile edge device based on a convolutional neural network in the training process, and proposes a model parameter aggregation method under heterogeneous conditions, which improves the convergence and accuracy of federated learning training and the generalization of the global model.

[0019] Specifically, the present application provides a federated learning heterogeneous model aggregation method based on fairness feature representation, the implementation schematic diagram of the method is as shown in Figure 1 , and the specific flow chart is as shown in Figure 2 , including the following processing steps:

[0020] Step S1: Each mobile edge device (such as a mobile phone, a smart watch, a smart doorbell, and any device with information collection, transmission, and processing capabilities) participating in federated learning is regarded as a client, each client establishes a convolutional neural network model with the same network structure, i.e., a global neural network model (hereinafter referred to as a global model), the central server initializes the global neural network model parameters, the network model parameters refer to the convolution kernel weights of the convolutional neural network, and a unified loss function is constructed.

[0021] Step S2: The central server quantizes the feature representation of the current global model.

[0022] Specifically, step S2 further includes:

[0023] Step S200, sequentially select the i-th layer of the global model.

[0024] Step S201, select a filter as the basic structure of the feature representation, and regard each filter in the i-th layer as a point in the Euclidean space.

[0025] Step S202, use each point in the i-th layer space to calculate the geometric center (Geometric Median, GM) of the i-th layer as the feature representation of the layer:

[0026]

[0027] In the formula, is the geometric center of the i-th layer of the global model, which is a three-dimensional matrix of size N i ×K×K, N i and N i+1 are the number of input and output channels of the i-th layer, K is the size of the convolution kernel, F i,j is the j-th filter of the i-th layer of the global model; R is the Euclidean space, and x is a point in the Euclidean space.

[0028] Step S203, determine whether the last layer of the global model is traversed, if yes, go to step S204, otherwise select the next layer i+1 and return to step S201.

[0029] Step S204, sequentially calculate the feature representation of each layer of the global model, and achieve the quantization of the feature representation of the global model.

[0030] Step S3: The central server distributes the parameters of the initialized global model and the quantized feature representation of the global model to each client.

[0031] Step S4: After each client receives the parameters and feature representation of the global model issued by the central server, the local heterogeneous data collected is used for training to obtain the trained local model. The heterogeneous data refers to the data collected by different clients which does not have the property of independent and identically distributed. The data categories and quantities collected by clients at different locations, facing different tasks and having different information collection capabilities are greatly different. For example Figure 1 The camera device 1 may collect more pictures of puppies, while the camera device 2 and the doorbell device may collect more pictures of birds and kittens respectively. Each participant randomly selects part of the training data owned by itself as batch data, and uses stochastic gradient descent (SGD) to train the model on the batch data, so that the model parameters W of each participant are updated.

[0032] Step S5: Each client compares the feature representation of the local model and the global model, that is, measures the relationship between the training results of the local model and the global model after each round of training, and determines the degree of feature mismatch.

[0033] Specifically, step S5 further includes:

[0034] Step S500: sequentially selecting the i-th layer of the local model.

[0035] Step S501: sequentially selecting the j-th filter of the i-th layer of the local model.

[0036] Step S502: calculating the similarity between the j-th filter of the i-th layer of the local model and the feature representation of the i-th layer of the global model.

[0037]

[0038] In the formula, is the j-th filter of the i-th layer of the local model of the k-th client, represents the similarity between the geometric center of the i-th layer of the global model and the j-th filter of the i-th layer of the local model of the k-th client; if an i-th layer filter is closer to the geometric center of the i-th layer of the global model, it is considered that the feature extracted by the filter is more similar to the common feature of the global model.

[0039] Step S503: determining whether the last filter of the layer is traversed, if yes, entering step S504, otherwise selecting the next filter j+1 and returning to step S501.

[0040] Step S504: determining whether the last layer of the model is traversed, if yes, entering step S506, otherwise selecting the next layer i+1, j=1 and returning to step S501.

[0041] Step S506, compare the similarity of each filter of the local model to the feature representation of the global model, if a filter is closer to the feature representation of the global model, it is considered that the feature extracted by the filter is more similar to the common feature of the global model.

[0042] Step S6: the client clusters the filters of each layer using the similarity of each filter of the local model to the feature representation of the global model.

[0043] Specifically, step S6 further comprises:

[0044] Step S600, set n distance threshold range τ.

[0045] Step S601, the filters with similarity within the threshold τ are divided into a group, and it is considered that the filters in the same group have more similar feature representations, and the client records the group of each filter.

[0046] Step S602, select the same proportion of filters in each group respectively, and select the filters with different feature representations as the filter candidate set F k .

[0047] Step S7: the client evaluates the communication overhead of uploading each group of filter candidate set.

[0048] Specifically, step S7 further comprises:

[0049] Step S700, according to the requirement of the quality of service of mobile edge computing, set the delay threshold γ th .

[0050] Step S701, calculate the transmission rate of each client k to the central server S, which is calculated as follows:

[0051]

[0052] In the formula, T k is the transmission power, , the variance of additive white Gaussian noise (AWGN) at the data receiving end, and the channel parameter is h k , the communication bandwidth of the local device k is B k .

[0053] Step S702, calculate the data transmission time required by each mobile edge device to upload different filter candidate set F k , which is calculated as follows:

[0054]

[0055] In the formula, is the data transmission time of device k uploading Uk the transmission time of the data volume of data to the server required; U k denotes the size of the data uploaded by the local device k, mostly in 32-bit floating-point format; C k is the transmission rate from the local device k to the server S.

[0056] Step S8: Client dynamically optimizes filter parameter selection, that is, the client determines the category of feature representation based on feature matching degree, selects the same proportion of filters in different categories, and dynamically selects communication parameters according to a heterogeneous network environment.

[0057] Specifically, step S8 further includes:

[0058] Step S800, under the constraint γ of communication delay th Next, an optimization problem is constructed to balance the size of uploaded data and the contribution of parameters, to ensure that both specific communication requirements and model accuracy can be met. For each device k, the specific dynamic optimization objective is as follows:

[0059]

[0060] In the formula, F k is the filter candidate set of the device k, S is a sparse term for implementing the above filter selection, is the communication time of the corresponding parameter, γ th is a delay threshold.

[0061] Step S801, embedding constraint conditions into the optimization objective to simplify the calculation process, and obtaining a Lagrange function:

[0062]

[0063] Step S802, introducing an auxiliary variable Z to replace the variable F in the functions S and communication constraints, and obtaining an augmented Lagrange function of the problem:

[0064]

[0065] In the formula, π is the Lagrange multiplier, and ρ is the penalty parameter and ρ>0.

[0066] Step S803, defining π k = ρu k , and further obtaining:

[0067]

[0068] Step S804, based on the augmented Lagrangian function, the Alternating Direction Method of Multipliers can be solved by using the following sub-problem iteration to find the filter candidate set that optimizes the dynamic optimization objective, as the data each mobile edge device needs to upload to the central server.

[0069]

[0070]

[0071]

[0072] Step S805, the client uploads the filter candidate set that optimizes the dynamic objective and the group each filter belongs to within the delay constraint γ th to ensure the quality of edge computing services.

[0073] Step S9: the central server performs classification of heterogeneous model aggregation, that is, the central server aggregates parameters with the same or similar feature representations according to the category of the feature representation.

[0074] Specifically, step S9 further includes:

[0075] Step S900, the central server waits until it receives all the filter updates uploaded by the clients and the groups the filters belong to.

[0076] Step S901, the central server aggregates the filters of the same group in each layer, and the aggregation method is as follows:

[0077]

[0078] In the formula, w i is the average parameter, K is the number of clients, w i,k is the communication parameter;

[0079] Step S902, after the parameters are aggregated, the central server updates all the models using the fused average parameters.

[0080] Step S10: repeat steps 2-9 until the set number of iterations is reached.

[0081] Based on the review, the application of a federated learning heterogeneous model aggregation method based on fairness feature representation uses heterogeneous data for training in heterogeneous computing resources, and solves the problems of feature representation mismatch and feature representation range mismatch brought by uploading in a heterogeneous network environment. The feature representation (i.e. filter) of each participant is quantized, the distance from the global model feature representation is measured, the common features between different filters are found, the filters with different feature representations are selected and uploaded to the central server, and finally the central server aggregates the common features in the heterogeneous model respectively, and the parameter update of each client is used fairly, so that the global model has better generalization, greatly reduces the influence of unordered feature extraction between different devices, and improves the model accuracy.

[0082] The above-described embodiments are only the preferred specific implementation of the present application, and the protection scope of the present application is not limited thereto, and any simple change or equivalent replacement of the technical solutions within the technical range disclosed by the present application can be obtained by those skilled in the art, which belongs to the protection scope of the present application.

Claims

1. A federated learning heterogeneous model aggregation method based on fairness feature representation, characterized in that, The method comprises the following steps: The central server initializes parameters of a global model uploaded by each client, quantizes a feature representation of the global model, and sends the parameters and the feature representation to each client; the specific steps are as follows: The central server uses filters of each layer of the global model as a basic structure of the quantized feature representation, regards each filter in the i-th layer of the global model as a point in a Euclidean space, and calculates a geometric center GM of the i-th layer as the feature representation of the layer; After the central server calculates the geometric center of each layer of the global model, the calculation result is sent to each client; Each client trains the received parameters and the feature representation by using locally collected heterogeneous data to obtain a trained local model; the heterogeneous data refers to data collected by different clients, which do not have the independent and identically distributed property, are located at different positions, and face different tasks; Each client compares the feature representations of the local model and the global model to determine a feature matching degree; The specific steps are as follows: The client compares the geometric center of the i-th layer of the global model with the j-th filter of the i-th layer of the local model of the k-th client, and calculates the similarity between the two; the similarity is evaluated based on the geometric center: before uploading the parameters of the local model each time, the similarity between each filter in the local model and the geometric center of the same layer of the global model is calculated in a loop; The client clusters each filter of the local model based on the matching degree of the filter and the feature representation of the global model to obtain a filter candidate set; the specific steps are as follows: Setting n distance threshold range τ; filters with similarity within the threshold τ are divided into a group, considering that filters in the same group have more similar feature representations, and the client records the group of each filter; select the same proportion of filters in each group respectively, and select filters with different feature representations as filter candidate set F by fairness k ; The client evaluates the communication overhead of each filter candidate set and uploads the communication overhead to the central server; The client determines the category of the feature representation based on the feature matching degree, selects filters in the same proportion in different categories, and dynamically selects communication parameters according to a heterogeneous network environment; The central server aggregates communication parameters with the same or similar feature representation according to the category of the feature representation, and performs heterogeneous model aggregation.

2. The federated learning heterogeneous model aggregation method based on fairness feature representation according to claim 1, characterized in that, The global model is a convolutional neural network model, and the parameters of the global model refer to convolution kernel weights of the convolutional neural network.

3. The federated learning heterogeneous model aggregation method based on fairness feature representation according to claim 1, characterized in that, The calculation method of the geometric center of the i-th layer of the global model is shown in formula (1): wherein is the geometric center of the i-th layer of the global model, of size N i is a three-dimensional matrix of size N i and N i+1 is the number of input and output channels of the i-th layer, K is the size of the convolution kernel, F i,j is the j-th filter of the i-th layer of the global model; R is a Euclidean space, and x is a point in the Euclidean space.

4. The federated learning heterogeneous model aggregation method based on fairness feature representation according to claim 3, characterized in that, The calculation method of the similarity between the geometric center of the i-th layer of the global model and the j-th filter of the i-th layer of the local model of the k-th client is shown in formula (2): wherein, is the jth filter of the ith layer of the local model of the kth client, represents the similarity between the geometric center of the ith layer of the global model and the jth filter of the ith layer of the local model of the kth client; if a filter of an ith layer is closer to the geometric center of the ith layer of the global model, it is considered that the feature extracted by this filter is more similar to the common features of the global model.

5. The federated learning heterogeneous model aggregation method based on fairness feature representation according to claim 4, characterized in that, The client evaluates the communication overhead of each filter candidate set, and the specific steps are as follows: Compute different filter candidate sets F uploaded by each client k Data transmission time required, ensuring that the parameter upload can be within the delay threshold γ th is completed, the calculation is shown in formula (3): wherein, Ukis the upload data size of the kth client; k the transmission time required for the data amount of data to the central server; U k Ukdenotes the data size uploaded by the kth client; C k is the transmission rate from the kth client to the central server, which is calculated as shown in equation (4): In the formula, T k is the transmission power, represents the variance of the additive white Gaussian noise at the data receiving end, and the channel parameter is h k , and the communication bandwidth of the kth client is B k .

6. The federated learning heterogeneous model aggregation method based on fairness feature representation according to claim 5, characterized in that, The client determines the category of the feature representation based on the feature matching degree, selects filters in the same proportion in different categories, and dynamically selects communication parameters according to a heterogeneous network environment; the specific steps are as follows: Under the constraint of communication delay, an optimization problem is constructed to balance the size of the uploaded data and the contribution of the parameters; for each client, the specific dynamic optimization objective is shown in formula (5): where Loss is the loss function of the model, F k is the filter candidate set for the kth client, S is a sparse term that implements the filter selection described above, is the communication time of the respective parameter, γ th is the delay threshold value; The constraint objective is embedded into formula (5) to obtain a Lagrange function shown in formula (6): In the formula, λ is a Lagrange multiplier. An auxiliary variable Z is introduced to replace the variable F in the function S and the communication constraint, and an augmented Lagrangian function of the problem is obtained; Based on the augmented Lagrangian function, an alternating direction multiplier method is performed to solve the filter candidate set that optimizes the dynamic optimization objective, as the data each client needs to upload to the central server.

7. The federated learning heterogeneous model aggregation method based on fairness feature representation according to claim 6, characterized in that, The central server aggregates the communication parameters with the same or similar feature representations according to the categories of the feature representations, and performs heterogeneous model aggregation, and the specific steps are as follows: The central server receives all the filters uploaded by the nodes and the group to which the filters belong within a delay threshold γ th time. The central server aggregates the filters in the same group in each layer, and the aggregation mode is shown in formula (7): where w i is an average parameter, K is the number of clients, w i,k is a communication parameter; After the parameters are aggregated, the average parameters will be sent back to the original convolution kernel in the local model, and in the next round of testing, the aggregated parameters and the uncommunicated parameters are trained using local data.

8. The federated learning heterogeneous model aggregation method based on fairness feature representation according to claim 1, characterized in that, It also includes that the central server constructs a loss function to supervise the training of the heterogeneous model through the loss function.

Images