A federated learning method for heterogeneous mixed data based on global sum estimation

Abstract

This invention relates to a federated learning method for heterogeneous hybrid data based on global sum estimation. Under the assumption of non-overlapping data among clients, a zero-padding operation is designed so that the sum of the original data from all clients is the global data. Next, a deep neural network with a fully connected first layer is considered. Then, an auxiliary variable is used to estimate the global sum of the first-layer network output with M times the local data. Subsequently, a multi-step gradient descent algorithm is executed based on randomly selected mini-batch samples to locally update the network parameters. Finally, each client interacts with the central server, ultimately obtaining a global neural network model with higher accuracy that can be tested on the full feature set. The hybrid federated learning algorithm proposed in this invention, based on the zero-padding operation and the first-layer network output sum estimation model, can effectively solve the most generalized non-uniform HBFL problem.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a federated learning method for heterogeneous mixed data based on global summation estimation. Background Technology

[0002] With the development of science and technology, the digitalization process of society is accelerating, and people's demands for living standards are constantly increasing. Smart applications in many fields, including smart homes, smart healthcare, and virtual reality, are gradually emerging. These diverse applications generate massive amounts of connected data. To analyze and compute such massive amounts of data, artificial intelligence algorithms have emerged. However, traditional AI applications are based on cloud computing centers. These centers collect massive amounts of data from various clients or smaller computing centers and then train machine learning models. But this approach faces two major challenges: high communication costs and communication quality loss due to latency, and the inability to upload raw data due to data privacy concerns.

[0003] To address these challenges, federated learning, as an emerging distributed and secure computing architecture with privacy protection features, has become a hot research topic, providing privacy-preserving computational support for AI algorithms such as machine learning. Under the federated learning framework, collaborative model learning can be conducted on a massive number of clients (e.g., mobile devices or institutions) without requiring access to their raw data. Specifically, a federated learning system consists of multiple distributed institutions with local data and a central control server. The distributed institutions cannot directly exchange data; instead, model training is performed through iterative exchange of relevant information via the central server. This effectively avoids transmitting raw data while fully utilizing the potential information from each party's data. Based on the distribution of data in the network, federated learning tasks can be divided into three categories: horizontal federated learning (HFL) problems, vertical federated learning (VFL) problems, and hybrid federated learning (HBFL) problems.

[0004] For hybrid federated learning techniques, one proposed algorithm is a block coordinate descent (BCD) type algorithm called Hybrid-federated matched averaging (HyFEM). Specifically, each client performs local stochastic gradient descent (SGD) to learn a local model and train a feature extractor, then uploads it to the server to learn a global feature extractor. Due to the dimensionality mismatch between the local inference model and the global extractor, the server dynamically optimizes a linear mapping to match both. HyFEM is flexible in a sense because clients can use partial features for local inference (by using their local parameters) or use all features for global inference by requesting the global feature extractor on the server. However, HyFEM is limited to HBFL settings with uniform patterns, such as multi-view learning problems. It cannot handle more general problems, i.e., non-uniform pattern cases.

[0005] Therefore, traditional FL algorithms are designed for HBFL frameworks in HL, VL, and uniform modes respectively, and cannot handle the most generalized non-uniform HBFL problem, which includes all of these architectures. However, in reality, non-uniform HBFL scenarios are more common, where each organization may have raw data with different characteristics from different users. Therefore, how to handle HBFL in non-uniform modes more simply and efficiently based on rigorous theoretical analysis has become an urgent problem to be solved.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a federated learning method for heterogeneous mixed data based on global summation estimation, which solves the most generalized non-uniform HBFL problem.

[0008] The objective of this invention is achieved through the following technical solution: a federated learning method for heterogeneous hybrid data based on global summation estimation, wherein the federated learning method includes:

[0009] Configure the system network environment with one central server and M mobile clients, and build a global data matrix by constructing a local database on each client. ,in Indicates the number of samples. Let the number of features for each sample be denoted as , and then let ,in Indicates the first Each client has only a portion of the samples and features in the system.

[0010] Constructing the client-side local data matrix: First, obtain the client-side local dataset, which contains only a subset of samples and some features of those samples; then, perform zero-padding, setting all samples and features not present in the client-side local dataset to zero, resulting in the data matrix represented by the client m. ,in Indicates the number of samples. This represents the number of features for each sample. This represents the i-th sample from the m-th client. Let i = 1, ..., S, if client m has samples The If there are 1 feature, then the feature at the client end is equal to the corresponding feature in the global data matrix, that is... If the client cannot observe the feature, then the feature is set to zero. ;

[0011] Analyzing the relationship between the client's local data matrix and the global data matrix, we can see from the process of building the local data matrix that the feature of each sample in the global data matrix is ​​the sum of the features of all samples in the client's local matrix, that is... ;

[0012] Construct a system model and build an HBFL problem based on non-uniform data;

[0013] Set model parameters;

[0014] Local model update: in the During round iteration, each client Q local model updates are performed in parallel. For client m, the client receives the first-layer network parameters from the central server. Network parameters for the second layer and subsequent multilayer networks The first layer network outputs estimated parameters. Then the 0th step of the initial local multistep iteration is ;

[0015] Model aggregation and broadcasting: Each client uploads its local model to the central server, and the central server averages the models from each client and then sends the average back to each client.

[0016] Repeat the local model update and model aggregation and broadcasting steps for round T to obtain the parameters based on the round T. and A common neural network model;

[0017] Each client or central server inputs sample data containing the entire feature set into the learned common neural network model, thereby obtaining high-accuracy image classification.

[0018] The specific details of building the client's local database are as follows:

[0019] The imaging device captures an image of an object from a certain angle and records the label of each object as a data source for the image classification task, and determines the type of the image to determine the data distribution;

[0020] Each object is treated as a sample, and the images obtained by each client are used as a partial feature set. Each client is set to have only a partial sample set and feature set, and the feature sets of different clients do not overlap.

[0021] Each sample is treated as a row in the data matrix, while the feature image of each client is stretched into a vector and used as a partial column of the data matrix.

[0022] set up Represents the global data matrix, where Indicates the number of samples. This represents the number of features for each sample, and then... ,in Indicates the first There are 10 samples, and each sample contains D features.

[0023] The specific details of constructing the system model are as follows:

[0024] A deep neural network model is selected for image classification. The deep neural network model is divided into two parts: the first layer and the subsequent multiple layers. The first layer is a fully connected layer, and the subsequent multiple layers include other types of deep learning networks.

[0025] set up Indicates the parameters of the first layer. This indicates the parameters for subsequent multiple layers, with the dimension set to J, and... This represents the number of neurons in the second layer.

[0026] For the sample Set the loss function of the deep neural network as follows: ;

[0027] All clients interact with the central server to minimize the loss function and train a common deep neural network model to complete the federated learning task.

[0028] The HBFL problem based on non-uniform data includes:

[0029] Will Vectorization ,make , satisfy Then, determine the loss function. Finally, the HBFL problem based on non-uniform data is constructed. ;

[0030] Solve the problem about variables x and The derivatives are set as follows: and ,in, express about The derivative of .

[0031] The steps for setting model parameters specifically include the following:

[0032] The central server randomly initializes global variables. and Then broadcast them to all clients;

[0033] Introducing auxiliary variables to estimate the loss function about Gradient: Each client m introduces a global summation model Central server introduction To estimate the global sum That is, the sum of the local data matrix of the client and the linear combination of the parameters of the first layer of the neural network, which is also the sum of the outputs of the first layer of the deep neural network of all clients;

[0034] Each client m has an initial auxiliary variable of: The data is uploaded to the central server, which then initializes it with their average values. And send it back to each client;

[0035] Assume each client updates locally Q times, and the client interacts with the central server for a total of T rounds. In each round of interaction, the client uploads its local model. The central server aggregates all received models on an average basis and then distributes them to each client.

[0036] The steps for updating the local model specifically include the following:

[0037] A1. Randomly selecting a batch of sample sets is represented as follows: The sample size is B. ;

[0038] A2. Update the global summation model based on the selected sample set. ,get That is, each client m receives the message sent by the central server. Then, first add M times the current local first-layer network output, and then subtract M times the local first-layer network output from the previous round;

[0039] A3, Output of the first layer of the global network based on local estimation Calculate the loss function with respect to the parameters of subsequent multilayer networks. derivative Then with step size Performing stochastic gradient descent, we obtain ;

[0040] A4. Global first-layer output based on local estimation Calculate the derivative of the loss function with respect to the first layer parameter x of the network. Then multiply by M, and then by the step size Performing stochastic gradient descent, we obtain ;

[0041] A5. Repeat steps A1-A4 Q times;

[0042] The output of the first layer of the global network, estimated locally at step Q. Synchronize to each sample to obtain That is, based on all samples, each client m in Based on this, first add M times the output of the local first-layer network in step Q of round r, and then subtract M times the output of the local first-layer network in round (r-1).

[0043] The model aggregation and broadcasting steps specifically include the following:

[0044] Each client will store the first-layer network parameters in its local model. Parameters of the second and subsequent multilayer networks There are also auxiliary variables. Uploaded to the central server;

[0045] After receiving the models from each client, the central server averages them to obtain... , and ;

[0046] The central server will send the following four average models back to each client, including the average Layer 1 network parameters. Average parameters of the second and subsequent multilayer networks There are also average auxiliary variables. .

[0047] This invention has the following advantages: a federated learning method for heterogeneous hybrid data based on global summation estimation, and a hybrid federated learning algorithm based on data zero-padding operation and first-layer network output summation estimation model, which can effectively solve the most generalized non-uniform HBFL problem. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the process of the present invention;

[0049] Figure 2 A diagram illustrating parameter settings for a deep neural network;

[0050] Figure 3 The simulation results of the training loss function value based on a federated learning scenario with 4 clients are shown in the figure.

[0051] Figure 4 The figure shows the simulation results of classification accuracy testing in a federated learning scenario with 4 clients. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application provided below with reference to the accompanying drawings is not intended to limit the scope of protection of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The present invention will be further described below with reference to the accompanying drawings.

[0053] This invention specifically relates to a federated learning method for heterogeneous mixed data based on global sum estimation. Under the assumption of non-overlapping data among clients, a data zero-padding operation is designed so that the sum of the original data from all clients is the global data. Next, a deep neural network with a fully connected first layer is considered. Then, an auxiliary variable is used to estimate the global sum of the first-layer network output using M times the amount of local data. Subsequently, a multi-step gradient descent algorithm is executed based on randomly selected mini-batch samples to locally update the network parameters. Finally, each client interacts with the central server, ultimately obtaining a global neural network model with higher accuracy that can be tested on the full feature set. Therefore, this invention can efficiently handle the generalized non-uniform HBFL problem based on large-scale data.

[0054] like Figure 1 As shown, it specifically includes the following:

[0055] Step 1: System network environment configuration: Considering a federated learning system, set up 1 central server and M mobile clients;

[0056] Step 2: Build the client's local database;

[0057] 1. Obtain data:

[0058] (1) Data collection: Use imaging devices such as cameras to capture images of objects from a certain angle and record the labels of each object as the data source for image classification tasks;

[0059] (2) Data preprocessing. Determine the image type, such as RGB or grayscale, to determine the data distribution.

[0060] 2. Simulated data matrix:

[0061] (1) Treat each object as a sample and the images obtained by each client as a partial feature set; assume that each client only has a partial sample set and a partial feature set, where the feature sets of different clients do not overlap.

[0062] (2) Each sample is taken as a row of the data matrix, and the feature image of each client is pulled into a vector as a partial column of the data matrix.

[0063] (3) Let Represents the global data matrix, where Indicates the number of samples. This represents the number of features for each sample, and then... ,in Indicates the first Each client has only a portion of the samples and features in the system.

[0064] Step 3: Zero-filling operation for the data matrix;

[0065] To construct the client-side local data matrix, first obtain the client-side local dataset, which contains only a subset of samples and some features of those samples. Then, perform a zero-padding operation, setting all samples and features not present in the client-side local dataset to zero, resulting in the data matrix represented by the client m. ,in Indicates the number of samples. This represents the number of features for each sample. This represents the i-th sample from the m-th client. Let i = 1, ..., S; specifically, if client m has samples The If there are 1 feature, then the feature at the client end is equal to the corresponding feature in the global data matrix, that is... Conversely, if the client cannot observe the feature, then the feature is set to zero. .

[0066] Analyzing the relationship between the client's local data matrix and the global data matrix: From the process of building the local data matrix, we can see that the feature of each sample in the global data matrix is ​​the sum of the features of all samples in the client's local matrix, that is... .

[0067] Step 4: Construct the system model;

[0068] 1. For example Figure 2 As shown, a suitable deep neural network (NN) model is selected for image classification. The NN model is divided into two parts: the first layer and subsequent layers. The first layer is a fully connected layer, and the subsequent layers can be various types of deep learning networks, such as convolutional neural networks, recurrent neural networks, etc.

[0069] 2. Order Indicates the parameters of the first layer. This indicates the parameters for subsequent multiple layers, with the dimension set to J, and... This represents the number of neurons in the second layer.

[0070] 3. For the sample Set the loss function of the deep neural network as follows: The loss function can be determined specifically based on the learning task. For example, in image classification tasks, a logistic regression function or a cross-entropy function can be used. It is also assumed that the function is smooth and non-convex.

[0071] 4. All clients interact with the central server to minimize the loss function and train a common deep neural network model to complete federated learning tasks, such as image classification.

[0072] Step 5: Construct system problems;

[0073] Will Vectorization ,make , satisfy Then, determine the loss function. Finally, the HBFL problem based on non-uniform data is constructed. ;

[0074] Furthermore, for ease of description, the objective function in the above problem is defined as follows: Solve the problem with respect to variables x and The derivatives are respectively and ,in, express about The derivative of .

[0075] Based on variables x and The derivative formulas and gradients of model variables both require the use of the global data matrix. However, the client only has a partial sample set and feature set, and cannot interact with the original data due to data privacy requirements, which makes general federated learning algorithms no longer applicable.

[0076] Step 6: Parameter settings;

[0077] 1. Initialize the model, assuming the initial iteration is the 0th iteration;

[0078] (1) The central server randomly initializes global variables. and Then broadcast them to all clients;

[0079] (2) Introduce auxiliary variables to estimate the loss function about Gradient: Each client m introduces a global summation model Central server introduction To estimate the global sum That is, the sum of the local data matrix of the client and the linear combination of the parameters of the first layer of the neural network, which is also the sum of the outputs of the first layer of the deep neural network of all clients;

[0080] (3) The initial auxiliary variable for each client m is: The data is uploaded to the central server, which then initializes it with their average values. And send it back to each client;

[0081] 2. Assume each client updates locally Q times, and the client interacts with the central server for a total of T rounds. In each round of interaction, the client uploads its local model. The central server aggregates all received models on an average basis and then distributes them to each client.

[0082] Step 7: Local model update;

[0083] In the During round iteration, each client Q local model updates are performed in parallel. For client m, the client receives the first-layer network parameters from the central server. Network parameters for the second layer and subsequent multilayer networks The first layer network outputs estimated parameters. Then the 0th step of the initial local multistep generation is ;

[0084] Assumption ;

[0085] 1. A small batch of samples is randomly selected and represented as follows: , where the sample size is B;

[0086] 2. Update the global summation model based on the selected sample set. ,get That is, each client m receives the message sent by the central server. Then, first add M times the current local first-layer network output, and then subtract M times the local first-layer network output from the previous round;

[0087] 3. Output of the first layer of the global network based on local estimation Calculate the loss function with respect to the parameters of subsequent multilayer networks. derivative Then with step size Performing stochastic gradient descent, we obtain ;

[0088] 4. Global first-layer output based on local estimation Calculate the derivative of the loss function with respect to the first layer parameter x of the network. Then multiply by M, and then by the step size Performing stochastic gradient descent, we obtain ;

[0089] 5. Repeat steps 1-4 Q times;

[0090] 6. The above steps are only for a small batch of selected samples. Therefore, the next step is to estimate the output of the first layer of the global network locally in step Q. Synchronize to each sample to obtain That is, based on all samples, each client m in Based on this, first add M times the output of the local first-layer network in step Q of round r, and then subtract M times the output of the local first-layer network in round (r-1).

[0091] Step 8: Model aggregation and broadcasting;

[0092] 1. Each client will store the parameters of the first layer of its local neural network model. Parameters of the second and subsequent multilayer networks There are also auxiliary variables. Uploaded to the central server;

[0093] 2. After receiving the models from each client, the central server averages them to obtain... , and ;

[0094] 3. The central server will transmit the following three average models back to each client, including the average Layer 1 network parameters. Average parameters of the second and subsequent multilayer networks There are also average auxiliary variables. .

[0095] Step 9: Repeat steps 7 and 8 of the T-round;

[0096] Finally, each client and the central server obtained the parameters based on the Tth round. and The common neural network model.

[0097] Step 10: Test the model;

[0098] During the testing phase, each client or central server inputs sample data containing the entire feature set into the learned common neural network model to obtain image classification with high accuracy.

[0099] The experimental simulation of this invention is as follows:

[0100] Scenario Setup: Consider the MNIST dataset, which contains 60,000 training samples (images with a dimension of 28*28) and 10,000 test samples. The training samples are divided into 8 mini-batches, each with 7,500 samples. Each sample image is then divided into four equal parts, forming four feature subsets. These feature subsets are then randomly assigned to four clients. The assignment results satisfy the non-uniform pattern of the HBFL scenario, and the data features of the clients do not overlap.

[0101] Deep Neural Networks and Loss Functions: Consider a two-layer fully connected neural network with 30 neurons, using cross-entropy as the loss function.

[0102] Parameter settings: Number of communications: 500, Step size: .

[0103] like Figure 3 and Figure 4 As shown, the FedHD algorithm proposed in this invention can effectively solve the non-uniformity problem of HBFL, and appropriately increasing the number of local updates can improve performance, such as... The time algorithm outperforms However, excessively increasing the number of local updates will increase the differences between client models, thus failing to effectively improve performance. Furthermore, the FedHD (Hybrid data based federated learning) algorithm proposed in this invention is far superior to locally trained models (trained locally using only local data).

[0104] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A federated learning method for heterogeneous hybrid data based on global summation estimation, characterized in that: The federated learning method includes: Configure the system network environment to include one central server and M A mobile client is used to build a local database on the client and obtain a global data matrix. ,in Indicates the number of samples. Let the number of features for each sample be denoted as , and then let ,in Indicates the first Each client has only a portion of the samples and features in the system. Constructing the client-side local data matrix: First, obtain the client-side local dataset, which contains only a subset of samples and some features of those samples; then, perform zero-padding, setting all samples and features not present in the client-side local dataset to zero, thus obtaining the client-side local data matrix. m The data matrix is ​​represented as follows ,in Indicates the number of samples. This represents the number of features for each sample. The first one to be constructed m The first client's i One sample, Let i = 1, ..., S, if the client m Having samples The If there are 1 feature, then the feature at the client end is equal to the corresponding feature in the global data matrix. If the client cannot observe the feature, then set the feature value to zero. ; Analyzing the relationship between the client's local data matrix and the global data matrix, we can see from the process of building the local data matrix that the features of each sample in the global data matrix are the sum of the features of all samples in the client's local matrix. ; Construct a system model and build an HBFL problem based on non-uniform data; Set model parameters; Local model update: in the During round iteration, each client Perform Q local model updates in parallel for the client. m The client receives the first-layer network parameters from the central server. Network parameters for the second layer and subsequent multilayer networks The first layer network outputs estimated parameters. Then the 0th step of the initial local multistep iteration is ; Model aggregation and broadcasting: Each client uploads its local model to the central server, and the central server averages the models from each client and then sends the average back to each client. Repeat the local model update and model aggregation and broadcasting steps for round T to obtain the parameters based on the round T. and A common neural network model; Each client or central server inputs sample data containing the entire feature set into a learned common neural network model to obtain image classification.

2. The federated learning method for heterogeneous hybrid data based on global summation estimation according to claim 1, characterized in that: The specific details of building the client's local database are as follows: The imaging device captures an image of an object from a certain angle and records the label of each object as a data source for the image classification task, and determines the type of the image to determine the data distribution; Each object is treated as a sample, and the images obtained by each client are used as a partial feature set. Each client is set to have only a partial sample set and feature set, and the feature sets of different clients do not overlap. Each sample is treated as a row in the data matrix, while the feature image of each client is stretched into a vector and used as a partial column of the data matrix. set up Represents the global data matrix, where Indicates the number of samples. This represents the number of features for each sample, and then... ,in Indicates the first There are 10 samples, and each sample contains D features.

3. The federated learning method for heterogeneous hybrid data based on global summation estimation according to claim 1, characterized in that: The specific details of constructing the system model are as follows: A deep neural network model is selected for image classification. The deep neural network model is divided into two parts: the first layer and the subsequent multiple layers. The first layer is a fully connected layer, and the subsequent multiple layers include other types of deep learning networks. set up This represents the network parameters of the first layer. This represents the network parameters for subsequent layers, with dimension J set, and... This represents the number of neurons in the second layer. For the sample Set the loss function of the deep neural network to be ; All clients interact with the central server to minimize the loss function and train a common deep neural network model to complete the federated learning task.

4. The federated learning method for heterogeneous hybrid data based on global summation estimation according to claim 3, characterized in that: The HBFL problem based on non-uniform data includes: Will Vectorization ,make , satisfy Then, determine the loss function. Finally, the HBFL problem based on non-uniform data is constructed. ; Solve the problem about variables x and The derivatives are set as follows: and ,in, express about The derivative of .

5. The federated learning method for heterogeneous hybrid data based on global summation estimation according to claim 4, characterized in that: The steps for setting model parameters specifically include the following: The central server randomly initializes global variables. and Then broadcast them to all clients; Introducing auxiliary variables to estimate the loss function about Gradient: per client m Introducing a global summation model Central server introduction To estimate the global sum ; Each client m The initial auxiliary variable is The data is uploaded to the central server, which then initializes it with their average values. And send it back to each client; Assume each client updates locally Q times, and the client interacts with the central server for a total of T rounds. In each round of interaction, the client uploads its local model. The central server aggregates all received models on an average basis and then distributes them to each client.

6. The federated learning method for heterogeneous hybrid data based on global summation estimation according to claim 5, characterized in that: The model aggregation and broadcasting steps specifically include the following: Each client will store the first-layer network parameters from its local model. Parameters of the second and subsequent multilayer networks There are also auxiliary variables. Uploaded to the central server; After receiving the models from each client, the central server averages them to obtain... , and ; The central server will send the following three average models back to each client, including the average Layer 1 network parameters. Average parameters of the second and subsequent multilayer networks There are also average auxiliary variables. .

Images