Adaptive clustering federated learning method based on DBSCAN

Abstract

The invention discloses a self-adaptive clustering federated learning method based on DBSCAN. The method comprises the following steps that A, a server initializes a global model and corresponding hyper-parameters; b, all clients use a local data training model to obtain feature vectors of local data and send the feature vectors to a server; c, the server adaptively determines the domain radius and the value of the minimum contained point number in the DBSCAN clustering algorithm, the clients are divided through the DBSCAN clustering algorithm, and a clustering result is sent to all the clients; and D, after receiving the clustering result, the client performs a federated learning model training process according to a cluster form until the model converges or reaches a specified communication round number. According to the method, the federal learning performance can be effectively improved, and the influence caused by data isomerism is relieved; meanwhile, parameters of the DBSCAN clustering algorithm are determined in a self-adaptive mode, and the clustering result is improved.

Description

Technical Field

[0001] The present invention relates to the field of federated learning and clustering technology, and in particular to an adaptive clustering federated learning method based on DBSCAN. Background Art

[0002] With the rapid development of big data-driven artificial intelligence technology, data silos and data privacy protection requirements have become key bottlenecks that limit the performance improvement and widespread application of AI models. Federated learning (FL), as an emerging distributed machine learning method, can combine multi-party data to establish a global model while protecting data privacy, effectively breaking down data barriers. However, although federated learning has been widely used, in real-world environments, data between different devices vary greatly in terms of order of magnitude, quantity, type, and features, leading to data heterogeneity. Data heterogeneity not only slows down the training of federated learning models, but also greatly reduces the accuracy of the models.

[0003] At present, clustering federated learning, as one of the main methods to solve data heterogeneity, can effectively alleviate the impact caused by data heterogeneity. For example, the technical solution disclosed in the patent application number 202210378145.X and the patent name "Client Selection Federated Learning Method Based on DBSCAN Clustering": After the parameter server initializes the model parameters, it distributes the initialized model to each client node; each client node performs local training operations; each client node sends the trained model parameters back to the parameter server; the parameter server clusters the received model parameters and divides them into different clusters; the parameter server performs multiple rounds of iterative learning in different clusters, and then weights and averages the model parameters of the selected client nodes in each round according to the sample weights, and finally obtains the global model. Based on privacy protection, the above technology does not need to know the local data of the client, and uses the DBSCAN clustering algorithm to cluster the local training model parameter information of the client node to reduce the impact of non-independent and identically distributed data.

[0004] Although the above-mentioned federated learning method based on DBSCAN clustering can reduce the impact of data heterogeneity, directly using model parameter information for clustering may lead to privacy leakage risks. In addition, the DBSCAN clustering algorithm is sensitive to the setting of hyperparameters (such as neighborhood radius eps and minimum number of samples MinPts). Unreasonable parameter selection may lead to unstable clustering results, which in turn affects the overall effect of federated learning and model convergence performance. Summary of the invention

[0005] The object of the present invention is to provide an adaptive clustering federated learning method based on DBSCAN, which can effectively improve the performance of federated learning and alleviate the impact brought by data heterogeneity; at the same time, by adaptively determining the parameters of the DBSCAN clustering algorithm, the clustering results can be improved.

[0006] The present invention adopts the following technical solutions:

[0007] An adaptive clustering federated learning method based on DBSCAN includes the following steps:

[0008] A: The server initializes the global model and corresponding hyperparameters, and several clients preprocess their local datasets to construct training sets and test sets; the hyperparameters include the number of communication rounds, the learning rate, and the number of local update rounds.

[0009] B: All clients use local data to train the model, obtain the feature vectors of the local data, and send them to the server; the model includes a basic encoder and a projection head; the basic encoder is used to map the local data of the client to the same latent vector space to obtain the high-dimensional feature representation of the local data; the projection head is used to map the high-dimensional features into low-dimensional features.

[0010] C: The server receives the low-dimensional features of all clients to form the feature matrix of all clients, the server adaptively determines the values of the neighborhood radius and the minimum number of points included in the DBSCAN clustering algorithm, divides the clients through the DBSCAN clustering algorithm, and sends the clustering results to all clients.

[0011] D: After receiving the clustering results, the clients perform the training process of the federated learning model in the form of clusters until the model converges or reaches the specified number of communication rounds.

[0012] Step A includes the following specific steps:

[0013] A1: The server initializes the global model, the number of communication rounds, the learning rate, and the number of local update rounds, and distributes them to all clients.

[0014] A2: The client preprocesses the local data.

[0015] A3: The client divides the preprocessed data into a training set and a test set.

[0016] Step B includes the following specific steps:

[0017] B1: Feature extraction is performed on the local data of each client through a shared encoder, and after mapping to a unified high-dimensional feature space, a high-dimensional feature representation is obtained.

[0018] B2: Receive the high-dimensional features extracted by the encoder through the projection head, map the features to a low-dimensional space, and obtain low-dimensional features after normalization;

[0019] B3: All clients send the low-dimensional features to the server.

[0020] In step C, the steps of the method for adaptively determining parameters are as follows:

[0021] C1: The server calculates the first distance matrix between clients based on the Euclidean distance;

[0022] C2: The server calculates the first similarity matrix between clients based on the cosine similarity, and simplifies it using the similarity threshold to obtain the second similarity matrix;

[0023] C3: Multiply the first distance matrix by the second similarity matrix to obtain the second distance matrix;

[0024] C4: Statistically calculate the minimum and maximum values other than 0 in the second distance matrix, then use the minimum and maximum values as the starting value and the ending value to divide m intervals, and count the frequencies of the values in the second distance matrix falling into each interval. Finally, use the left endpoint value of the interval with the highest frequency as the value of the neighborhood radius; Set the value of the minimum number of points to be included as 1;

[0025] C5: According to the value of the neighborhood radius and the minimum number of points to be included, the server uses the DBSCAN clustering algorithm to divide the clients based on the feature matrices of all clients, divides the clients with similar data distributions into the same cluster, and sends the clustering results to all clients.

[0026] In step D, each cluster trains a local model and uploads the model update information to the server. The server aggregates the locally updated models of all clusters in a weighted average manner to form a new global model, and distributes it to each cluster; Repeat the above process until the model converges or reaches the specified number of communication rounds

[0027] Step D includes the following specific steps:

[0028] D1: The server initializes the global model and distributes it to all clients, and all clients train local models based on local data;

[0029] D2: According to the clustering results, calculate the local shared model of each cluster, and upload the updated model parameter information to the server;

[0030] D3: The server weights and aggregates the local shared models of all clusters to calculate and form a new round of global model;

[0031] D4: The server distributes a new round of the global model to all clients and repeats steps D1 to D3 until the model converges or reaches the specified number of communication rounds.

[0032] The number m of intervals is the same as the number of clients.

[0033] In step A2, first, missing values, outliers, and duplicate values in the local data of each client are processed; for clients with insufficient local data volume, StyleGAN-T is used to generate synthetic data.

[0034] The encoder adopts a convolutional neural network architecture, including multiple convolutional layers, non-linear transformations, batch normalization, and pooling layers; the projection head adopts a multi-layer perceptron architecture, including an input layer, a hidden layer, and an output layer.

[0035] According to the set similarity threshold, the values in the first similarity matrix with a similarity greater than or equal to the similarity threshold are set to 1, and the values less than the similarity threshold are set to 0, finally obtaining the simplified second similarity matrix.

[0036] In view of the data heterogeneity problem existing in federated learning, the present invention preprocesses the local data of clients through data cleaning technology. At the same time, for clients with a small amount of data, a generative adversarial network is used to synthesize high-quality data to expand the diversity of the local data of clients. Then, the clients extract effective low-dimensional feature vectors of the local data through the encoder and the projection head and send them to the server. The server adaptively determines the parameters of the clustering algorithm, uses the DBSCAN clustering algorithm to cluster the low-dimensional feature vectors of the clients, and sends the clustering results to the clients. Finally, the clients perform the training process of the federated learning model in the form of clusters according to the clustering results until the model converges or reaches the specified number of communication rounds. The present invention can effectively protect the personal data privacy of clients, improve the clustering results, and enhance the training efficiency of the federated learning model by using a clustering method with adaptive parameter selection to cluster the feature vectors of the local data of clients. Description of the Drawings

[0037] Figure 1 It is a schematic flow chart of the present invention. Detailed Embodiments

[0038] The present invention will be described in detail below with reference to the drawings and embodiments:

[0039] As Figure 1 shown, the adaptive clustering federated learning method based on DBSCAN of the present invention includes the following steps:

[0040] A: In the data preprocessing stage, the server initializes the global model w 0and corresponding hyperparameters. Several clients preprocess their local datasets D i ={X i , Y i} to construct training sets and test sets after preprocessing.

[0041] Among them, the hyperparameters include the number of communication rounds T, the learning rate η, and the number of local update rounds E, which are used for the training of the federated learning model. When the clients perform preprocessing: First, the data of each client is cleaned to handle missing values, outliers, and duplicate values to ensure data integrity. Since the clients participating in federated learning may include multiple resource-constrained devices, resulting in insufficient local data volume, a pre-trained generative adversarial network is used to generate high-quality synthetic data to expand the diversity and complexity of the local data. Subsequently, the local data of the clients is standardized (for example, for image data, the local images of the clients are cropped to the same size) to facilitate model training. Finally, the clients divide the dataset into training sets and test sets according to a ratio.

[0042] In this embodiment, step A includes the following specific steps:

[0043] A1: The server initializes the global model

[0044] The server initializes the global model w 0 and information on the number of communication rounds T, the learning rate η, and the number of local update rounds E, and distributes them to all clients. The server is also called the coordinator, and the clients are also called the participants. The server can be a node with relatively strong computing power, responsible for coordinating and managing the global model training. The local clients can be mobile phones or tablets, which can train based on the local private datasets and upload the model update information to the server.

[0045] A2: The clients preprocess the local data

[0046] First, the data distributions of each client are inconsistent. In this embodiment, data cleaning techniques are used for all clients' local data. Specifically, for the local data of each client, it is necessary to handle its missing values, outliers, and duplicate values to ensure data integrity. Second, in view of the insufficient local data volume of the clients, StyleGAN-T is used to generate high-quality synthetic data. StyleGAN-T replaces the traditional CNN with a Transformer structure, introduces a multi-head self-attention mechanism (MHSA) in the generator to capture global features, and combines style mapping to control the image style, thereby achieving efficient and lightweight image generation, which is suitable for resource-constrained environments. Finally, the data of all clients is standardized and normalized to ensure meeting the requirements of the input of the federated learning model and enhancing the training efficiency of the federated learning model.

[0047] A3: The client divides the preprocessed data into a training set and a test set.

[0048] B: In the feature extraction stage, all clients use local data to train a model, obtain the feature vectors of the local data, and send them to the server.

[0049] Among them, the model mainly consists of a basic encoder and a projection head. The basic encoder is used to map the local data of the client to the same latent vector space and obtain the high-dimensional feature representation h of the local data i ; The projection head is used to map the high-dimensional feature h i to the low-dimensional feature z i . Finally, all clients upload their low-dimensional feature information z i for the subsequent clustering process. To avoid additional computational and storage overhead, we use a relatively simple convolutional neural network (CNN) architecture for the basic encoder and a multi-layer perceptron (MLP) architecture for the projection head.

[0050] In this embodiment, step B mainly consists of the following steps:

[0051] B1: Use an encoder for feature extraction;

[0052] At each client, the local data X i undergoes feature extraction through a shared encoder, mapping the original data of the client to a unified high-dimensional feature space. The encoder consists of multiple convolutional layers, non-linear transformation (ReLU), batch normalization (BatchNorm), pooling layer (Pooling), etc. The input data X i obtains the high-dimensional feature representation h after passing through this encoder i :

[0053] h i = f(X i ; θ) = GAP(σ(W c *X i + b))

[0054] where X i is the local data of the client, θ is the parameter of the encoder, * represents the convolution operation, W c and b are the weights and biases of the convolutional layer, σ(·) is the activation function (such as ReLU), and GAP (Global Average Pooling) is the average pooling operation, which is used to reduce the dimension and improve the stability of the features.

[0055] B2: Use a projection head for feature mapping;

[0056] To improve the distinguishability of features and avoid the influence of high-dimensional feature vectors on the clustering results, we add a Projection Head on the basis of the encoder to further map the high-dimensional feature vector h i into a low-dimensional embedding space z i . The MLP projection head consists of an input layer, a hidden layer, and an output layer. The input layer receives the high-dimensional feature h extracted by the encoder i . The hidden layer contains 2 to 3 fully connected networks (FC), followed by a ReLU activation function and BatchNorm for normalization after each layer. The output layer maps the features to a low-dimensional space and performs L2 normalization to improve the feature stability and separability, finally obtaining the low-dimensional feature z i :

[0057] z i = W3·σ(W2·σ(W1h i + b1)+ b2)+ b3

[0058] where W1, W2, and W3 are the weight matrices of the projection head; b1, b2, and b3 are the bias terms; σ(·) is a non-linear activation function (such as ReLU). Finally, we perform L2 normalization on the final output to obtain the low-dimensional feature z i :

[0059]

[0060] B3: All clients send the low-dimensional feature z i to the server for the subsequent clustering process. The low-dimensional feature z i is not easy to leak personal information and meets the privacy protection requirements in federated learning.

[0061] C: In the clustering stage, the server receives the low-dimensional features z of all clients i to form the feature matrix of all clients. Subsequently, the server adaptively determines the parameter information of the neighborhood radius (eps) and the minimum number of points (MinPts) in the DBSCAN clustering algorithm. Finally, the server uses the DBSCAN clustering algorithm to partition the clients and sends the clustering results to all clients.

[0062] Among them, the method for adaptively determining parameters is used to set appropriate parameter values. The DBSCAN clustering algorithm is sensitive to the selection of parameters. When the parameters are set unreasonably, the clustering results will also be affected. And using the DBSCAN clustering algorithm does not require pre-setting the number of clusters. It clusters by defining the density relationship between data points, with stronger flexibility and adaptability.

[0063] In the present invention, the specific steps of the method for adaptively determining parameters are as follows:

[0064] C1: First, the server calculates the first distance matrix between clients based on the Euclidean distance as follows:

[0065]

[0066] where z i and z j represent the feature vectors of clients C i and C j respectively.

[0067] C2: After obtaining the first distance matrix, the server calculates the first similarity matrix between clients based on the cosine similarity:

[0068]

[0069] Subsequently, set the similarity threshold ∈ = 0.5, set the values greater than or equal to ∈ to 1, and the values less than ∈ to 0. Finally, obtain the simplified second similarity matrix sim_matrix_new. The obtained second similarity matrix sim_matrix_new is a 0-1 matrix;

[0070] C3: Multiply the first distance matrix Dist_matrix by the second similarity matrix sim_matrix_new to obtain the second distance matrix Dist_matrix_new as follows:

[0071] Dist_matrix_new = Dist_matrix ⊙ sim_matrix_new

[0072] where ⊙ is the element-wise product, that is, multiply the corresponding positions.

[0073] C4: Statistically find the minimum value min_value and the maximum value max_value other than 0 in the second distance matrix Dist_matrix_new. Then, use the minimum value and the maximum value as the starting value and the ending value to divide m (m is the number of clients) intervals, and count the frequencies of the values in the second distance matrix Dist_matrix_new falling into each interval. Finally, use the left endpoint value of the interval with the highest frequency as the value of eps.

[0074] In the present invention, set the value of MinPts to 1 to ensure that each client participating in federated learning will not be identified as a noise point.

[0075] C5: After calculating the values of eps and MinPts, the server divides the clients based on the feature matrices of all clients using the DBSCAN clustering algorithm, grouping clients with similar data distributions into the same cluster. Here, similar data distributions mean that two or more data sets have a high degree of consistency or negligible differences in statistical characteristics. The clustering results are then sent to all clients.

[0076] D: In the federated learning model training phase, after all clients receive the clustering results, they conduct the federated learning model training process in the form of clusters until the model converges or reaches the specified number of communication rounds.

[0077] Among them, each cluster trains a local model and uploads the model update information to the server. The server aggregates the locally updated models of all clusters in a weighted average manner to form a new global model and distributes it to each cluster. The above process is repeated until the model converges or reaches the specified number of communication rounds.

[0078] In this embodiment, step D mainly consists of the following steps:

[0079] D1: The server initializes the global model and distributes it to all clients, and all clients train local models based on local data.

[0080] In the t-th round of training, after each client receives the global model, it trains its own local model using local data. Among them, each client C i optimizes the local model through gradient descent

[0081]

[0082] Among them, represents the model parameters of client C at the t-th round of training i , is the loss function of client C i , η is the learning rate, represents the model parameters of client C after update i .

[0083] D2: According to the clustering results, calculate the local shared model of each cluster and upload the updated model parameter information to the server.

[0084] In the t-th round of training, for each cluster G j , the local shared model of each cluster is obtained by weighted aggregation of the model parameter information of all clients within the cluster:

[0085]

[0086] Among them, is the local shared model for the j-th cluster, n k is the data volume of client k, is the total data volume within the cluster G j is the model parameter of the client belonging to the cluster G j After obtaining the local shared model of each cluster, upload its model parameter information to the server.

[0087] D3: The server weighted-aggregates the local shared models of all clusters and calculates to form a new round of global model.

[0088] In the t-th round of training, the server receives and integrates the local shared models of all clusters to form a new round of global model, as follows:

[0089]

[0090] where, w t+1 represents the new round of global model, N = ∑ j N j is the total data volume of all clients.

[0091] D4: The server distributes the new round of global model w t+1 to all clients, and repeats the above three steps until the model converges or reaches the specified number of communication rounds.

[0092] In summary, the present invention does not need to know the local data distribution of the clients. It uses the DBSCAN clustering algorithm to cluster the model parameter information of the clients, does not need to specify the number of clusters in advance, can discover clusters of any shape. At the same time, the present invention uses the clustering parameter adaptive selection method to improve the clustering accuracy to obtain better grouping. Since it does not need to know the local data distribution of the clients, it also ensures the confidentiality of the local data of the clients. Introducing the clustering algorithm into the federated learning framework effectively solves the data heterogeneity problem. By dividing similar clients into the same cluster, it speeds up the training speed of the federated learning model and improves the performance of the federated learning model.

[0093] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.​

Claims

1. An adaptive clustering federated learning method based on DBSCAN, characterized in that, It includes the following steps: A: The server initializes the global model and corresponding hyperparameters. Several clients preprocess their local datasets and then construct training sets and test sets. The hyperparameters include the number of communication rounds, the learning rate, and the number of local update rounds. B: All clients train the model using local data, obtain the feature vectors of the local data, and send them to the server. The model includes a basic encoder and a projection head. The basic encoder is used to map the local data of the client into the same latent vector space to obtain the high-dimensional feature representation of the local data. The projection head is used to map the high-dimensional features into low-dimensional features. C: The server receives the low-dimensional features of all clients to form the feature matrix of all clients. The server adaptively determines the values of the neighborhood radius and the minimum number of points to be included in the DBSCAN clustering algorithm, divides the clients through the DBSCAN clustering algorithm, and sends the clustering results to all clients. D: After receiving the clustering results, the clients perform the federated learning model training process in the form of clusters until the model converges or reaches the specified number of communication rounds.

2. The adaptive clustering federated learning method based on DBSCAN according to claim 1, wherein Step A includes the following specific steps: A1: The server initializes the global model, the number of communication rounds, the learning rate, and the number of local update rounds, and distributes them to all clients. A2: The client preprocesses the local data. A3: The client divides the preprocessed data into a training set and a test set.

3. The adaptive clustering federated learning method based on DBSCAN according to claim 1, characterized in that Step B includes the following specific steps: B1: Feature extraction is performed on the local data of each client through a shared encoder, and after mapping to a unified high-dimensional feature space, the high-dimensional feature representation is obtained. B2: The projection head receives the high-dimensional features extracted by the encoder, maps the features to a low-dimensional space, and obtains low-dimensional features after normalization. B3: All clients send the low-dimensional features to the server.

4. The adaptive clustering federated learning method based on DBSCAN according to claim 1, wherein In step C, the steps of adaptively determining the parameters are as follows: C1: The server calculates the first distance matrix between clients based on the Euclidean distance. C2: The server calculates the first similarity matrix between clients based on the cosine similarity, and simplifies it using a similarity threshold to obtain the second similarity matrix. C3: Multiply the first distance matrix by the second similarity matrix to obtain the second distance matrix. C4: Statistically calculate the minimum and maximum values other than 0 in the second distance matrix, then use the minimum and maximum values as the starting value and the ending value to divide m intervals, and count the frequencies of the values in the second distance matrix falling into each interval. Finally, use the left endpoint value of the interval with the highest frequency as the value of the neighborhood radius. Set the value of the minimum number of points to be included as 1. C5: Based on the values of the neighborhood radius and the minimum number of points to be included, the server divides the clients using the DBSCAN clustering algorithm based on the feature matrix of all clients, divides the clients with similar data distributions into the same cluster, and sends the clustering results to all clients.

5. The adaptive clustering federated learning method based on DBSCAN according to claim 1, characterized in that: In step D, each cluster trains a local model, uploads the model update information to the server, and the server aggregates the locally updated models of all clusters in a weighted average manner to form a new global model, and distributes it to each cluster. Repeat the above process until the model converges or reaches the specified number of communication rounds.

6. The adaptive clustering federated learning method based on DBSCAN according to claim 1, wherein Step D includes the following specific steps: D1: The server initializes the global model and distributes it to all clients, and all clients train local models based on local data; D2: According to the clustering results, calculate the local shared model of each cluster, and upload the updated model parameter information to the server; D3: The server weighted aggregates the local shared models of all clusters to calculate and form a new round of global model; D4: The server distributes the new round of global model to all clients, and repeats steps D1 to D3 until the model converges or reaches the specified number of communication rounds.

7. The adaptive clustering federated learning method based on DBSCAN according to claim 5, wherein: The number m of intervals is the same as the number of clients.

8. The adaptive clustering federated learning method based on DBSCAN according to claim 1, wherein: In step A2, first process the missing values, outliers and duplicate values in the local data of each client; for clients with insufficient local data volume, use StyleGAN-T to generate synthetic data.

9. The adaptive clustering federated learning method based on DBSCAN according to claim 3, wherein: The encoder adopts a convolutional neural network architecture, including multiple convolutional layers, non-linear transformation, batch normalization and pooling layers; the projection head adopts a multi-layer perceptron architecture, including an input layer, a hidden layer and an output layer.

10. The adaptive clustering federated learning method based on DBSCAN according to claim 4, wherein: According to the set similarity threshold, set the values in the first similarity matrix with similarity greater than or equal to the similarity threshold to 1, and the values less than the similarity threshold to 0, and finally obtain the simplified second similarity matrix.

Images