Clustering federal learning method based on cluster layer features

Abstract

The invention provides a clustering federated learning method based on cluster layer features. The method comprises the following steps: a client selects an optimal cluster model for local updating to obtain a feature matrix, and uploads the feature matrix and the cluster model to a server; the server clusters the clients according to the received feature matrix, divides the clients with similar data distribution into the same cluster model, and calculates a global model; and the server calculates the importance score of each layer in the cluster model by using the global model, each cluster model keeps a set number of layers with the highest importance score as personalized layers, and the rest layers except the personalized layers in each cluster model are aggregated according to the importance scores to obtain an updated cluster model. And the information is issued to each client. According to the invention, the clustering federal learning system based on the cluster layer features is realized, the personalized capability of the client is improved, and the performance of the model can be optimized and improved.

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology, and in particular to a clustering federated learning method based on cluster layer features. Background Art

[0002] With the development of science and technology, data has shown explosive growth. The development of machine learning and artificial intelligence cannot be separated from a large amount of data. However, the traditional centralized data training of a model has high requirements for hardware equipment, and because edge devices such as the Internet of Things and sensors need to be centrally uploaded to the server for model training, the user's privacy data cannot be protected. In order to protect data privacy and make better use of data information, federated learning methods have emerged. Users do not need to upload data to the server, but train on their own devices. For example, the FedAvg method uploads the trained model to the server for aggregation under the coordination of the central server, that is, multiple clients collaborate to train a shared global model by sharing model parameter information.

[0003] In practical applications, the data distribution of users often presents a non-independent and identically distributed (Non-IID) state, that is, the data of different users is unevenly distributed, resulting in slow model training. This data heterogeneity is still a key issue in federated learning. In order to solve this problem, clients with similar distributions are clustered, and clients with similar data distributions are divided into a cluster. These users collaborate and learn with each other, thereby improving the personalization capabilities of clients within the cluster without being disturbed by other clients with large differences in data distribution.

[0004] When the data distribution between clusters is very pathological, that is, the data distribution does not overlap at all, there is no need for clusters to collaborate in training. However, in the real world, there are often many complex scenarios, that is, the data distribution between clusters overlaps, and it is often not enough for the client to only aggregate and learn within the cluster. When the data distribution between clusters overlaps a lot, the knowledge of other users is not fully utilized, and collaboration between clusters is also required. Summary of the invention

[0005] An embodiment of the present invention provides a clustering federated learning method based on cluster layer features to implement clustering federated learning based on cluster layer features that provides a personalized model for a client.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme.

[0007] According to one aspect of the present invention, a clustering federated learning method based on cluster layer features is provided, comprising:

[0008] The client selects the optimal cluster model, uses the cluster model to perform local updates to obtain a feature matrix, and uploads the feature matrix and the cluster model to the server;

[0009] The server clusters the clients according to the received feature matrix, divides the clients with similar data distribution into the same cluster model, and calculates a global model according to all the cluster models;

[0010] The server calculates the importance score of each layer in the cluster model using the global model, retains a set number of layers with the highest importance scores in each cluster model as personalized layers, and aggregates the remaining layers in each cluster model except the personalized layers according to the importance scores to obtain an updated cluster model;

[0011] The server sends the updated cluster model to each client belonging to the cluster model.

[0012] Preferably, the client selects the optimal cluster model, uses the cluster model to perform local update to obtain a feature matrix, and uploads the feature matrix and the cluster model to the server, including:

[0013] Each client collects multiple cluster models sent from the server, trains all cluster models on the client's local data set, obtains the loss values ​​of the corresponding cluster models, sorts these losses, and selects the cluster model with the minimum loss value as the cluster model to which the client belongs;

[0014] The client uses a public data set to perform forward propagation on the cluster model to which the client belongs, and obtains a feature matrix representing the data feature information of the client. The feature matrix is ​​used to extract feature information. Each client uploads the selected cluster model and the calculated feature matrix to the server.

[0015] Preferably, each client collects multiple cluster models sent from the server, trains all cluster models on the local data set of the client, obtains loss values ​​of corresponding cluster models, sorts these losses, and selects the cluster model with the minimum loss value as the cluster model to which the client belongs, including:

[0016] In a federated learning system instance, there are a total of K clients, where each client k holds a private local dataset where |x k |=N k represents the sample size of the dataset of client k, Indicates the dataset size of all clients. The datasets of different clients show different data distributions.

[0017] The client receives all cluster models sent by the server as follows: t represents the tth communication round, client k uses the private dataset D k Update the cluster model with small batches of data in:

[0018]

[0019] in, Indicates private data D k In the model The predicted output on ;

[0020] α represents the learning rate;

[0021] Indicates that the customer is in the cluster model The loss function on ;

[0022] According to the loss function Calculate the loss value on each cluster model and select the cluster model with the smallest loss value As the optimal cluster model to which the client belongs, the number i of the optimal cluster model is recorded, indicating that the client k belongs to the i-th cluster.

[0023] Preferably, the server clusters the clients according to the received feature matrix, divides the clients with similar data distribution into the same cluster model, and calculates a global model according to all cluster models, including:

[0024] The server selects a specified clustering algorithm based on the received feature matrix and uses the K-Means method to cluster the clients, and divides the clients with similar data distribution into the same cluster model, thereby obtaining C cluster models;

[0025] The model weight of each client is proportional to the sample size of its local training data. The server calculates the corresponding weight based on the sample size of each client, performs federated averaging on all clients within each cluster model, and obtains the model parameters of the cluster model. Perform federated averaging on all cluster model parameters to obtain the global model ω t .

[0026] Preferably, the server calculates the importance score of each layer in the cluster model using the global model, and each cluster model retains the layers with the highest importance scores of the first set number as personalized layers, and aggregates the remaining layers in each cluster model except the personalized layers according to the importance scores to obtain an updated cluster model, including:

[0027] Calculate the importance parameters of each layer in the cluster model

[0028]

[0029] in, Represents the global model ω t The number of parameters in the first layer of , L represents the number of layers in the model, Representation cluster model The lth layer and the global model ω t The distance metric of the lth layer is The larger the value, the farther away from the average knowledge, and the higher the degree of personalization at this level;

[0030] For cluster model i, select the layer importance score The first T layers with high values ​​are used as personalization layers. The personalization layers do not participate in the communication and collaboration between clusters. The remaining layers in each cluster model except the personalization layer are aggregated in units of model layers according to the following formula:

[0031]

[0032] in, Representation cluster model The normalized weight of the l-th layer importance score;

[0033] The weights of the cluster models are weighted and aggregated layer by layer to obtain the updated cluster model i.

[0034] According to another aspect of the present invention, there is provided a clustering federated learning system based on cluster layer features, comprising: a client and a server;

[0035] The client is used to select an optimal cluster model, perform local update using the cluster model to obtain a feature matrix, and upload the feature matrix and the cluster model to a server;

[0036] The server is used to cluster the clients according to the received feature matrix, divide the clients with similar data distribution into the same cluster model, and calculate the global model according to all the cluster models; use the global model to calculate the importance score of each layer in the cluster model, each cluster model retains the layers with the highest importance scores of the first set number as personalized layers, and aggregate the remaining layers in each cluster model except the personalized layers according to the importance scores to obtain an updated cluster model;

[0037] The updated cluster model is sent to each client belonging to the cluster model.

[0038] Preferably, the client is further used to collect multiple cluster models sent from the server, train all cluster models on the local data set of the client, obtain loss values ​​of corresponding cluster models, sort these losses, and select the cluster model with the minimum loss value as the cluster model to which the client belongs;

[0039] Use the public data set to perform forward propagation on the cluster model to which the client belongs to obtain a feature matrix, which is used to extract feature information. Each client uploads the selected cluster model and the calculated feature matrix to the server;

[0040] In a federated learning system instance, there are a total of K clients, where each client k holds a private local dataset where |x k |=N k represents the sample size of the dataset of client k, Indicates the dataset size of all clients. The datasets of different clients show different data distributions.

[0041] The client receives all cluster models sent by the server as follows: t represents the tth communication round, client k uses the private dataset D k Update the cluster model with small batches of data in:

[0042]

[0043] in, Indicates private data D k In the model The predicted output on ;

[0044] α represents the learning rate;

[0045] Indicates that the customer is in the cluster model The loss function on ;

[0046] According to the loss function Calculate the loss value on each cluster model and select the cluster model with the smallest loss value As the optimal cluster model to which the client belongs, the number i of the optimal cluster model is recorded, indicating that the client k belongs to the i-th cluster.

[0047] Preferably, the server is further used to select a specified clustering algorithm according to the received feature matrix to cluster the clients using the K-Means method, and divide the clients with similar data distribution into the same cluster model, thereby obtaining C cluster models;

[0048] The model weight of each client is proportional to the sample size of its local training data. The server calculates the corresponding weight based on the sample size of each client, performs federated averaging on all clients within each cluster model, and obtains the model parameters of the cluster model. Perform federated averaging on all cluster model parameters to obtain the global model ω t .

[0049] Preferably, the server is also used to calculate the importance parameter of each layer in the cluster model

[0050]

[0051] in, Represents the global model ω t The number of parameters in the lth layer of , L represents the number of layers in the model, Representation cluster model The lth layer and the global model ω t The distance metric of the lth layer is The larger the value, the farther away from the average knowledge, and the higher the degree of personalization at this level;

[0052] For cluster model i, select the layer importance score The first T layers with high values ​​are used as personalization layers. The personalization layers do not participate in the communication and collaboration between clusters. The remaining layers in each cluster model except the personalization layer are aggregated in units of model layers according to the following formula:

[0053]

[0054] in, Representation cluster model The normalized weight of the l-th layer importance score;

[0055] The weights of the cluster models are weighted and aggregated layer by layer to obtain the updated cluster model i.

[0056] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the present invention realizes a clustering federated learning system based on cluster layer features, which not only improves the personalization ability of the client, but also optimizes and improves the performance of the model, so that each client can not only learn from each other within the cluster, but also communicate and collaborate with each other between clusters. This method has important application value in the task of processing and learning large-scale distributed data.

[0057] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0059] Figure 1 A processing flow chart of a clustering federated learning method based on cluster layer features provided in an embodiment of the present invention.

[0060] Figure 2 A schematic diagram of a problem existing in cluster internal aggregation provided by an embodiment of the present invention.

[0061] Figure 3 A schematic diagram of a cluster model provided in an embodiment of the present invention for clustering based on features. DETAILED DESCRIPTION

[0062] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0063] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0064] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0065] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0066] The embodiment of the present invention provides a processing flow of a personalized federated learning method based on cluster model layer feature clustering. Figure 1 As shown, the following processing steps are included:

[0067] Step S0: The client performs local update to select the optimal cluster model, obtains a feature matrix, and uploads the feature matrix and the cluster model to the server.

[0068] Step S1: The server clusters the clients according to the feature matrix, and the clients classified into a cluster have more similar data distribution.

[0069] Step S2: The server integrates the global model and the cluster model;

[0070] Step S3, calculating layer importance parameters, retaining only the layer with the highest importance score as the personalized layer for each cluster model, and aggregating the remaining layers according to the importance scores to obtain a new cluster model;

[0071] Step S4: Update cluster model parameters and broadcast them to the client;

[0072] Preferably, the above step S0 includes: each client collects multiple cluster models sent from the server, trains all cluster models on the client's local data set, obtains loss values ​​of corresponding cluster models, sorts these losses, and selects the cluster model with the minimum loss value as the cluster model to which the client belongs;

[0073] The client uses a public data set to perform forward propagation on the cluster model to which the client belongs, and obtains a feature matrix representing the data feature information of the client. The feature matrix is ​​used to extract feature information, and the cluster model and feature matrix calculated on the client are uploaded to the server.

[0074] Forward propagation is the process of passing data from the input layer to the output layer in a neural network. In this process, the output of each hidden layer is regarded as a feature representation. These feature representations capture the information of the data after being processed in the neural network. The input data is calculated by each layer of the neural network, and finally a feature matrix is ​​obtained. The rows of the feature matrix represent different samples, and each row corresponds to a data point; the columns of the feature matrix represent different features or attributes, and each column corresponds to a feature; the feature matrix represents the data feature information of the client. Through these features, the model can learn the data distribution and characteristics of the client without directly sharing the original data.

[0075] Preferably, the above step S1 includes: the server selects a specified clustering algorithm based on the received feature matrix and clusters the clients using the K-Means method, thereby dividing clients with similar data distribution into one cluster.

[0076] Preferably, the above step S2 includes: the server has been divided into clusters in the previous step, each cluster contains clients with similar data distribution, for one of the clusters i, a federated average operation is performed on all clients in the cluster according to the sample size, the model weight of each client is proportional to the sample size of its local training data, the corresponding weight is calculated according to the sample size of each client, the federated average operation is performed, mutual learning is performed within the cluster, the performance and generalization ability of the model within the cluster are improved, and the cluster model is obtained. The sample size refers to the amount of data that each client trains locally. The model weight of each client is proportional to the size of its local data, to ensure that clients with large amounts of data have a greater impact on the global model. The feature matrix sent by the client to the server contains sample size information.

[0077] Federate all cluster models to get the global model ω t , where t represents the tth communication round;

[0078] The structures of cluster models and global models are usually the same, that is, they have the same network structure, number of layers and number of neurons, which helps to ensure that the model parameters can be effectively weighted averaged during the aggregation process. Cluster models and global models usually contain many layers, usually a multi-layer neural network model, including an input layer, multiple hidden layers and an output layer; each layer contains a group of neurons, each neuron has associated weights and biases, which are used to extract features from the data; for example, the commonly used convolutional neural network includes an input layer for receiving image data, multiple convolutional layers and pooling layers for extracting image features, a fully connected layer for feature mapping, and an output layer for outputting the results of category prediction. These layers can be designed according to specific application requirements and network structures.

[0079] Preferably, the above step S3 includes: calculating the importance parameter of each layer of the cluster model Representation cluster model The lth layer and the global model ω t The distance metric of the lth layer is The larger the value, the further it deviates from the global model, and the higher the personalization level of this layer is, which is used to measure the personalization level of layer l. For each cluster model, aggregation is performed based on the model layer, that is, the personalization layer is not aggregated, and the personalization layer in the cluster is retained. This method has a finer aggregation granularity and can extract more required information than the aggregation of the entire model.

[0080] For cluster model i, the first T layers with high layer importance scores are selected as personalized layers, which do not participate in the communication and collaboration between clusters; the remaining layers are aggregated according to the importance scores, and the weight of the cluster model is weighted layer by layer. Then each cluster retains its own personalized layer and collaborates with other clusters, thus obtaining a new and more generalized cluster model i;

[0081] Preferably, the above step S4 includes: updating the cluster model parameters and sending them to the client, including: overwriting the previous round of cluster model with the new cluster model, and then updating the cluster model parameters It is sent to each client belonging to the cluster model to conduct a new round of model training.

[0082] Example 1

[0083] In embodiment 1 of the present invention, a clustering federated learning method based on cluster layer features is provided, comprising the following processing steps:

[0084] Step S0: The client selects the optimal cluster model for local training and uploads the model: In this step, the client receives all cluster models sent by the server. Where t represents the tth communication round. A cluster with the greatest correlation with itself is selected for training, and the feature matrix and model are uploaded to the server so that the cluster model can obtain client knowledge with similar data distribution.

[0085] In a federated learning system instance, there are a total of K clients, where each client k holds a private local dataset where |x k |=N k represents the sample size of the dataset of client k, Represents the dataset size of all clients. The datasets of different clients present different data distributions, which conforms to the non-independent and identically distributed (Non-IID) characteristic.

[0086] If it is initial training, that is, communication round t=0, the server will initialize C cluster models and send these C cluster models to all clients; if it is subsequent training, that is, communication round t>0, the server will send the C cluster models updated in the previous step to all clients.

[0087] First, client k receives the cluster model Using private dataset D k Update the model with small batches of data in:

[0088]

[0089] in, Indicates private data D k In the model The predicted output on ;

[0090] α represents the learning rate;

[0091] Indicates that the customer is in the cluster model The loss function on ;

[0092] According to the loss function Calculate the loss value on each cluster model and select the cluster model with the smallest loss value As the optimal cluster to which the client belongs, it indicates that the cluster has the greatest degree of association with the client, and records the cluster number i, indicating that client k belongs to the i-th cluster;

[0093] Second, client k uses the public dataset to select the optimal cluster model Perform forward propagation on it, and after calculation and processing by the model, obtain the feature matrix;

[0094] Finally, the client sends the optimal cluster model obtained above and feature matrix are uploaded to the server;

[0095] Step S1, clustering the clients according to the feature matrix: the server selects an appropriate clustering algorithm to cluster the feature matrix of the clients. In the example of the present invention, the commonly used unsupervised learning K-Means algorithm is selected as the clustering algorithm, and clients with similar features are divided into a cluster, thereby obtaining C clusters.

[0096] Step S2: Fusion of global model and cluster model: The clients within each cluster are considered to have more similar data distribution. Federated averaging is performed on all clients within the cluster, and mutual learning is performed within the cluster to improve the performance and generalization ability of the model within the cluster, and the model of the cluster is obtained. Perform federated averaging on all clusters to obtain the global model ω t .

[0097] Step S3, calculate layer importance scores and aggregate: the server calculates the layer importance scores of each cluster model, and selects the first T layers with high scores as personalized layers according to the layer importance score values, indicating that these T layers retain more feature information of the cluster. When aggregating between clusters, the personalized layers are retained, and other layers are aggregated according to weights to obtain a balance between the cluster and outside.

[0098] The present invention is aimed at the problem of having similar distributions in different clusters. Figure 2This is a schematic diagram of the problem of internal aggregation in a cluster according to an embodiment of the present invention. When the data distribution of the client is pathologically distributed, that is, when the distributions of different clients do not overlap, Figure 2 There are three distributions on the left side of the figure. The data distributions in the three clusters do not overlap. Clients assigned to a cluster can learn from each other, so there is no need for communication between clusters. However, in real data, clients in different clusters also have similar data distributions. The three distributions on the right side of the figure overlap. Therefore, in addition to learning within clusters, communication and collaboration between clusters is also required.

[0099] The embodiment of the present invention provides a clustering federated learning method based on cluster layer features, where each cluster model The importance score of each layer is calculated according to the following formula

[0100]

[0101] in, Represents the global model ω t The number of parameters in the lth layer;

[0102] L represents the number of levels the model has;

[0103] Representation cluster model The lth layer and the global model ω t The distance metric of the lth layer; it measures the cluster model The personalization degree of the parameters of the first layer, the larger its value, the farther it deviates from the average knowledge, and the higher the personalization degree of this layer;

[0104] The embodiment of the present invention provides a clustering federated learning method based on cluster layer features, where each cluster model After obtaining the importance scores of the model layers, the first T layers with the highest importance scores are selected as personalized layers without aggregation to retain the personalized features in the cluster. For the remaining layers, the model layer is aggregated according to the following formula:

[0105]

[0106] in, Representation cluster model The normalized weight of the l-th layer importance score;

[0107] Figure 3This is a schematic diagram of a cluster model according to an embodiment of the present invention being aggregated according to layer importance. The schematic diagram shows the network structure of each cluster model. For each cluster model, the layer importance score of the corresponding model is calculated according to the above formula, and the layer with the highest layer importance score is retained as the personalized layer, which does not participate in the aggregation. The remaining layers are aggregated between model layers according to the normalized weights of the layer importance scores; each cluster retains the personalized layer, which does not participate in the communication and collaboration between clusters, and the remaining layers are aggregated according to the importance scores, and a new cluster model is obtained by weighted aggregation layer by layer;

[0108] Step S4: Update cluster model parameters and issue them: After the above steps are aggregated, the new cluster model is And distribute it to all clients to start the next round of communication training until the model converges;

[0109] In summary, the present invention utilizes the importance information of the cluster model layer, retains the personalized layer in the model, and aggregates other layers, so as to conduct communication and learning between clusters. It can better adapt to the data distribution in the real world, and the cluster model can converge and generalize better. At the same time, it has better personalization capabilities. Through learning aggregation between clusters, it provides customers with a more personalized model and achieves a balance between the inside and outside of the cluster.

[0110] Example 2

[0111] In embodiment 2 of the present invention, a clustering federated learning system based on cluster layer features is provided, comprising the following processing modules: including: a client and a server;

[0112] The client is used to select an optimal cluster model, perform local update using the cluster model to obtain a feature matrix, and upload the feature matrix and the cluster model to a server;

[0113] The server is used to cluster the clients according to the received feature matrix, divide the clients with similar data distribution into the same cluster model, and calculate the global model according to all the cluster models; use the global model to calculate the importance score of each layer in the cluster model, each cluster model retains the layers with the highest importance scores of the first set number as personalized layers, and aggregate the remaining layers in each cluster model except the personalized layers according to the importance scores to obtain an updated cluster model;

[0114] The updated cluster model is sent to each client belonging to the cluster model.

[0115] Preferably, the client is further used to collect multiple cluster models sent from the server, train all cluster models on the local data set of the client, obtain loss values ​​of corresponding cluster models, sort these losses, and select the cluster model with the minimum loss value as the cluster model to which the client belongs;

[0116] Use the public data set to perform forward propagation on the cluster model to which the client belongs to obtain a feature matrix, which is used to extract feature information. Each client uploads the selected cluster model and the calculated feature matrix to the server;

[0117] In a federated learning system instance, there are a total of K clients, where each client k holds a private local dataset where |x k |=N k represents the sample size of the dataset of client k, Indicates the dataset size of all clients. The datasets of different clients show different data distributions.

[0118] The client receives all cluster models sent by the server as follows: t represents the tth communication round, client k uses the private dataset D k Update the cluster model with small batches of data in:

[0119]

[0120] in, Indicates private data D k In the model The predicted output on ;

[0121] α represents the learning rate;

[0122] Indicates that the customer is in the cluster model The loss function on ;

[0123] According to the loss function Calculate the loss value on each cluster model and select the cluster model with the smallest loss value As the optimal cluster model to which the client belongs, the number i of the optimal cluster model is recorded, indicating that the client k belongs to the i-th cluster.

[0124] Preferably, the server is further used to select a specified clustering algorithm according to the received feature matrix to cluster the clients using the K-Means method, and divide the clients with similar data distribution into the same cluster model, thereby obtaining C cluster models;

[0125] Perform federated averaging on all clients within each cluster model to obtain the model parameters of the cluster model. Perform federated averaging on all cluster model parameters to obtain the global model ω t .

[0126] Preferably, the server is also used to calculate the importance parameter of each layer in the cluster model

[0127]

[0128] in, Represents the global model ω t The number of parameters in the lth layer of , L represents the number of layers in the model, Representation cluster model The lth layer and the global model ω t The distance metric of the lth layer is The larger the value, the farther away from the average knowledge, and the higher the degree of personalization at this level;

[0129] For cluster model i, select the layer importance score The first T layers with high values ​​are used as personalization layers. The personalization layers do not participate in the communication and collaboration between clusters. The remaining layers in each cluster model except the personalization layer are aggregated in units of model layers according to the following formula:

[0130]

[0131] in, Representation cluster model The normalized weight of the l-th layer importance score;

[0132] The weights of the cluster models are weighted and aggregated layer by layer to obtain the updated cluster model i.

[0133] The specific process of using the system of the embodiment of the present invention to perform the clustering federated learning method based on cluster layer features is similar to the aforementioned method embodiment and will not be repeated here.

[0134] In this embodiment, the above-mentioned system is used to implement a clustering federated learning method based on cluster layer features, thereby improving the personalization capability of the client and the performance of the model.

[0135] Example 3

[0136] In embodiment 3 of the present invention, a distributed computer program is provided. The program adopts a distributed computing architecture, including client nodes and server nodes, which communicate and compute via a network.

[0137] Client nodes refer to individual devices or computers that participate in federated learning, such as mobile devices such as mobile phones and tablets, embedded devices such as the Internet of Things and sensors, and personal computer devices such as computers. These nodes usually have limited computing and storage resources. Local updates, feature extraction, and model parameter transmission are performed on these client nodes;

[0138] Local update: The client node uses the local data set to train and update the cluster model, and selects the optimal cluster model as the cluster to which the client node belongs;

[0139] Feature extraction: The client node performs forward propagation on the optimal clustering model using the public dataset to obtain the feature matrix;

[0140] Model parameter transmission: The client node transmits the model parameters and feature matrix obtained through local training to the server node for fusion and aggregation;

[0141] Server nodes are used to coordinate and manage central computers and clusters in the federated learning process. They have multi-core processors and large memory capacity. Server nodes have high computing and storage capabilities. Server nodes are used to perform model initialization, clustering operations, model fusion and aggregation, and model distribution.

[0142] Model initialization: The server node is responsible for initializing the cluster model;

[0143] Clustering operation: The server node receives the feature matrix extracted by the client node and uses a clustering algorithm to divide the client nodes into different clusters;

[0144] Model fusion and aggregation: The server node collects the model parameters of the clients in each cluster and calculates the model parameters of each cluster and the global model parameters using methods such as federated averaging;

[0145] Model distribution: The server node distributes the updated global model parameters to each client node so that they can use them in the next round of local updates;

[0146] The client and server use a higher network bandwidth to communicate to support large-scale data transmission and model parameter synchronization. Through network transmission, the client node uploads the local model and feature matrix to the server node, and the server node sends the cluster model parameters to the client node. During the transmission process, the model parameters remain synchronized, improving the computing efficiency and the overall performance of the system.

[0147] In summary, the embodiments of the present invention extract the feature matrix of the customer model, cluster the clients with similar data distribution according to the traditional classic clustering algorithm, obtain the cluster model, calculate the importance score of the cluster model layer, classify them according to the score, retain the layer with high importance score as the personalized layer, and aggregate the other layers, so as to achieve aggregate communication between clusters, thereby effectively improving the performance and generalization ability of the client model, and being more adaptable to non-independent and identically distributed data in the real world, so as to have better personalization capabilities.

[0148] The embodiment of the present invention provides a clustering federated learning method and system based on cluster layer features that utilizes the correlation between the cluster and the central server model, allows the cluster model to retain the personalized layer locally, and aggregates the remaining layers according to the importance scores, thereby providing customers with a personalized model.

[0149] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0150] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0151] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0152] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A clustering federated learning method based on cluster layer features, characterized in that: include: The client selects the optimal cluster model, uses the cluster model to perform local updates to obtain a feature matrix, and uploads the feature matrix and the cluster model to the server; The server clusters the clients according to the received feature matrix, divides the clients with similar data distribution into the same cluster model, and calculates a global model according to all the cluster models; The server calculates the importance score of each layer in the cluster model using the global model, retains a set number of layers with the highest importance scores in each cluster model as personalized layers, and aggregates the remaining layers in each cluster model except the personalized layers according to the importance scores to obtain an updated cluster model; The server sends the updated cluster model to each client belonging to the cluster model.

2. The method according to claim 1, characterized in that The client selects the optimal cluster model, uses the cluster model to perform local update to obtain a feature matrix, and uploads the feature matrix and the cluster model to the server, including: Each client collects multiple cluster models sent from the server, trains all cluster models on the client's local data set, obtains the loss values ​​of the corresponding cluster models, sorts these losses, and selects the cluster model with the minimum loss value as the cluster model to which the client belongs; The client uses a public data set to perform forward propagation on the cluster model to which the client belongs, and obtains a feature matrix representing the data feature information of the client. The feature matrix is ​​used to extract feature information. Each client uploads the selected cluster model and the calculated feature matrix to the server.

3. The method according to claim 2, characterized in that Each client collects multiple cluster models sent from the server, trains all cluster models on the client's local data set, obtains loss values ​​of corresponding cluster models, sorts these losses, and selects the cluster model with the minimum loss value as the cluster model to which the client belongs, including: In a federated learning system instance, there are a total of K clients, where each client k holds a private local dataset where |x k |=N k represents the sample size of the dataset of client k, Indicates the dataset size of all clients. The datasets of different clients show different data distributions. The client receives all cluster models sent by the server as follows: t represents the tth communication round, client k uses the private dataset D k Update the cluster model with small batches of data in: in, Indicates private data D k In the model The predicted output on ; α represents the learning rate; Indicates that the customer is in the cluster model The loss function on ; According to the loss function Calculate the loss value on each cluster model and select the cluster model with the smallest loss value As the optimal cluster model to which the client belongs, the number i of the optimal cluster model is recorded, indicating that the client k belongs to the i-th cluster.

4. The method according to claim 1, characterized in that: The server clusters the clients according to the received feature matrix, divides the clients with similar data distribution into the same cluster model, and calculates a global model according to all cluster models, including: The server selects a specified clustering algorithm based on the received feature matrix and uses the K-Means method to cluster the clients, and divides the clients with similar data distribution into the same cluster model, thereby obtaining C cluster models; The model weight of each client is proportional to the sample size of its local training data. The server calculates the corresponding weight based on the sample size of each client, performs federated averaging on all clients within each cluster model, and obtains the model parameters of the cluster model. Perform federated averaging on all cluster model parameters to obtain the global model ω t .

5. The method according to claim 4, characterized in that The server uses the global model to calculate the importance score of each layer in the cluster model, and each cluster model retains the layers with the highest importance scores as the personalized layers. The remaining layers in each cluster model except the personalized layers are aggregated according to the importance scores to obtain an updated cluster model, including: Calculate the importance parameters of each layer in the cluster model in, Represents the global model ω t The number of parameters in the lth layer of , L represents the number of layers in the model, Representation cluster model The lth layer and the global model ω t The distance metric of the lth layer is The larger the value, the farther away from the average knowledge, and the higher the degree of personalization at this level; For cluster model i, select the layer importance score The first T layers with high values ​​are used as personalization layers. The personalization layers do not participate in the communication and collaboration between clusters. The remaining layers in each cluster model except the personalization layer are aggregated in units of model layers according to the following formula: in, Representation cluster model The normalized weight of the l-th layer importance score; The weights of the cluster models are weighted and aggregated layer by layer to obtain the updated cluster model i.

6. A clustering federated learning system based on cluster layer features, characterized in that: include: Client and server; The client is used to select an optimal cluster model, perform local update using the cluster model to obtain a feature matrix, and upload the feature matrix and the cluster model to a server; The server is used to cluster the clients according to the received feature matrix, divide the clients with similar data distribution into the same cluster model, and calculate the global model according to all the cluster models; use the global model to calculate the importance score of each layer in the cluster model, each cluster model retains the layers with the highest importance scores of the first set number as personalized layers, and aggregate the remaining layers in each cluster model except the personalized layers according to the importance scores to obtain an updated cluster model; The updated cluster model is sent to each client belonging to the cluster model.

7. The system according to claim 6, characterized in that: The client is further used to collect multiple cluster models sent from the server, train all cluster models on the local data set of the client, obtain loss values ​​of the corresponding cluster models, sort these losses, and select the cluster model with the minimum loss value as the cluster model to which the client belongs; Use the public data set to perform forward propagation on the cluster model to which the client belongs to obtain a feature matrix, which is used to extract feature information. Each client uploads the selected cluster model and the calculated feature matrix to the server; In a federated learning system instance, there are a total of K clients, where each client k holds a private local dataset where |x k |=N k represents the sample size of the dataset of client k, Indicates the dataset size of all clients. The datasets of different clients show different data distributions. The client receives all cluster models sent by the server as follows: t represents the tth communication round, client k uses the private dataset D k Update the cluster model with small batches of data in: in, Indicates private data D k In the model The predicted output on ; α represents the learning rate; Indicates that the customer is in the cluster model The loss function on ; According to the loss function Calculate the loss value on each cluster model and select the cluster model with the smallest loss value As the optimal cluster model to which the client belongs, the number i of the optimal cluster model is recorded, indicating that the client k belongs to the i-th cluster.

8. The system according to claim 7, characterized in that: The server is further used to select a specified clustering algorithm based on the received feature matrix to cluster the clients using the K-Means method, and to classify clients with similar data distribution into the same cluster model, thereby obtaining C cluster models; The model weight of each client is proportional to the sample size of its local training data. The server calculates the corresponding weight based on the sample size of each client, performs federated averaging on all clients within each cluster model, and obtains the model parameters of the cluster model. Perform federated averaging on all cluster model parameters to obtain the global model ω t .

9. The system according to claim 8, characterized in that: The server is also used to calculate the importance parameters of each layer in the cluster model in, Represents the global model ω t The number of parameters in the lth layer of , L represents the number of layers in the model, Representation cluster model The lth layer and the global model ω t The distance metric of the lth layer is The larger the value, the farther away from the average knowledge, and the higher the degree of personalization at this level; For cluster model i, select the layer importance score The first T layers with high values ​​are used as personalization layers. The personalization layers do not participate in the communication and collaboration between clusters. The remaining layers in each cluster model except the personalization layer are aggregated in units of model layers according to the following formula: in, Representation cluster model The normalized weight of the l-th layer importance score; The weights of the cluster models are weighted and aggregated layer by layer to obtain the updated cluster model i.

10. A distributed computer program, using a distributed computing architecture, comprising: The client nodes and the server nodes communicate and calculate through the network to implement the clustering federated learning method based on cluster layer features as described in any one of claims 1-5.

Images