Clustering federal learning method and system based on prediction mechanism and electronic equipment
Through the prediction mechanism of multi-center clustering and dynamic parameter fusion, the problems of slow convergence and dynamic changes of clients in federated learning in non-IID environments are solved, and efficient collaborative learning of models in fields such as the Internet of Things is achieved.
Patent Information
- Application Number
- CN202510821334.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-19
- Publication Date
- 2025-10-03
AI Technical Summary
Existing federated learning methods perform poorly in non-IID environments, have slow convergence speeds, and are difficult to adapt to dynamic changes in clients and the addition of new clients, especially in applications in fields such as the Internet of Things.
A clustering federated learning method based on prediction mechanism is adopted. Through multi-center clustering and dynamic parameter fusion, the random expectation maximization algorithm is used to update the cluster allocation and cluster center parameters. Combined with partial client selection strategy and dynamic weight fusion, the model training process is optimized.
The adaptability and convergence speed of the model in non-IID environments are improved, the adaptability to new clients is enhanced, and the generalization ability and performance of the model are improved.
Smart Images
Figure CN120745874A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed machine learning technology, and in particular to a clustering federated learning method, system, and electronic device based on a prediction mechanism. Background Art
[0002] Federated learning, a distributed machine learning technology that protects user privacy, has been widely used in fields such as the Internet of Things and healthcare. However, in practice, non-IID characteristics such as large variations in client data distribution and uneven training participation lead to poor performance and slow convergence of traditional federated learning methods.
[0003] Existing technologies, such as FedAvg and FedProx, use single global models that perform poorly in non-IID environments. While K-Means-based multi-center clustering methods (such as FeSEM) have shown improvement, they rely too heavily on all clients participating in training and struggle to adapt to newly added clients, limiting deployment efficiency. A robust federated learning approach is urgently needed that can accommodate partial client participation and support the dynamic addition of new clients.
[0004] After searching, Chinese invention patent application publication number CN113033820B discloses a federated learning method, apparatus, and device. The method includes: determining multiple client groups obtained by clustering based on similarity between clients, and a representative client in the client group, wherein the client is used on an IoT device; sending a training request to the client group via the representative client; receiving training data for a portion of the clients in the client group fed back by the representative client, wherein the portion of the clients is determined based on the client status using a pre-trained elastic upload prediction model. Each client in the client group uses the pre-trained elastic upload prediction model to determine whether it is suitable for data upload under its current client status. The portion of the clients refers to the clients that are currently eligible for training data upload as predicted by the elastic upload prediction model. The client status includes the client's service request frequency, the resource utilization rate of the IoT device where it is located, and the required training data; determining gradient data using the training data, and sending the gradient data to the representative client to enable the clients in the client group to perform gradient updates. This existing patent application suffers from poor adaptability to dynamic client changes and slow convergence.
[0005] How to realize federated learning with dynamic client participation has become a technical problem that needs to be solved. Summary of the Invention
[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a clustering federated learning method, system and electronic device based on a prediction mechanism.
[0007] The purpose of the present invention can be achieved by the following technical solutions:
[0008] According to one aspect of the present invention, a clustering federated learning method based on a prediction mechanism is provided, the method comprising:
[0009] Configure the client and server architecture and initialize the local data distribution of each client;
[0010] Cluster the clients using a multi-center clustering mechanism and determine K cluster centers.
[0011] Use the randomized expectation maximization algorithm to update the cluster assignment and cluster center global parameters;
[0012] Based on the clustering results, the model is trained through some client selection strategies in each of the K cluster centers to update the local model parameters.
[0013] Dynamically fuse the cluster center global parameters with the local model parameters to form fusion parameters;
[0014] Use the fusion parameters to perform prediction evaluation on the local test set and output the optimized multi-center clustering model.
[0015] Preferably, the multi-center clustering mechanism includes:
[0016] Perform initial clustering of clients using the K-MEANS algorithm to determine K cluster centers;
[0017] By calculating the cosine similarity between the client's local model parameters and each cluster center, the optimal cluster assignment is determined and the clustering results are obtained;
[0018] The clustering results and the client's local model parameters are passed to the server for global parameter aggregation.
[0019] Preferably, the partial client selection strategy includes cluster ratio sampling, round interval evaluation and new client adaptation mechanism.
[0020] More preferably, the cluster ratio sampling is specifically: randomly selecting a certain proportion of clients from each cluster center to form a training set sc k , the calculation formula is:
[0021]
[0022] Among them, C is the selected proportional coefficient, represents the number of clients in the kth cluster.
[0023] More preferably, the round interval evaluation is specifically as follows: calculating the interval rounds of the client participating in training nt=t-lt, where t is the current round and lt is the round of the most recent training participation;
[0024] The new client adaptation mechanism is specifically as follows: for a newly joined client, parameter fusion is performed using a parameter fusion ratio α, where α=1 / (1+nt).
[0025] Preferably, the updating of cluster assignment and cluster center global parameters includes:
[0026] Calculate the distance between each client and each cluster center and update the cluster assignment;
[0027] Fixed client cluster assignments and updated cluster center global parameters;
[0028] The updated global parameters of the cluster center are broadcast to the clients of the corresponding cluster center to update the local model parameters.
[0029] Preferably, the process of forming the fusion parameters includes:
[0030] Dynamically adjust the fusion weight α according to the client participation interval nt, and set the cluster center global parameter and client local model parameters W i To perform fusion, the fusion formula is:
[0031]
[0032] Among them, W pred is the fusion parameter.
[0033] Preferably, performing prediction evaluation on a local test set using fusion parameters includes:
[0034] The dynamic weight fusion strategy is used to perform the prediction evaluation of the model, and the dynamic weight W i The calculation formula is:
[0035] W i =(1-β·nt)·lw+β·nt·gw
[0036] Among them, β is the weight coefficient; nt is the round interval, lw is the local model weight, and gw is the global model weight.
[0037] According to another aspect of the present invention, a clustering federated learning system based on a prediction mechanism is provided, the method comprising: the system comprising a server-side clustering module and a client-side selection module deployed on a server, and a parameter fusion prediction module and an update round tracking module deployed on a client;
[0038] Server-side clustering module: responsible for client-side model parameter clustering allocation and cluster center global parameter update;
[0039] Client selection module: Based on the partial client selection strategy, some clients are randomly selected to participate in training in each round;
[0040] Parameter fusion prediction module: fuses the global parameters of the cluster center and the local parameters of the client, and performs prediction and evaluation on the local test set based on the fused parameters;
[0041] Update the round tracking module: record the client's participation in training, calculate the interval rounds of the client's participation in training, and dynamically adjust the fusion ratio based on the interval rounds.
[0042] According to a third aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method when executing the program.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] 1) This invention solves the problem of traditional multi-center clustering federated learning relying on full client participation through partial client training and dynamic parameter fusion mechanism; it enhances the adaptability to new clients by using dynamic parameter fusion; thus, it provides adaptability of heterogeneous computing nodes in non-IID environments, ensures model accuracy, and provides an effective solution for collaborative learning in distributed environments.
[0045] 2) The present invention adopts a dynamic weight fusion strategy to dynamically fuse the global parameters of the cluster center with the local parameters of the client during the prediction stage, thereby enhancing the model's adaptability to new clients and improving the generalization ability of the local model.
[0046] 3) The present invention improves the model performance and convergence speed by adaptively adjusting the fusion ratio of global and local parameters through updating round tracking optimization. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 Flowchart of the clustering federated learning method based on the prediction mechanism in the present invention;
[0048] Figure 2 Schematic diagram of the overall framework of federated learning based on multi-center clustering in the present invention;
[0049] Figure 3 Flowchart of the clustering federated learning method based on the prediction mechanism in the present invention;
[0050] Figure 4 Schematic diagram of the parameter fusion mechanism in the present invention;
[0051] Figure 5 This is a curve diagram showing the relationship between client participation and performance in the present invention. DETAILED DESCRIPTION
[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.
[0053] To address the problems of slow model convergence and performance degradation in existing federated learning frameworks in non-IID environments due to data heterogeneity, partial client participation, and dynamic topology changes, this paper proposes a clustering federated learning method (ClusterPredict algorithm) based on a prediction mechanism.
[0054] Example 1
[0055] This embodiment relates to a clustering federated learning method based on a prediction mechanism. The method is based on multi-center clustering and local-global parameter fusion, and is suitable for model training and prediction in a non-independent and identically distributed (non-IID) data environment.
[0056] like Figure 1 , the method comprises the following steps:
[0057] S1. Configure the client and server architecture and initialize the local data distribution of each client;
[0058] S2. Clients are grouped through a multi-center clustering mechanism, and the FedPredict mechanism is used to combine global and local parameters to enhance the prediction effect;
[0059] S3. Use the randomized expectation maximization algorithm to perform steps E and M on the client, update the cluster assignment and optimize the cluster center parameters by calculating the distance between the device and the cluster center;
[0060] S4. Based on the clustering results, training is performed using a partial client selection strategy, wherein the partial client selection strategy includes cluster ratio sampling, round interval evaluation, and new client adaptation mechanism;
[0061] S5. Output the optimized multi-center clustering model and calculate the weighted average accuracy and F1 score.
[0062] like Figure 3 As shown, the core of this method includes:
[0063] 1) Multi-center clustering mechanism: Cluster and group client model parameters to provide differentiated global models for clients with different data distribution characteristics.
[0064] 2) Partial client training strategy: Only some clients are selected to participate in each round of training, reducing communication overhead and improving system scalability;
[0065] 3) Global-local parameter fusion mechanism: Dynamically fuses the global parameters of the cluster center with the local parameters of the client during the prediction phase to enhance the model's adaptability to new clients;
[0066] 4) Update round tracking optimization: By recording the difference in rounds of training participated by the client, the fusion ratio of global and local parameters is adaptively adjusted.
[0067] A detailed introduction to each core point is as follows.
[0068] Step 1: Initialization and model clustering, including:
[0069] 1.1, Global initialization: The server randomly initializes K cluster center parameters The initial model is distributed to all clients. The clients are initially clustered using the K-MEANS algorithm to determine K cluster centers.
[0070] 1.2, Local training: Each client uses local data to train the model and obtain the local model parameters W i ;
[0071] 1.3, Parameter clustering: The server collects the client model parameters, uses the improved expectation maximization algorithm for clustering, and calculates the distance d between each client and the cluster center ik .
[0072] Step 2: Partial client training
[0073] 2.1, Client selection: Randomly select some clients from each of the K clusters to form a set sc k , specifically including:
[0074] S211, Cluster Proportional Sampling: Randomly select a certain proportion of clients from each cluster to form the training set sc k , the calculation formula is: Where C is the selected proportional coefficient, ranging from 0 to 1. represents the number of clients in the kth cluster;
[0075] S212, round interval evaluation: calculate the interval rounds of the client's participation in training nt = t-lt, where t is the current round and lt is the round of the most recent training participation;
[0076] S213. New client adaptation mechanism: For a newly joined client, a parameter fusion ratio α=1 / (1+nt) is used, where nt is the number of rounds after the client joins, that is, the interval rounds.
[0077] 2.2, Local Update: The selected client loads the most recently saved local model parameters and uses the global parameters received from the corresponding cluster center Conduct local training;
[0078] 2.3 Update Record: Calculate and record the client's training round interval nt = t - lt, which reflects the number of rounds since the client last updated the model parameters. Where t is the client's current round, and lt is the client's most recent parameter training round.
[0079] Step 3: Cluster center update
[0080] 3.1, E step: Calculate the distance between each client and each cluster center and update the cluster assignment
[0081] 3.2, M step: fix client cluster assignment and update cluster center global parameters
[0082] 3.3, Broadcast parameters: Broadcast the updated global parameters of the cluster center to the client of the corresponding cluster center to update the local model parameters.
[0083] Step 4: Global and local parameter fusion prediction
[0084] 4.1, parameter fusion: such as Figure 4 , dynamically adjust the fusion weight α according to the client participation interval nt, and set the cluster center global parameter and client local model parameters W i To perform fusion, the fusion formula is: Where W pred is the fusion parameter.
[0085] Specifically include:
[0086] 411) Calculate the client's update level ul and the global evolution level el according to the interval round;
[0087] 412) Calculate the fusion weight of the local model weight lw and the global model weight gw based on the update level and the evolution level.
[0088] The larger the interval round nt, the larger the fusion weight α, and the more biased towards global knowledge.
[0089] 4.2, Model Performance Evaluation: Using Fusion Parameter W predPerform prediction evaluation on the local test set and calculate the accuracy and F1 score. Specifically, calculate the weighted accuracy for each client, where the weight is proportional to the amount of client data; evaluate the F1 score of the clustering model and calculate the weighted average F1 score of each cluster center. The F1 score is weighted averaged by the amount of data in the cluster center.
[0090] The dynamic weight fusion strategy is used to optimize the prediction effect. The calculation formula of the fusion weight is:
[0091] W i =(1-β·nt)·lw+β·nt·gw
[0092] Where β is the weight coefficient, set to 0.1 here; nt is the round interval, lw is the local model weight, and gw is the global model weight. A weight coefficient of 0.1 ensures a smooth convergence of the global model and avoids drastic fluctuations. In scenarios with high data heterogeneity, a low β value can preserve more local features and improve personalization performance.
[0093] In the early stage, the interval between training rounds is small, which encourages the local model to fully learn local features and avoid being biased by the global model too early; in the later stage, the interval between training rounds is large, which promotes the local model to align with the global optimal solution and improve the generalization ability.
[0094] The local model may overfit due to data bias. The weighted accuracy uses the data volume weight to filter out local models with stronger generalization capabilities, thereby improving the reliability of the local model weight.
[0095] 4.3, Adaptive adjustment: Adjust the next round of clustering and fusion parameters based on the evaluation results of the model performance.
[0096] The implementation process of the multi-center clustering mechanism includes:
[0097] S21. Initially cluster the clients using the K-MEANS algorithm to determine K cluster centers.
[0098] S22. Determine the optimal cluster assignment by calculating the cosine similarity between the client model parameters and each cluster center; the cosine similarity calculation method is:
[0099] S221, client model parameter W i and the cluster center parameter W k Perform vectorized processing;
[0100] S222, calculate the cosine similarity d between vectors ik =cos(W i ,W k );
[0101] S223: Assign the client to the cluster center with the highest similarity
[0102] S23. The clustering results and the client local model parameters are passed to the server for global parameter aggregation.
[0103] The cluster federated learning method based on the prediction mechanism of the present invention (i.e., the ClusterPredict algorithm) and the traditional FedAvg algorithm are applied to the FEMNIST benchmark dataset. The weighted accuracy and F1 score of the ClusterPredict algorithm are improved by 4.16% and 6.7% respectively compared with the FedAvg algorithm. Figure 5 Therefore, the method of the present invention significantly improves the adaptability to non-IID environments and heterogeneous computing nodes while maintaining the accuracy of the model, and provides an effective solution for collaborative learning in a distributed environment.
[0104] Example 2
[0105] This embodiment also relates to a clustering federated learning system based on a prediction mechanism, such as Figure 1 ,The system includes a server-side clustering module and a client-side selection module deployed on the server side, and a parameter fusion prediction module and ,update round tracking module deployed on the client side.
[0106] Server-side clustering module: responsible for client-side model parameter clustering and center update;
[0107] Client selection module: randomly selects some clients to participate in training in each round;
[0108] Parameter fusion prediction module: The global parameters of cluster centers are and client local parameter W i Perform fusion and make predictions and evaluations on the local test set based on the fusion parameters;
[0109] Updated round tracking module: records client participation in training and dynamically adjusts the fusion ratio.
[0110] In addition, the system selects some clients to reduce communication costs, while ensuring model accuracy through dynamic parameter fusion and update round tracking optimization. It is suitable for distributed learning scenarios such as the Internet of Things and mobile devices with limited resources and heterogeneous clients.
[0111] Example 3
[0112] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.
[0113] Many components in a device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.
[0114] The processing unit performs the various methods and processes described above. For example, in some embodiments, the method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the CPU can be configured to execute the method in any other appropriate manner (e.g., by means of firmware).
[0115] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0116] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0117] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0118] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.
Claims
1. A clustering federated learning method based on prediction mechanism, characterized by: The method includes: Configure the client and server architecture and initialize the local data distribution of each client; Cluster the clients using a multi-center clustering mechanism and determine K cluster centers. Use the randomized expectation maximization algorithm to update the cluster assignment and cluster center global parameters; Based on the clustering results, the model is trained through some client selection strategies in each of the K cluster centers to update the local model parameters. Dynamically fuse the cluster center global parameters with the local model parameters to form fusion parameters; Use the fusion parameters to perform prediction evaluation on the local test set and output the optimized multi-center clustering model.
2. A clustering federated learning method based on a prediction mechanism according to claim 1, characterized in that: The multi-center clustering mechanism includes: Perform initial clustering of clients using the K-MEANS algorithm to determine K cluster centers; By calculating the cosine similarity between the client's local model parameters and each cluster center, the optimal cluster assignment is determined and the clustering results are obtained; The clustering results and the client's local model parameters are passed to the server for global parameter aggregation.
3. The clustering federated learning method based on prediction mechanism according to claim 1, characterized in that: The described partial client selection strategies include cluster proportion sampling, round interval evaluation and new client adaptation mechanism.
4. A clustering federated learning method based on a prediction mechanism according to claim 3, characterized in that: The cluster ratio sampling is specifically: randomly select a certain proportion of clients from each cluster center to form a training set sc k , the calculation formula is: Among them, C is the selected proportional coefficient, represents the number of clients in the kth cluster.
5. The clustering federated learning method based on prediction mechanism according to claim 3, characterized in that: The round interval evaluation is specifically as follows: calculating the interval rounds of the client's participation in training nt = t-lt, where t is the current round and lt is the round of the most recent participation in training; The new client adaptation mechanism is specifically as follows: for a newly joined client, parameter fusion is performed using a parameter fusion ratio α, where α=1 / (1+nt).
6. The cluster federated learning method based on prediction mechanism according to claim 1, characterized in that: The updating of cluster assignment and cluster center global parameters includes: Calculate the distance between each client and each cluster center and update the cluster assignment; Fixed client cluster assignments and updated cluster center global parameters; The updated global parameters of the cluster center are broadcast to the clients of the corresponding cluster center to update the local model parameters.
7. The clustering federated learning method based on prediction mechanism according to claim 1, characterized in that: The process of forming the fusion parameters includes: Dynamically adjust the fusion weight α according to the client participation interval nt, and set the cluster center global parameter and client local model parameters W i To perform fusion, the fusion formula is: Among them, W pred is the fusion parameter.
8. The cluster federated learning method based on prediction mechanism according to claim 1, characterized in that: The prediction evaluation using the fusion parameters on the local test set includes: The dynamic weight fusion strategy is used to perform the prediction evaluation of the model, and the dynamic weight W i The calculation formula is: W i =(1-β·nt)·lw+β·nt·gw Among them, β is the weight coefficient; nt is the round interval, lw is the local model weight, and gw is the global model weight.
9. A system using the clustering federated learning method based on the prediction mechanism according to any one of claims 1 to 8, characterized in that: The method comprises: the system comprises a server-side clustering module and a client-side selection module deployed on the server side, and a parameter fusion prediction module and an update round tracking module deployed on the client side; Server-side clustering module: responsible for client-side model parameter clustering allocation and cluster center global parameter update; Client selection module: Based on the partial client selection strategy, some clients are randomly selected to participate in training in each round; Parameter fusion prediction module: fuses the global parameters of the cluster center and the local parameters of the client, and performs prediction and evaluation on the local test set based on the fused parameters; Update the round tracking module: record the client's participation in training, calculate the interval rounds of the client's participation in training, and dynamically adjust the fusion ratio based on the interval rounds.
10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Federated learning methods, devices and equipment
CN113033820B