Federal width learning optimization method for data isomerism
By performing dimensionality reduction and clustering on the client parameter update matrix in federated width learning, model performance and training efficiency problems caused by data heterogeneity are solved, and faster model convergence is achieved.
Patent Information
- Application Number
- CN202411465522.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-01-17
AI Technical Summary
In federated width learning, due to inconsistent parameter update matrix dimensions and data heterogeneity between clients, the global performance and training efficiency of the model are affected.
The principal component analysis method is used to reduce the dimensionality of the parameter update matrices of all clients so that they have the same dimensions, and the cosine similarity between the parameter matrices of each client is calculated, and the AP algorithm is used for clustering to divide the clients into several group, thus transforming it into an approximately independent and identically distributed problem.
It ensures that the parameter matrices of different clients can be compared in a unified space, improves the accuracy and effectiveness of clustering, and significantly accelerates the convergence process of the model.
Smart Images
Figure CN119323272A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of federated wide learning, and in particular, to a federated wide learning optimization method targeting data heterogeneity. Background Art
[0002] Traditional machine learning algorithms require users to upload source data to high-computing cloud servers for centralized training, which leads to uncontrollable data flow and sensitive data leakage. As a result, Google proposed federated learning technology in 2016 to deal with this privacy issue. The general purpose of federated learning is to achieve decentralized machine learning in a privacy-preserving way. Specifically, traditional federation is an architecture with a central server that coordinates multiple clients to train shared models using their local data and achieve global learning goals. During the training process, the data is saved locally on the client, and only the model parameters are shared. At present, federated learning technology has been applied to healthcare, finance, smartphones and other fields, providing better protection for personal privacy.
[0003] As a new type of neural network, the wide learning system has attracted people's attention since it was first proposed. It has shown good performance in image processing, face recognition and other fields. Wide learning has several significant advantages over deep learning. First, it avoids complex network structure and a large number of hyperparameter adjustments, thereby greatly reducing training time. Second, the wide learning method is easier to implement and deploy, while supporting incremental learning, which allows more flexible model updates when faced with new data. Training a wide learning model requires comprehensive data, but the data for training the model is usually held by different companies, which are reluctant to share data. To solve this problem, federated wide learning came into being. Compared with existing wide learning systems, it can integrate multi-party data to obtain comprehensive information and realize the training of wide learning systems in the case of isolated data islands. However, federated wide learning still has the problem of data heterogeneity.
[0004] To solve the data heterogeneity problem, we consider grouping clients with similar data distribution together and transforming a Non-IID (non-independent and identically distributed) problem into multiple approximate IID (independent and identically distributed) problems.
[0005] Specifically, we use principal component analysis (PCA) to reduce the dimension of the client parameter update matrix and convert the parameter matrices of all clients to the same dimension. After the dimension reduction is completed, the cosine similarity between the parameter matrices of each client is calculated to evaluate the similarity between them. Then, the AP (Affinity Propagation) algorithm is used to cluster the clients, thereby converting a Non-IID problem into multiple approximate IID problems.
[0006] In 2021, Junqing Le et al. proposed a federated continuous learning framework based on width learning (FCL-BL). They proposed a weighted processing strategy to solve the problem of catastrophic forgetting, so that the FCL-BL framework can handle continuous learning. In addition, they also introduced a batch asynchronous method and width learning (BL) technology to ensure the high efficiency of FCL-BL. Specifically, the asynchronous method they proposed reduces the number of interaction rounds between the client and the server, and the width learning technology supports incremental learning without generating new data for retraining during learning.
[0007] The Chinese invention application with application number 202211270637.3 discloses "A secure vertical federated wide learning method". Its technical solution introduces matrix encryption technology into the vertical federated wide learning system and proposes a secure vertical federated wide learning method, which aims to build a better wide learning system with collaborative computing and storage capacity for feature data holders with limited data privacy protection. However, this technical solution does not solve the data heterogeneity problem of federated wide learning. Summary of the invention
[0008] In order to solve the technical problem that the existing federated wide learning has inconsistent parameter update matrix dimensions and data heterogeneity between clients, which affects the global performance and training efficiency of the model, the present invention provides a federated wide learning optimization method for data heterogeneity. The technical solution adopted by the present invention is:
[0009] A first aspect of the present invention provides a federated width learning optimization method for data heterogeneity, the method comprising:
[0010] Get the parameter update matrix for each client;
[0011] Perform dimensionality reduction on the parameter update matrix of each client;
[0012] After completing the dimensionality reduction process, the cosine similarity between the parameter matrices of each client is calculated;
[0013] The number of clusters is determined according to the cosine similarity between the parameter matrices of each client, and the clients are clustered to obtain several groups of client sets;
[0014] Train a corresponding in-group local width model for each client set to obtain a corresponding in-group local width model parameter matrix;
[0015] The global width model parameters are updated through the local width model parameter matrix within each group.
[0016] As a preferred solution, the method for obtaining the parameter update matrix of each client includes:
[0017] After each client receives the initialized global width model, it updates the model through local training to obtain the local model and defines the client C i The parameter update matrix O of (i∈l) i :
[0018]
[0019] Among them, the matrix O i ∈Y S*F , matrix O i Each row represents a parameter vector and each column represents a parameter.
[0020] As a preferred solution, the method for reducing the dimension of the parameter update matrix of each client includes:
[0021] The parameter update matrix of each client is reduced in dimension through principal component analysis.
[0022] As a preferred solution, the method of reducing the dimension of the parameter update matrix of each client through principal component analysis includes:
[0023] For the matrix O l Each parameter in (l=0,1,2,...,L,O0=ξ), that is, each column Standardize it to have a mean of 0 and a variance of 1:
[0024]
[0025] Among them, μ q and σ q O l The mean and standard deviation of the qth column, Q l is the normalized parameter update matrix, is the qth column of the matrix;
[0026] Calculate Q l The covariance matrix of the covariance matrix is used to determine the number of principal components k, Q to be retained according to the number of eigenvalues corresponding to the covariance matrix. l The covariance matrix of is:
[0027]
[0028] Performing eigenvalue decomposition on the covariance matrix can obtain the eigenvalues and corresponding eigenvectors;
[0029] With the minimum number of principal components k min The matrix of is taken as the benchmark, all matrices will sort the corresponding eigenvalues in order from large to small, and select the first k minThe eigenvectors corresponding to the eigenvalues construct the projection matrix P = [v1,v2,...,v kmin ], Q l Project to a lower dimensional space to achieve dimensionality reduction:
[0030] Q′ l =Q l P.
[0031] As a preferred solution, after the dimensionality reduction process is completed, the method for calculating the cosine similarity between the parameter matrices of each client includes:
[0032] Once all matrices are of the same dimension, calculate each parameter update matrix Q l The cosine similarity s between (l=1,2,...,L) and the parameter matrix Q0 of the global width model l :
[0033]
[0034] As a preferred solution, the method of determining the number of clusters according to the cosine similarity between the parameter matrices of each client and clustering the clients to obtain several groups of client sets includes:
[0035] The number of clusters is determined by a preset clustering algorithm according to the cosine similarity between the parameter matrices of each client, and clients with similar data distribution are clustered in the same set, thereby obtaining several client sets.
[0036] As a preferred solution, the steps of the preset clustering algorithm include:
[0037] A1: Construct a similarity matrix based on cosine similarity values Where S(a,b) represents S a and S b The Euclidean distance between (a∈N,b∈N);
[0038] A2: Initialize the preset responsibility matrix and the preset availability matrix to zero matrices respectively; In the example, R(a,b) represents the a to b Information, indicating s b As s a The suitability of the cluster center; in the availability matrix In the example, V(a,b) represents the b to a Information, indicating s b As s a After the cluster center of s b degree of suitability;
[0039] A3: Each preset data object updates its availability matrix and responsibility matrix according to the received message, and sends its availability matrix and responsibility matrix to other data objects;
[0040] A4: Repeat step A3 until the values of the availability matrix and responsibility matrix no longer change, that is, convergence is achieved;
[0041] A5: When convergence is achieved, the data object with the highest availability is selected as the cluster center, and then other data objects are assigned to the cluster center according to the responsibility matrix;
[0042] A6: All data points assigned to the same cluster center are output as a single client set, and finally several groups of client sets are obtained.
[0043] As a preferred solution, a method for training a corresponding in-group local width model for each client set and obtaining a corresponding in-group local width model parameter matrix includes:
[0044] Get a labeled multi-class dataset {(X,Y)|X∈R N×D ,Y∈R N×C}, where N represents the sample set, D represents the total number of feature nodes, and C represents the number of classes;
[0045] The training data set X is converted into feature nodes Z through n sets of feature maps n =[Z1,Z2,...,Z n ], in is the activation function, W zi ∈R D×e is a randomly generated input weight matrix, β zi ∈R N×e represents the bias term, e represents the number of feature nodes in the feature node group;
[0046] Taking feature nodes as input, they are transformed into enhanced nodes H through nonlinear activation functions m =[H1,H2,...,H m ], H j =ζ j (Z n W hj +β hj ),j=1,2,...,m, where W hj ∈R ne×u is a randomly generated input weight matrix, β hj ∈R N ×u is the bias term, ζ() is the nonlinear activation function, and u represents the number of enhanced nodes in the enhanced node group;
[0047] Finally, the enhanced node is used to train the corresponding in-group local width model for each client set to obtain the corresponding in-group local width model parameter matrix.
[0048] A second aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned federated width learning optimization method for data heterogeneity.
[0049] The third aspect of the present invention provides a computer device, including a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of a federated width learning optimization method for data heterogeneity as claimed in the preceding claim.
[0050] Compared with the prior art, the present invention has the following beneficial effects:
[0051] The present invention reduces the dimension of the parameter update matrices of all clients by principal component analysis, so that all matrices have the same dimension. This step ensures that the parameter matrices of different clients can be compared in a unified space, reduces the problems caused by inconsistent parameter dimensions, and improves the accuracy and effectiveness of subsequent clustering.
[0052] After dimensionality reduction, the present invention can dynamically determine the optimal number of clusters by calculating the cosine similarity between clients and using the AP algorithm for clustering. This method ensures that clients can be effectively grouped according to the similarity of their parameter matrices, thereby converting complex Non-IID data distribution problems into multiple approximate IID problems, significantly accelerating the convergence process of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 A flow chart of a federated width learning optimization method for data heterogeneity provided in this embodiment;
[0054] Figure 2 A structural diagram of a federated wide learning model for data heterogeneity provided in this embodiment. DETAILED DESCRIPTION
[0055] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;
[0056] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.
[0057] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0058] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.
[0059] In addition, in the description of this application, unless otherwise specified, "multiple" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0060] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0061] Example 1
[0062] Please refer to Figure 1 as well as Figure 2 , this embodiment provides a federated width learning optimization method for data heterogeneity, the method comprising:
[0063] S1: Get the parameter update matrix of each client;
[0064] In a specific embodiment, the method for obtaining the parameter update matrix of each client includes:
[0065] After each client receives the initialized global width model, it updates the model through local training to obtain the local model and defines the client C i The parameter update matrix O of (i∈l) i :
[0066]
[0067] Among them, the matrix O i ∈Y S*F , matrix O i Each row represents a parameter vector and each column represents a parameter.
[0068] S2: Perform dimensionality reduction on the parameter update matrix of each client;
[0069] In a specific embodiment, the method for performing dimensionality reduction processing on the parameter update matrix of each client includes:
[0070] The parameter update matrix of each client is reduced in dimension through principal component analysis.
[0071] In a specific embodiment, the method of performing dimensionality reduction processing on the parameter update matrix of each client through principal component analysis includes:
[0072] For the matrix O l Each parameter in (l=0,1,2,...,L,O0=ξ), that is, each column Standardize it to have a mean of 0 and a variance of 1:
[0073]
[0074] Among them, μ q and σ q O l The mean and standard deviation of the qth column, Q l is the normalized parameter update matrix, is the qth column of the matrix;
[0075] Calculate Q l The covariance matrix of the covariance matrix is used to determine the number of principal components k, Q to be retained according to the number of eigenvalues corresponding to the covariance matrix. l The covariance matrix of is:
[0076]
[0077] Performing eigenvalue decomposition on the covariance matrix can obtain the eigenvalues and corresponding eigenvectors;
[0078] With the minimum number of principal components k min The matrix of is taken as the benchmark, all matrices will sort the corresponding eigenvalues in order from large to small, and select the first k min The eigenvectors corresponding to the eigenvalues construct the projection matrix Q l Project to a lower dimensional space to achieve dimensionality reduction:
[0079] Q′ l =Q l P.
[0080] S3: After the dimensionality reduction process is completed, the cosine similarity between the parameter matrices of each client is calculated;
[0081] In a specific embodiment, after the dimensionality reduction process is completed, the method for calculating the cosine similarity between the parameter matrices of each client includes:
[0082] Once all matrices are of the same dimension, calculate each parameter update matrix Q l The cosine similarity s between (l=1,2,...,L) and the parameter matrix Q0 of the global width model l :
[0083]
[0084] S4: Determine the number of clusters according to the cosine similarity between the parameter matrices of each client, and cluster the clients to obtain several groups of client sets;
[0085] In a specific embodiment, the method of determining the number of clusters according to the cosine similarity between the parameter matrices of each client, and clustering the clients to obtain a plurality of client sets includes:
[0086] The number of clusters is determined by a preset clustering algorithm according to the cosine similarity between the parameter matrices of each client, and clients with similar data distribution are clustered in the same set, thereby obtaining several client sets.
[0087] In a specific embodiment, the steps of the preset clustering algorithm include:
[0088] A1: Construct a similarity matrix based on cosine similarity values Where S(a,b) represents S a and S b The Euclidean distance between (a∈N,b∈N);
[0089] A2: Initialize the preset responsibility matrix and the preset availability matrix to zero matrices respectively; In the example, R(a,b) represents the a to b Information, indicating s b As s a The suitability of the cluster center; in the availability matrix In the example, V(a,b) represents the b to a Information, indicating s b As s aAfter the cluster center of s b degree of suitability;
[0090] A3: Each preset data object updates its availability matrix and responsibility matrix according to the received message, and sends its availability matrix and responsibility matrix to other data objects;
[0091] A4: Repeat step A3 until the values of the availability matrix and responsibility matrix no longer change, that is, convergence is achieved;
[0092] A5: When convergence is achieved, the data object with the highest availability is selected as the cluster center, and then other data objects are assigned to the cluster center according to the responsibility matrix;
[0093] A6: All data points assigned to the same cluster center are output as a single client set, and finally several groups of client sets are obtained.
[0094] S5: training a corresponding in-group local width model for each client set to obtain a corresponding in-group local width model parameter matrix;
[0095] In a specific embodiment, a method for training a corresponding in-group local width model for each client set to obtain a corresponding in-group local width model parameter matrix includes:
[0096] Get a labeled multi-class dataset {(X,Y)|X∈R N×D ,Y∈R N×C}, where N represents the sample set, D represents the total number of feature nodes, and C represents the number of classes;
[0097] The training data set X is converted into feature nodes Z through n sets of feature maps n =[Z1,Z2,...,Z n ], in is the activation function, W zi ∈R D×e is a randomly generated input weight matrix, β zi ∈R N×e represents the bias term, e represents the number of feature nodes in the feature node group;
[0098] Taking feature nodes as input, they are transformed into enhanced nodes H through nonlinear activation functions m =[H1,H2,...,H m ], H j =ζ j (Z n W hj +β hj ),j=1,2,...,m, where W hj ∈Rne×u is a randomly generated input weight matrix, β hj ∈R N ×u is the bias term, ζ() is the nonlinear activation function, and u represents the number of enhanced nodes in the enhanced node group;
[0099] Finally, the enhanced node is used to train the corresponding in-group local width model for each client set to obtain the corresponding in-group local width model parameter matrix.
[0100] S6: updating the global width model parameters through the local width model parameter matrix within each group.
[0101] Example 2
[0102] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of a federated width learning optimization method for data heterogeneity described in Example 1 are implemented.
[0103] Example 3
[0104] A computer device includes a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor. When the computer program is executed by the processor, the steps of a federated width learning optimization method for data heterogeneity described in Example 1 are implemented.
[0105] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A federated wide learning optimization method for data heterogeneity, characterized in that: The method comprises: Get the parameter update matrix for each client; Perform dimensionality reduction on the parameter update matrix of each client; After completing the dimensionality reduction process, the cosine similarity between the parameter matrices of each client is calculated; The number of clusters is determined according to the cosine similarity between the parameter matrices of each client, and the clients are clustered to obtain several groups of client sets; Train a corresponding in-group local width model for each client set to obtain a corresponding in-group local width model parameter matrix; The global width model parameters are updated through the local width model parameter matrix within each group.
2. The method for optimizing federated wide learning for data heterogeneity according to claim 1, characterized in that: Methods for obtaining the parameter update matrix for each client include: After each client receives the initialized global width model, it updates the model through local training to obtain the local model and defines the client C i The parameter update matrix O of (i∈l) i : Among them, the matrix O i ∈Y S*F , matrix O i Each row represents a parameter vector and each column represents a parameter.
3. The method for optimizing federated wide learning for data heterogeneity according to claim 2, characterized in that: Methods for reducing the dimension of the parameter update matrix of each client include: The parameter update matrix of each client is reduced in dimension through principal component analysis.
4. The method for optimizing federated wide learning for data heterogeneity according to claim 3, characterized in that: Methods for reducing the dimension of the parameter update matrix of each client through principal component analysis include: For the matrix O l Each parameter in (l=0,1,2,...,L,O0=ξ), that is, each column O' l q (q∈N) is standardized to have a mean of 0 and a variance of 1: Among them, μ q and σ q O l The mean and standard deviation of the qth column, Q l is the normalized parameter update matrix, is the qth column of the matrix; Calculate Q l The covariance matrix of the covariance matrix is used to determine the number of principal components k, Q to be retained according to the number of eigenvalues corresponding to the covariance matrix. l The covariance matrix of is: Performing eigenvalue decomposition on the covariance matrix can obtain the eigenvalues and corresponding eigenvectors; With the minimum number of principal components k min The matrix of is taken as the benchmark, all matrices will sort the corresponding eigenvalues in order from large to small, and select the first k min The eigenvectors corresponding to the eigenvalues construct the projection matrix Q l Project to a lower dimensional space to achieve dimensionality reduction: Q′ l =Q l P。 5. The method for optimizing federated wide learning for data heterogeneity according to claim 4, characterized in that: After the dimensionality reduction process is completed, the method for calculating the cosine similarity between the parameter matrices of each client includes: Once all matrices are of the same dimension, calculate each parameter update matrix Q l The cosine similarity s between (l=1,2,...,L) and the parameter matrix Q0 of the global width model l :
6. The method for optimizing federated wide learning for data heterogeneity according to claim 5, characterized in that: The method of determining the number of clusters according to the cosine similarity between the parameter matrices of each client and clustering the clients to obtain several groups of client sets includes: The number of clusters is determined by a preset clustering algorithm according to the cosine similarity between the parameter matrices of each client, and clients with similar data distribution are clustered in the same set, thereby obtaining several client sets.
7. The method for optimizing federated wide learning for data heterogeneity according to claim 6, characterized in that: The steps of the preset clustering algorithm include: A1: Construct a similarity matrix based on cosine similarity values Where S(a,b) represents S a and S b The Euclidean distance between (a∈N,b∈N); A2: Initialize the preset responsibility matrix and the preset availability matrix to zero matrices respectively; In the example, R(a,b) represents the a to b Information, indicating s b As s a The suitability of the cluster center; in the availability matrix In the example, V(a,b) represents the b to a Information, indicating s b As s a After the cluster center of s b degree of suitability; A3: Each preset data object updates its availability matrix and responsibility matrix according to the received message, and sends its availability matrix and responsibility matrix to other data objects; A4: Repeat step A3 until the values of the availability matrix and responsibility matrix no longer change, that is, convergence is achieved; A5: When convergence is achieved, the data object with the highest availability is selected as the cluster center, and then other data objects are assigned to the cluster center according to the responsibility matrix; A6: All data points assigned to the same cluster center are output as a single client set, and finally several groups of client sets are obtained.
8. The method for optimizing federated wide learning for data heterogeneity according to claim 7, characterized in that: The method of training a corresponding in-group local width model for each client set and obtaining a corresponding in-group local width model parameter matrix includes: Get a labeled multi-class dataset {(X,Y)|X∈R N×D ,Y∈R N×C }, where N represents the sample set, D represents the total number of feature nodes, and C represents the number of classes; The training data set X is converted into feature nodes Z through n sets of feature maps n =[Z1,Z2,...,Z n ], in is the activation function, W zi ∈R D×e is a randomly generated input weight matrix, β zi ∈R N×e represents the bias term, e represents the number of feature nodes in the feature node group; Taking feature nodes as input, they are transformed into enhanced nodes H through nonlinear activation functions m =[H1,H2,...,H m ], H j =ζ j (Z n W hj +β hj ),j=1,2,...,m, where W hj ∈R ne×u is a randomly generated input weight matrix, β hj ∈R N×u is the bias term, ζ() is the nonlinear activation function, and u represents the number of enhanced nodes in the enhanced node group; Finally, the enhanced node is used to train the corresponding in-group local width model for each client set to obtain the corresponding in-group local width model parameter matrix.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a federated width learning optimization method for data heterogeneity as described in any one of claims 1 to 8 are implemented.
10. A computer device, characterized in that: It includes a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, and when the computer program is executed by the processor, it implements the steps of a federated width learning optimization method for data heterogeneity as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Safe longitudinal federal width learning method
CN116050533A