A federated learning method based on feature selection and feature alignment

Abstract

The application discloses a federated learning method based on feature selection and feature alignment. The steps of the application are as follows: 1, data set acquisition; 2, the central server issues the current global feature extraction module to all clients. The client generates an intermediate layer feature and uploads it to the central server. The central server trains the client discriminator and the feature attention vector by receiving the intermediate layer feature; 3, the central server selects a client subset to participate in the current round of federated learning and issues a global model; 4, the selected client uses the obtained feature attention vector to construct a feature mask vector for feature screening and performs feature alignment; 5, the global model is updated by using the obtained client discriminator, combining the alignment loss and the prediction loss, and then uploaded to the central server; 6, the central server aggregates the received local model to generate a new global model. The application can improve the convergence speed and model prediction accuracy of federated learning training.

Description

Technical Field

[0001] This invention relates to the field of federated learning, and in particular to a method for handling non-independent and identically distributed feature spaces in federated learning. Background Technology

[0002] Google proposed the FedAvg federated learning algorithm in 2019, enabling effective joint training of deep models without sharing data among the parties involved. It has been widely applied in real-world scenarios such as behavior recognition, malware classification, and product recommendation. While FedAvg has found widespread use in many fields, the non-independent and identically distributed nature of data in real-world scenarios often leads to convergence difficulties and poor model prediction accuracy when simply applying FedAvg. Mitigating the impact of non-independent and identically distributed data on federated learning remains a challenging task within the field.

[0003] Recently, several methods have been developed to mitigate the problem of non-independent and identically distributed (i.i.d.) data in federated learning by imposing constraints. These methods primarily involve adding constraint terms to the loss function to constrain the direction of model updates. Generally, the previous global model is used as an anchor point, ensuring that updated local models revolve around it, thus mitigating client drift caused by non-independent and identically distributed data. Additionally, some methods estimate the model's update direction and correct it to ensure consistency.

[0004] Most existing federated learning algorithms constrain the model and focus on the non-independent and identically distributed (i.I.D.) label space problem. While imposing constraints can alleviate client drift to some extent, it prevents the model from updating quickly and effectively, limiting its update potential and slowing down its convergence speed. This invention differs from existing federated learning algorithms that primarily focus on the non-independent and identically distributed (i.I.D.) label space problem; instead, it focuses on the non-independent and identically distributed (i.I.D.) feature space problem. It fundamentally alleviates client drift at the data level through a gradient-inverted feature adversarial alignment method. Since features of different dimensions are not necessarily non-independent and identically distributed, this invention uses a feature attention vector to generate a mask vector to filter features and perform targeted alignment of non-independent and identically distributed feature dimensions. Simultaneously, to mitigate the impact of feature alignment on the original task, this invention uses gradient truncation to prevent the original task features from receiving gradients from the feature alignment loss. Summary of the Invention

[0005] This invention provides a federated learning method based on feature selection and feature alignment. This method employs selective adversarial feature alignment to collaboratively train the client discriminator and feature selection vectors. This effectively filters out non-independent and identically distributed feature dimensions and aligns the feature distributions of each party through personalized affine transformations, mitigating the adverse effects of non-independent and identically distributed data on federated learning training from a data-level perspective. Experimental results show that this method can improve the convergence speed and prediction accuracy of federated learning training.

[0006] A federated learning method based on feature selection and feature alignment, the steps of which are as follows:

[0007] Step (1) Dataset acquisition: We obtain local datasets for different clients by applying different Gaussian noises to the public dataset.

[0008] Step (2): The central server distributes the current global feature extraction module to all clients. Each client uses its local dataset and the received global feature extraction module to generate intermediate layer features and uploads them to the central server. The central server uses the received intermediate layer features from each client to train the client discriminator and feature attention vector.

[0009] Step (3): The central server selects a subset of clients to participate in this round of federated learning according to a certain strategy, and distributes the global model, including the client discriminator and feature attention vector.

[0010] Step (4): The selected client uses the obtained feature attention vector to construct a feature mask vector for feature filtering, and uses the feature affine transformation module for feature alignment.

[0011] Step (5): The selected client uses the obtained client discriminator to update the global model by combining the alignment loss and the prediction loss, and finally uploads the obtained local model to the central server.

[0012] Step (6): The central server aggregates the received local models according to a certain strategy to generate a new global model.

[0013] Furthermore, step (1), which refers to applying different Gaussian noise to the local datasets of different clients, means adding noise with a mean of 0 and a variance of... Gaussian noise is added to dataset I to generate the local dataset I for client i. i The specific formula is as follows:

[0014]

[0015] Among them, variance The number is determined by the client's ID i. We use N to represent the total number of clients, β to represent the maximum variance, and variance... The specific formula for calculation is as follows:

[0016]

[0017] For each client i, we generate a noisy image of the same size for each original image in dataset I, where each pixel of the noisy image is derived from the corresponding Gaussian distribution. A random value obtained by sampling from the original image. We add the noisy image and the original image together to obtain the final local dataset image.

[0018] The intermediate layer feature acquisition and client discriminator and feature selection vector training process described in step (2) are as follows:

[0019] To better capture the differences in local data distribution across different client machines, we use a unified global feature extractor. The original features are encoded to obtain the intermediate layer features r, as shown in the following formula:

[0020]

[0021] r = r1∪r2…∪r N (Formula 3)

[0022] Where, r i I represents the set of all intermediate layer features of client i. i Indicates original features.

[0023] The client discriminator and feature selection vector training data are intermediate layer features r. i The client ID *i* is used as the label. We use α to represent the feature attention vector, which is multiplied by each feature dimension and then fed into the client discriminator. The client discriminator relies on the differences in the data distribution of each client to distinguish the intermediate layer features from each client; that is, dimensions with a higher degree of non-independent and identically distributed nature are more important to the client discriminator and will be given higher attention weights. We perform unified optimization of the client discriminator and the feature attention vector on the central server, with the following specific formula:

[0024]

[0025]

[0026] Among them, D * α * Let r represent the updated client discriminator and feature attention vector, respectively. i,j r represents the feature set of the i-th client. i The j-th feature, |I i| represents the number of original features of the i-th client, and N represents the total number of clients.

[0027] The Sigmoid function and scaling factor ζ are used to accelerate the convergence of the feature attention vector and achieve binarization. Specifically, the feature attention vector is generated by the following formula:

[0028] α = Digmoid(ζp)

[0029]

[0030] Where p is a trainable parameter used to generate the feature attention vector, and x represents the function input value.

[0031] Step (3) refers to client selection, which involves randomly sampling from all N clients to obtain a subset S containing n clients. r , and participate in the rth round of federated learning training.

[0032] The feature selection and feature transformation process in step (4) is mainly implemented by the mask vector M and the feature affine transformation module G.

[0033] First, the mask vector M is generated using the feature attention vector α obtained in step (2) according to the threshold θ, as shown in the following formula:

[0034]

[0035] Where, α k M represents the value of the k-th bit of the feature attention vector α. k This represents the value of the k-th position in the mask vector M. The feature at the position where the mask vector M is 1 represents the selected feature. This mask vector M is used to filter the original intermediate layer features to obtain a subset of feature dimensions to be aligned.

[0036] Characteristic Affine Transformation T i Includes trainable parameter scaling factor s i and the bias coefficient b i Specifically, the affine transformation formula is as follows:

[0037] T i (x)=s i ⊙x+b i (Formula 7)

[0038] Where x represents the function input value.

[0039] It is important to note that this affine transformation only applies to the selected non-independent and identically distributed feature dimensions, while we do not modify the independent and identically distributed feature dimensions.

[0040] In summary, the formulas for the affine transformation module are as follows:

[0041] G i (r i ,M)=C(Τ i (Θ(r i ,M)),Θ(r i ,1-M)) (Formula 8)

[0042] Where Θ represents the feature filtering operation, and C represents the concatenation operation, which concatenates the aligned non-independent and identically distributed feature dimension subsets and the independent and identically distributed feature dimension subsets into a feature of the same size as the original.

[0043] The local model training in step (5) refers to training the local model w. i ={φ i G i ,P i The feature extraction module φ is updated accordingly. i Feature Affine Transformation Module G i Prediction network module P i .

[0044] After each client receives the global model sent by the central server, it first replaces its local model with the global model. This invention preserves the individuality of the feature affine transformation module G, enabling it to perform personalized transformations for different client data distributions; that is, the initial local model w to be updated in this round... i for:

[0045]

[0046] Among them, the feature extraction module φ i It only accepts gradients from the task loss function to ensure feature separability and reduce the impact of feature alignment on the original task. Feature extraction module φ i Prediction network module P i The specific formula for the update is as follows:

[0047]

[0048]

[0049] Where m refers to dataset I i The number of categories included. For the j-th feature r of client i i,j The feature obtained after transformation is given by the following formula:

[0050]

[0051] For training the feature affine transformation module G, we used a module R called the gradient inversion layer. This module maintains the identity mapping during forward propagation and inverts the gradient during backpropagation. Its derivative formula during backpropagation is as follows:

[0052]

[0053] Where λ is the inversion coefficient and x represents the function input value.

[0054] Meanwhile, the feature affine transformation module G needs to take into account the prediction task loss, and its update formula is as follows:

[0055]

[0056]

[0057] The model aggregation operation referred to in step (6) means averaging the received local models to serve as the global model for the next round of federated learning. Since we use a personalized affine transformation module G, aggregation is unnecessary; only the feature extraction module φ and the prediction network module P need to be aggregated. The specific formula is as follows:

[0058]

[0059] Among them, S r Let be the set of clients obtained from the r-th round of sampling, and n be the number of clients in this set.

[0060] The beneficial effects of this invention are as follows:

[0061] This invention aligns the non-independent and identically distributed (i.i.d.) feature dimensions of the parties involved through adversarial feature alignment, mitigating the adverse effects of data non-independent and identically distributed (i.i.d.) on federated learning at the data level. Furthermore, through adaptive feature selection, the model actively learns the degree of non-independent and identically distributed (i.i.d.) of each feature dimension, selecting only those non-independent and identically distributed feature dimensions for alignment, thus reducing computational load and accelerating training. Compared to constraining model updates, this invention reduces the forced constraints on the direction and amount of model updates, accelerating model convergence. Compared to correcting model updates, it reduces the additional computational and storage overhead associated with model correction. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the specific process of the present invention.

[0063] Figure 2 This is a schematic diagram of the network framework of the present invention.

[0064] Figure 3 This is a schematic diagram of the feature transformation of the present invention. Detailed Implementation

[0065] The present invention will be further described in detail below with reference to the accompanying drawings:

[0066] like Figure 1 As shown, this invention provides a federated learning algorithm based on feature selection and feature alignment, which can be used to alleviate the problem of non-independent and identically distributed features. Specific process details are as follows... Figure 2 As attached Figure 2 As shown, this diagram illustrates the application scenarios, model structure, and training process of the present invention. Sub-diagram (a) illustrates the application scenario of the present invention in cross-border e-commerce, where different national sites communicate with the central server to collaboratively train the global model without transmitting the original data. Sub-diagram (b) illustrates the training process of the traditional FedAvg algorithm, where each client updates the distributed global model using its local dataset and then uploads it to the central server, which aggregates the data and distributes a new round of global models. Sub-diagram (c) illustrates the overall training framework of the present invention, which is divided into a distributed training process and a centralized training process. The following will be based on the attached diagram. Figure 1 The sub-diagram (c) in the diagram is used to illustrate the details of each step. The specific implementation steps are as follows:

[0067] During distributed training, each client uses its local model w i ={φ i G i ,P i The feature extraction module φ is updated accordingly. i Feature Affine Transformation Module G i Prediction network module P i Its training data comes from a local dataset and consists of a feature mask module M and a feature affine transformation module G. i Features are aligned to mitigate the problem of non-independent and identically distributed data in federated learning. The specific formulas are summarized below:

[0068]

[0069]

[0070]

[0071] During centralized training, the client discriminator D and the feature attention vector α are updated, enabling the client discriminator to learn the differences between the local data distributions of different clients, thus guiding the feature affine transformation module G during distributed training. i The update enables the feature attention vector α to learn the degree of non-independent and identically distributed nature of different feature dimensions, thereby guiding feature selection during distributed training.

[0072] As attached Figure 3 The diagram illustrates the feature transformation process of this invention. First, a mask vector M is obtained through the feature attention vector, and this M is used to divide the original features into independent and identically distributed subsets and non-independent and identically distributed subsets. The independent and identically distributed subsets remain unchanged through an identity mapping, while the non-independent and identically distributed subsets are transformed by the feature affine transformation module G. i The transformation is performed to generate an aligned feature distribution, and then the independent and identically distributed subsets and the non-independent and identically distributed subsets are re-concatenated into a feature by the concatenation function C.

[0073] As shown in Table 1, the industrial dataset CE-CCP is selected. This dataset comes from real traffic logs of a cross-border e-commerce platform. The dataset contains click tags and purchase tags; in this invention, only click tags are used for CTR prediction. Different countries are considered as different customers. This dataset includes four main countries: the United States, Brazil, Spain, and France. The number of samples and the ratio of positive to negative samples are similar across different countries. However, due to the different purchasing behaviors of users in different countries, the feature distributions are extremely dissimilar, exhibiting non-independent and identically distributed characteristics. The statistical information of the dataset is shown in Table 1.

[0074] Table 1 Statistical Information of CE-CCP Dataset

[0075]

[0076] The comparative model used in the experiment is an existing federated learning algorithm for solving data that is not independent and identically distributed. We use AUC, a commonly used metric in recommender systems, to evaluate the model's predictive performance. AUC is defined as the area under the ROC curve. Its meaning is: when randomly selecting a pair of positive and negative samples, the higher the probability that the score of the positive sample is greater than the score of the negative sample, the higher the AUC, indicating better model classification performance for positive and negative samples. As shown in Table 2, our proposed method, based on a federated learning algorithm using feature selection and adversarial feature alignment, has achieved significant results in the cross-border e-commerce federated system, with the average prediction accuracy AUC improving by 0.14% compared to local training.

[0077] Table 2 CE-CCP Experimental Results

[0078]

[0079] We conducted experiments on the public dataset CIFAR10, constructing the dataset using the method shown in step (1). We used ACC as the evaluation metric for model performance, which represents the proportion of correctly predicted samples out of the total samples. We also plotted the prediction performance changes during training and calculated the number of epochs required for convergence.

[0080] Table 3(a) CIFAR-10 Experimental Results Table 3(b) CIFAR-10 Experimental Results

[0081]

[0082] As shown in Table 3, our proposed method also achieves significant results on public datasets. As shown in Table 3(a), the average prediction accuracy (ACC) of our invention is improved by 20.68% compared to local training. Meanwhile, as shown in Table 3(b), our invention significantly accelerates the convergence speed, reducing the number of rounds required to achieve the same prediction accuracy by a factor of 2 compared to the traditional federated learning method FedAvg.

Claims

1. A federated learning method based on feature selection and feature alignment, characterized in that... It includes the following steps: Step (1) Dataset acquisition: By applying different Gaussian noises to the public dataset, local datasets for different clients are obtained. Step (2): The central server distributes the current global feature extraction module to all clients; each client uses its local dataset and the received global feature extraction module to generate intermediate layer features and uploads them to the central server; the central server uses the received intermediate layer features from each party to train the client discriminator and feature attention vector. Step (3): The central server selects a subset of clients to participate in this round of federated learning according to the specified strategy and distributes the global model, including the client discriminator and feature attention vector; Step (4): The selected client uses the obtained feature attention vector to construct a feature mask vector for feature filtering, and uses the feature affine transformation module for feature alignment. Step (5): The selected client uses the obtained client discriminator to update the global model by combining the alignment loss and the prediction loss, and finally uploads the obtained local model to the central server. Step (6): The central server aggregates the received local models according to a certain strategy to generate a new global model; Step (2) is implemented as follows: To better capture the differences in local data distribution across different client machines, a unified global feature extractor is used. The original features are encoded to obtain the intermediate layer feature set r for each client, as shown in the following formula: r = r1∪r2…∪r N (Formula 1) Where, r i I represents the set of all intermediate layer features of client i. i Indicates original features; The client discriminator D and the feature attention vector α are uniformly optimized on the central server, and their training data comes from the feature set r of the i-th corresponding client. i The specific formula is as follows: Among them, D * α * Let r represent the updated client discriminator and feature attention vector, respectively. i,j r represents the feature set of the i-th client. i The j-th feature, |I i | represents the number of original features of the i-th client, and N represents the total number of clients; Simultaneously, the Sigmoid function and scaling factor ζ are used to accelerate the convergence of the feature attention vector and achieve binarization; specifically, the feature attention vector is generated by the following formula: α = Sigmoid(ζp) Where p is a trainable parameter used to generate the feature attention vector, and x represents the function input value; Step (4) is implemented as follows: 4-1. Using the optimized feature attention vector α obtained in step (2) * The mask vector M is generated according to the threshold θ, as shown in the following formula: Where, α k M represents the value of the k-th bit of the feature attention vector α. k This represents the value of the k-th bit in the mask vector M; 4-2. Based on the mask vector M, select the corresponding subset of non-independent and identically distributed feature dimensions, and apply an affine transformation T to it. i The formula for the feature affine transformation module is as follows: G i (r i ,M)=C(T i (Θ(r i ,M)),Θ(r i ,1-M)) T i (x)=s i ⊙x+b i (Formula 5) Where Θ represents the feature filtering operation, and C represents the concatenation operation, which concatenates the aligned subsets of non-independent and identically distributed feature dimensions with the subsets of independent and identically distributed feature dimensions to form a feature set of the same size as the original. i b i Let be the trainable parameters of the affine transformation corresponding to client i, and let x represent the function input value.

2. The federated learning method based on feature selection and feature alignment according to claim 1, characterized in that... Step (5) is implemented as follows: After each client receives the global model sent by the central server, it first replaces its local model with the global model; the individuality of the feature affine transformation module G in the local model is preserved so that it can perform personalized transformations for different client data distributions, i.e., the initial local model w to be updated. i for: The local model includes a feature extraction module φ i Feature Affine Transformation Module G i Prediction network module P i ; A gradient inversion layer R is inserted between the affine transformation module G and the client discriminator to train the affine transformation module. The gradient inversion layer R introduces an inversion operation during backpropagation, as shown in the following formula: Where λ is the inversion coefficient, representing the degree of inversion, and x represents the function input value; Preventing gradient truncation from the loss function The gradient is backpropagated to the feature extraction module φ i ; For the feature extraction module φ i Feature Affine Transformation Module G i Prediction network module P i The overall optimization goals are as follows: Among them, P i * ,φ i * P represents the optimized feature extraction module and prediction network module, respectively. i , This represents the optimized feature affine transformation module, where m represents the number of data categories. For the j-th feature r of client i i,j The features obtained after transformation.

3. The federated learning method based on feature selection and feature alignment according to claim 2, characterized in that... The model aggregation operation referred to in step (6) means averaging the received local models to serve as the global model for the next round of federated learning. Since a personalized affine transformation module G is used, it is not necessary to aggregate it; only the feature extraction module φ and the prediction network module P need to be aggregated. The specific formula is as follows: Among them, S r Let be the set of clients obtained from the r-th round of sampling, and n be the number of clients in this set.

Images