Method and system for realizing federated domain generalization by using data enhancement
Through data augmentation technology and flat perception optimization, combined with global model constraints, the generalization ability problem caused by domain offset in federated learning is solved, and a flatter minimum value is found in federated learning is achieved, which significantly improves the generalization of the model ability.
Patent Information
- Application Number
- CN202411291268.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2025-01-07
AI Technical Summary
In federated learning, the training model has a domain offset when deployed on a client that is not involved in training, resulting in limited generalization capabilities. The existing methods are difficult to effectively solve the flat minimum value of the global model, resulting in a suboptimal solution.
Using data augmentation technology, through adversarial data augmentation strategy and flat-aware optimization methods, combined with global model constraints, multiple rounds of server-client communication are carried out, and augmented network parameters and local model parameters are optimized, and the minimum value of global flatness is sought.
It realizes finding flatter minimums in federated learning, improves the generalization ability of the model, and has proven that it performs better than previous work on multiple benchmarks, especially in the FedDG scenarios, which significantly improves performance.
Smart Images
Figure BDA0005044667490000024 
Figure BDA0005044667490000033 
Figure BDA0005044667490000041
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of federated learning, and specifically relates to a method and system for realizing federated domain generalization by utilizing data enhancement. Background Art
[0002] In recent years, Federated Learning (FL) has become a popular distributed learning paradigm with the advantage of data privacy protection. In federated learning, distributed clients keep data locally and do not share data between clients. Clients collaborate to train a global model, and the server does not participate in specific data transmission. In each communication round, each client trains a local model on local data and uploads it to the server. The server then aggregates the information to obtain a global model and distributes it to each client. In this way, the global model performs well on clients that participated in the training. However, in practical scenarios, the federated training model may be deployed to clients that did not participate in the training and have domain shifts. This poses a challenge to the generalization ability of the trained model, which is called the Federated Domain Generalization Problem (FedDG).
[0003] The problem of federated domain generalization has attracted extensive attention. Most promising methods try to align the behaviors of local models from different perspectives, such as: FedADG aligns feature distributions, FedIIR aligns gradients to learn domain-invariant representations, and StableFDG enables style sharing between different clients. Different from these methods, we focus on solving the optimization problem of the global model from the perspective of loss surface flatness. There is a wealth of research in the literature exploring the connection between loss surface flatness and model generalization ability. In addition, empirical results show the effectiveness of seeking flat minima in many centralized tasks. Although in federated learning, the feasibility of directly obtaining global data is low due to data privacy issues, many studies circumvent this problem by seeking flat minima on local data, but this inevitably leads to suboptimal solutions. To address this problem, there is an urgent need for a new algorithm that aims to approach the global flat minimum (GFM) of the global model. Summary of the invention
[0004] The purpose of the present invention is to address the defects in the prior art and to propose a method and system for realizing federated domain generalization by using data enhancement.
[0005] The method for realizing federated domain generalization by using data enhancement of the present invention is used in the training phase of the federated model to perform multiple rounds of server-client communication, each round of communication including the following steps:
[0006] S1. The server sends the global model to each client;
[0007] S2. The client performs adversarial enhancement training based on local data, i.e. iteratively performs S2.1 and S2.2;
[0008] S2.1, the client combines the global model to perform adversarial data enhancement constrained by the global model;
[0009] S2.2, the client trains a local model on the enhanced data;
[0010] S3: The client uploads the local model to the server for aggregation.
[0011] S1 specifically includes the following steps:
[0012] The global model is a shallow convolutional neural network, ResNet18 network or ResNet50; if it is the first round of training, initialize the global model parameter θ to θ 0 :For shallow convolutional neural networks, the parameters are initialized using the Kaiming method; for ResNet18 / ResNet50 networks, ImageNet pre-trained parameters can be used for initialization; otherwise, assuming it is the rth round of training, the server sends the global model parameters θ according to the communication network r-1 To each client.
[0013] S2 specifically includes the following steps:
[0014] Assume that there are M clients and this is the rth round of training. For any client i, initialize the local model parameters to the model parameters issued in step S1: If it is the first round of training, the Kaiming method is used to initialize the data augmentation network parameters φ i , otherwise the parameter φ at the end of the previous round is used i As the wheel parameter φ i ;
[0015] Iterate steps S2.1 and S2.2 to optimize the augmented network parameter φ i and local model parameters For every one step S2.1, step S2.2 is performed c times; the parameter φ obtained after step S2.1 is i As the augmented network parameters during local model training in step S2.2, the local model parameters after step S2.2 will be As the local model parameters for calculating the loss in step S2.1.
[0016] S2.1 specifically includes the following steps:
[0017] The following optimization objective is used to optimize the augmented network a(·,φ i ) to optimize:
[0018]
[0019] in, The local dataset reserved for client i, (x, y) is the specific sample and label. is the local model, f(·;θ r-1 ) is the global model, is the cross entropy loss.
[0020] S2.2 specifically includes the following steps:
[0021] By using the augmented network a(·,φ obtained in step S2.1 i ), enhance local dataset Get a(D i ; φ i ) as a substitute for global data; the flat perception optimization method SAM is used to replace the global data a(D i ; φ i ) to optimize the local model First, calculate the direction in which the gradient grows fastest and perturb the network parameters accordingly:
[0022]
[0023] Where γ is the perturbation distance, Then, the gradient is calculated on the perturbed model and the gradient descent optimization is performed on the original parameters:
[0024]
[0025] S3 specifically includes the following steps:
[0026] After completing the communication round r, each client o will send the local model parameters Uploaded to the server through the communication network, aggregated on the server to obtain a new global model θ r :
[0027]
[0028] Among them, p i is the model aggregation coefficient.
[0029] The present invention also relates to a system for implementing federated domain generalization by using data enhancement, comprising a computer module, wherein the module applies the above-mentioned method for implementing federated domain generalization by using data enhancement.
[0030] Beneficial Effects
[0031] The present invention's method is used to find globally flat minima in a federated domain generalization setting. The entire algorithm can be explained by viewing it as minimizing the upper bound of the robust risk of the global model on the global data distribution. Specifically, a global model-constrained adversarial data augmentation strategy is proposed to find alternatives to global data, and acuity-aware minimization is used to pursue flatter minima. Flatness measurement and loss surface visualization experiments verify that the global model found by GFM has flatter minima than FedAvg and methods seeking local flatness. In addition, extensive experiments on four FedDG benchmarks confirm the improved performance of GFM when compared or combined with previous work. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 This is a quantitative result diagram of the flatness of the present invention measured;
[0033] Figure 2 This is a visualization diagram of the loss surface tested on PACS by the present invention.
[0034] Specific implementation
[0035] The present embodiment is described in detail below with reference to the accompanying drawings.
[0036] The method for realizing federated domain generalization by using data enhancement of the present invention is used in the training phase of the federated model, and multiple rounds of server-client communication are performed, and each round of communication includes the following steps:
[0037] S1. The server sends the global model to each client;
[0038] S1 specifically includes the following steps:
[0039] The global model is a shallow convolutional neural network, ResNet18 network or ResNet50; if it is the first round of training, initialize the global model parameter θ to θ 0 :For shallow convolutional neural networks, the parameters are initialized using the Kaiming method; for ResNet18 / ResNet50 networks, ImageNet pre-trained parameters can be used for initialization; otherwise, assuming it is the rth round of training, the server sends the global model parameters θ according to the communication network r-1 To each client.
[0040] S2. The client performs adversarial enhancement training based on local data, i.e. iteratively performs S2.1 and S2.2;
[0041] S2.1. The client performs adversarial data enhancement with global model constraints in combination with the global model. The following optimization objective is used to optimize the augmented network a(·,φ i ) to optimize:
[0042]
[0043] in, The local dataset reserved for client i, (x, y) is the specific sample and label. is the local model, f(·;θ r-1 ) is the global model, is the cross entropy loss.
[0044] S2.2, the client trains a local model on the enhanced data; by using the augmented network a(·,φ obtained in step S2.1 i ), enhance local dataset Get a(D i ; φ i ) as a substitute for global data; the flat perception optimization method SAM is used to replace the global data a(D i ; φ i ) to optimize the local model First, calculate the direction in which the gradient grows fastest and perturb the network parameters accordingly:
[0045]
[0046] Where γ is the perturbation distance, Then, the gradient is calculated on the perturbed model and the gradient descent optimization is performed on the original parameters:
[0047]
[0048] Assume that there are M clients and this is the rth round of training. For any client i, initialize the local model parameters to the model parameters issued in step S1: If it is the first round of training, the Kaiming method is used to initialize the data augmentation network parameters φ i , otherwise the parameter φ at the end of the previous round is used i As the wheel parameter φ i ;
[0049] Iterate steps S2.1 and S2.2 to optimize the augmented network parameter φ i and local model parameters For every one step S2.1, step S2.2 is performed c times; the parameter φ obtained after step S2.1 is i As the augmented network parameters during local model training in step S2.2, the local model parameters after step S2.2 will be As the local model parameters for calculating the loss in step S2.1.
[0050] S3: The client uploads the local model to the server for aggregation.
[0051] S3 specifically includes the following steps:
[0052] After completing communication round r, each client i will set the local model parameters Uploaded to the server through the communication network, aggregated on the server to obtain a new global model θ r :
[0053]
[0054] Among them, p i is the model aggregation coefficient. Example
[0055] In this example, the following FedDG benchmarks are used to evaluate different methods: Digits-DG (24,000 images, 10 categories, four domains), PACS (9,991 images, 7 categories, four domains), OfficeHome (15,588 images, 65 categories, four domains), and TerraInc (24,788 images, 10 categories, four domains). These benchmarks are selected to cover a wide range of conditions in digital and real-world scenarios. All benchmarks are evaluated with one domain left out, that is, the data of one domain is reserved for testing by unseen clients in turn, and the data of other domains is assigned to training clients. The backbone architectures used are CNN for Digits-DG, ImageNet pre-trained ResNet18 for PACS and OfficeHome benchmarks, and ImageNet pre-trained ResNet50 for TerraInc benchmark. For the SAM optimizer, γ is set to 0.02. The following are the individual performance analysis experiments:
[0056] Domain generalization performance
[0057] The key components of GFM are the novel enhancement strategy and SAM optimizer, so it is orthogonal to some previous works. Therefore, the superior performance of GFM is demonstrated in two ways: 1) direct comparison with other baselines; 2) combining GFM with previous works. The considered baselines are briefly introduced as follows: FedAvg. A commonly used baseline in federated learning; FedSAM. The SAM optimizer is adopted in local training, but it does not meet the requirement of global flatness; FedSR. Aims to learn a simple representation of the data to obtain better generalization; StableFDG. Enables each client to explore new styles through style sharing; GA. Proposes to aggregate models in the server according to the generalization gap. FedIIR: Proposes to align the gradients of different clients to obtain an invariant classifier.
[0058] The experimental results of different methods are summarized in Table 1. From the table, it can be concluded that using only GFM (GFM+FedAvg) can achieve SOTA performance on average and on many datasets. In addition, the direct comparison between FedSAM and GFM shows that for FedDG, other factors are required in addition to local flatness, which proves the effectiveness of the proposed global model constrained adversarial data augmentation. It should be noted that FedSAM leads to a significant performance degradation in the Digits-DG dataset. In addition, combining GFM with other methods can continuously improve the generalization ability and achieve better performance, especially GFM(GA). GFM(GA) outperforms the previous SOTA method by 1.7 percentage points on average. The success of GFM(GA) can be attributed to the improvement of flatness in both local training and aggregation stages. Table 1 Average classification accuracy. GFM(X) represents the method combining GFM and X.
[0059] Table 1
[0060]
[0061] Smoothness analysis
[0062] We experimentally compare the flatness of the solutions found by GFM and other methods. Specifically, we use the expected loss value to vary F γ (θ) is used as an indicator. For a model with parameter θ, F γ (θ) Calculate the expected change in loss between θ and θ + γ on a sphere of radius γ as follows:
[0063]
[0064] Larger F γ (θ) indicates that the loss changes significantly when moving from θ to the sphere of radius γ, which reveals a sharp minimum, and vice versa. According to the Monte Carlo method, F can be estimated efficiently with finite samples γ (θ), because F γ (θ) has an unbiased finite sample estimator and is computationally efficient. In the experiment, F γ (θ) is approximated with 50 samples. The F of the global model trained by FedAvg, FedSAM, and GFM on all unseen domains of the PACS dataset is quantitatively measured. γ (θ). FedAvg represents the baseline without special flatness design, FedSAM focuses on the flatness of the local model, and the proposed GFM attempts to approach global flatness. The results are shown in Figure 1It can be concluded that both FedSAM and GFM help improve global flatness, GFM can find flatter minima than FedSAM and FedAvg in all experiments, and the effectiveness of GFM in seeking global flat minima can be verified on both the training set and the test set.
[0065] Attached Figure 1 The flatness of the present invention is determined by F γ (θ) quantitative results of the measurement. Each column represents an independent experiment. (For example, the first column represents an experiment with the photo domain as the unseen test client in the leave-one-out-of-domain evaluation setting.) The training results are calculated on the data of all seen clients, while the test results are calculated on the unseen test domain. For each figure, the Y axis represents the flatness F γ (θ), the X-axis represents the radius γ.
[0066] In addition to the F γ (θ) is measured in random directions, also considering the special case within the aggregation plane. Figure 2 The test loss surfaces of three local models and the aggregated global model derived by different methods on the PACS dataset are plotted in Figure 2. The client model in FedAvg is located in a relatively high loss region, so the resulting global model is far away from a good minimum. Fortunately, seeking a flatter minimum in the local update can improve this situation and tend to find solutions in flat and low-loss regions. Figure 2 The results in show that GFM's solutions on the Art and Cartoon test domains strictly meet this expectation, while the solutions on Photo and Sketch partially meet it, i.e., by finding solutions in low-loss regions with comparable flatness. Moreover, the loss surface of FedSAM can be viewed as a "middle point" between FedAvg and GFM.
[0067] Figure 2 Visualization of the loss surface tested on PACS for the present invention. In each sub-figure, the contours from left to right belong to FedAvg, FedSAM, and GFM, respectively. The triangle mark indicates the local model, and the cross mark indicates the global model. The color bar is log-normalized, and the flatness can be roughly compared by observing the size of the third layer (from high to low) and above.
[0068] Ablation analysis
[0069] The key components of GFM are the global model-constrained adversarial data augmentation strategy (GCA) and the SAM optimizer. Both GCA and SAM are applied in the local training phase and can be used independently. In order to verify the effectiveness of each component of GFM, SAM and GCA are removed from the training respectively, and four combinations are obtained: (1) FedAvg. GFM degenerates to FedAvg without SAM optimizer and GCA; (2) FedSAM. GFM degenerates to the FedSAM baseline without GCA; (3) FedAvg+GCA. FedAvg baseline with GCA; (4) GFM. A complete method containing both components. Experiments were conducted on the PACS dataset, as shown in Table 2.
[0070] Table 2 Ablation analysis
[0071]
[0072] The results show that each component of GFM can help improve the final generalization performance. It is important to note that GCA has a more significant impact on the final performance in Art and Photo domains compared to SAM, while SAM helps improve the performance in Cartoon domain. In addition, both methods perform well in Sketch domain. Combining them can complement their respective strengths in domains with inconsistent appearance and contribute cumulatively in domains with similar performance.
[0073] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
[0074] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.
Claims
1. A method for achieving federated domain generalization using data augmentation, the method is used in the training phase of a federated model, and multiple rounds of server-client communication are performed, each round of communication includes the following steps: S1. The server sends the global model to each client; S2. The client performs adversarial enhancement training based on local data, i.e. iteratively performs S2.1 and S2.2; S2.1, the client combines the global model to perform adversarial data enhancement constrained by the global model; S2.2, the client trains a local model on the enhanced data; S3: The client uploads the local model to the server for aggregation.
2. The method for realizing federated domain generalization by using data enhancement according to claim 1, characterized in that: S1 specifically includes the following steps: The global model is a shallow convolutional neural network, ResNet18 network or ResNet50; if it is the first round of training, initialize the global model parameter θ to θ 0 :For shallow convolutional neural networks, the parameters are initialized using the Kaiming method; for ResNet18 / ResNet50 networks, ImageNet pre-trained parameters can be used for initialization; otherwise, assuming it is the rth round of training, the server sends the global model parameters θ according to the communication network r-1 To each client.
3. The method for realizing federated domain generalization by using data enhancement according to claim 1, characterized in that: S2 specifically includes the following steps: Assume that there are M clients and this is the rth round of training. For any client i, initialize the local model parameters to the model parameters issued in step S1: If it is the first round of training, the Kaiming method is used to initialize the data augmentation network parameters φ i , otherwise the parameter φ at the end of the previous round is used i As the wheel parameter φ i ; Iterate steps S2.1 and S2.2 to optimize the augmented network parameters φi and the local model parameters For every one step S2.1, step S2.2 is performed c times; the parameter φ obtained after step S2.1 is i As the augmented network parameters during local model training in step S2.2, the local model parameters after step S2.2 will be As the local model parameters for calculating the loss in step S2.
1.
4. The method for achieving federated domain generalization using data augmentation according to claim 1, characterized in that: S2.1 specifically includes the following steps: The following optimization objective is used to optimize the augmented network a(·, φ i ) for optimization: in, The local dataset reserved for client i, (x, y) is the specific sample and label, is the local model, f(·;θ r-1 ) is the global model, and l(·,·) is the cross entropy loss.
5. The method for achieving federated domain generalization using data augmentation according to claim 1, characterized in that: S2.2 specifically includes the following steps: By using the augmented network a(·, φ obtained in step S2.1 i ), enhance local dataset Get a(D i ; φ i ) as a replacement for global data; Using the flat-aware optimization method SAM to replace the global data a(D i ; φ i ) to optimize the local model First, calculate the direction in which the gradient grows fastest and perturb the network parameters accordingly: Where γ is the perturbation distance, Then, the gradient is calculated on the perturbed model and the gradient descent optimization is performed on the original parameters:
6. The method for achieving federated domain generalization using data augmentation according to claim 1, characterized in that: S3 specifically includes the following steps: After completing communication round r, each client i will set the local model parameters Uploaded to the server through the communication network, aggregated on the server to obtain a new global model θ r : Among them, p i is the model aggregation coefficient.
7. A system for implementing federated domain generalization using data enhancement, comprising a computer module, which applies the data enhancement method of claims 1 to 6 to implement federated domain generalization.