Method for enhancing generalization ability of distributed system

Abstract

The invention discloses a method for enhancing the generalization ability of a distributed system, and the method comprises the following steps: S1, constructing a global model, and transmitting the global model to all distributed source domains through a server; s2, training a local model by each source domain to calculate empirical loss, and generating a respective local model; s3, calculating an evaluation score of each source domain on the generalization ability of the global model; s4, uploading the evaluation score of each source domain and calculating the weight of each source domain; and S5, aggregating a new global model, and publishing the new global model to each distribution source domain. According to the method, through wide experiments on a reference data set including a PACS data set, an OfficeHome data set and an Office-31 data set, it is proved that the proposed method FedIF is remarkably superior to an existing weight strategy and other methods, and the generalization performance of a global model trained through distributed learning is further improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed technology, and in particular to a method for enhancing the generalization capability of a distributed system. Background Art

[0002] With the popularity of distributed learning, federated domain generalization (FedDG) has attracted more and more attention. The goal of federated domain generalization is to train a model that performs well not only on the source domain involved in the training, but also on the target domain that is not involved in the training. Currently, an effective way to improve the domain generalization performance of distributed systems is to aggregate local models according to appropriate weights. However, existing methods for calculating weights do not fully utilize the impact of data on global model updates. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a method for enhancing the generalization ability of a distributed system, comprising the following steps: S1. constructing a global model, and sending the global model to all distributed source domains through a server; S2. each source domain trains a local model to calculate the empirical loss, and generates its own local model; S3. calculating the evaluation score of each source domain for the generalization ability of the global model; S4. uploading the evaluation scores of each source domain and calculating the weights of each source domain; S5. aggregating the new global model and publishing it to each distributed source domain.

[0004] Furthermore, the calculation formula for calculating the empirical loss R(θ) in step S2 is: Where N k represents the total number of distribution source domains, L(z i ,θ) represents data z i The loss function for the parameters θ.

[0005] Furthermore, the calculation formula for calculating the evaluation score of each source domain on the generalization ability of the global model in the S3 step is: In the formula, represents the first-order derivative with respect to parameter θ, represents the inverse matrix of the Hessian matrix, z c represents the core set randomly obtained from the source domain training set, z i represents the training data in each source domain, L is the loss function, represents the global model sent to each source domain in round r-1. Represents the final parameters of the local model trained in the kth source domain in the rth round.

[0006] Furthermore, in step S4, the weight ω of each source domain is calculated k The calculation formula is: Among them, Score krepresents the evaluation score of the k-th source domain, Represents the total evaluation score of the source domain.

[0007] Furthermore, in step S5, the new global model is aggregated The specific calculation formula is: In the formula, represents the final parameters of the local model trained in the kth source domain in the rth round, ω k represents the weight of the k-th source domain.

[0008] Furthermore, it also includes a round determination step, specifically: S6. Determine whether the number of training times reaches the end round, if not, repeat step S2.

[0009] The present invention provides a method for enhancing the generalization capability of a distributed system, which has the following beneficial effects:

[0010] Through extensive experiments on benchmark datasets including PACS, OfficeHome and Office-31 datasets, the proposed method FedIF is demonstrated to be significantly superior to existing weight strategies and other methods, further improving the generalization performance of the global model trained by distributed learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0012] Figure 1 The present invention provides a flow chart of the method. DETAILED DESCRIPTION

[0013] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0014] The following is a detailed description of the implementation method of the present invention in conjunction with the accompanying drawings. Only some embodiments are described, not all embodiments. For the purpose of clarity, representations and descriptions that are not related to the present invention are omitted in the drawings and descriptions.

[0015] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below. Obviously, the implementation cases described are part of the embodiments of the present invention, not all of the embodiments, and cannot be understood as limiting the scope of the implementation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0016] like Figure 1 As shown, the present invention provides a method for enhancing the generalization ability of a distributed system, comprising the following steps: S1. constructing a global model, and sending the global model to all distributed source domains through a server; S2. each source domain trains a local model to calculate the empirical loss, and generates its own local model; S3. calculating the evaluation score of each source domain for the generalization ability of the global model; S4. uploading the evaluation scores of each source domain and calculating the weights of each source domain; S5. aggregating the new global model and publishing it to each distributed source domain.

[0017] The calculation formula for calculating the empirical loss R(θ) in step S2 is: Where N k represents the total number of distribution source domains, L(z i ,θ) represents data z i The loss function for the parameters θ.

[0018] IC (Influence Compute) is used to calculate the evaluation score of each source domain on the generalization ability of the global model. The higher the score, the more influential the source domain is on the generalization effect of the global model in this round of aggregation. The calculation formula for calculating the evaluation score of each source domain on the generalization ability of the global model in step S3 is: In the formula, represents the first-order derivative with respect to parameter θ, represents the inverse matrix of the Hessian matrix, z c represents the core set randomly obtained from the source domain training set, z i represents the training data in each source domain, L is the loss function, represents the global model sent to each source domain in round r-1. Represents the final parameters of the local model trained in the kth source domain in the rth round.

[0019] When aggregating the global model, the weight of the kth child node is 0<ω k <1, by default, the Score value of each child node is normalized.

[0020] Upload the score of each distributed node. The server center node scales and limits the score through the IA (Influence Adjust) module to obtain the weight of each distributed node. Among them, the weight of each source domain is calculated k The calculation formula is: Among them, Score k represents the evaluation score of the k-th source domain, represents the total evaluation score of the source domain, and That is, the sum of the weights is 1. By default, the score can be directly used for normalization to distribute weights. This is the default setting of IA. In non-default situations, prior knowledge is added manually to increase disturbances or weights.

[0021] Aggregate the new global model in step S5 The specific calculation formula is: In the formula, represents the final parameters of the local model trained in the kth source domain in the rth round, ω k represents the weight of the k-th source domain.

[0022] It also includes a round determination step, specifically: S6. Determine whether the number of training times reaches the end round, if not, repeat step S2.

[0023] Through extensive experiments on benchmark datasets including PACS, OfficeHome and Office-31 datasets, the proposed method FedIF is demonstrated to be significantly superior to existing weight strategies and other methods, further improving the generalization performance of the global model trained by distributed learning.

[0024] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for enhancing the generalization capability of a distributed system, characterized in that: The following steps are involved: S1. Build a global model and send it to all distributed source domains through the server; S2. Each source domain trains a local model to calculate the empirical loss and generate its own local model; S3. Calculate the evaluation score of each source domain on the generalization ability of the global model; S4. Upload the evaluation scores of each source domain and calculate the weight of each source domain; S5. Aggregate the new global model and publish it to various distribution source domains.

2. The method for enhancing the generalization capability of a distributed system according to claim 1, characterized in that: The calculation formula for calculating the empirical loss R(θ) in step S2 is: Where N k represents the total number of distribution source domains, L(z i ,θ) represents data z i The loss function for the parameters θ.

3. The method for enhancing the generalization capability of a distributed system according to claim 1, characterized in that: The calculation formula for calculating the evaluation score of each source domain on the generalization ability of the global model in the S3 step is: In the formula, ▽ θ represents the first-order derivative with respect to parameter θ, represents the inverse matrix of the Hessian matrix, z c represents the core set randomly obtained from the source domain training set, z i represents the training data in each source domain, L is the loss function, represents the global model sent to each source domain in round r-1. Represents the final parameters of the local model trained in the kth source domain in the rth round.

4. The method for enhancing the generalization capability of a distributed system according to claim 1, characterized in that: In the step S4, the weight ω of each source domain is calculated k The calculation formula is: Among them, Score k represents the evaluation score of the k-th source domain, Represents the total evaluation score of the source domain.

5. The method for enhancing the generalization capability of a distributed system according to claim 1, characterized in that: The new global model is aggregated in step S5 The specific calculation formula is: In the formula, represents the final parameters of the local model trained in the kth source domain in the rth round, ω k represents the weight of the k-th source domain.

6. The method for enhancing the generalization capability of a distributed system according to claim 1, characterized in that: It also includes a round determination step, specifically: S6. Determine whether the number of training times reaches the end round, if not, repeat step S2.

Images