Federal learning system, model learning device, federal learning method, and model learning program
By setting pre-determined initial values and hyperparameters in the federated learning system, using methods such as small batch extraction and dual variable calculation, the stability problems caused by data variance and performance differences are solved, and the leakage risk and model are balanced through noise processing Accurate, achieving stable and efficient federated learning.
Patent Information
- Application Number
- CN202280095719.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-25
- Publication Date
- 2024-12-17
AI Technical Summary
The existing federated learning system is difficult to conduct stable learning when the data variance maintained by the dispersed is large and the user's computing and communication performance is different, and it also maintains model accuracy while reducing the risk of leakage.
By setting predetermined initial values and hyperparameters in the federated learning system, small batch extraction, model parameter update, dual variable calculation and sending units are used to learn, and the update degree is adjusted through coefficient γ to ensure stable update of model parameters. , while adding noise when necessary to reduce the risk of leakage.
Stable federated learning is achieved under the situation of large data variance and performance differences, and the risk of leakage is reduced and model accuracy is improved by adjusting the noise level balance.
Smart Images
Figure CN119156626A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a federated learning system, a model learning device, a federated learning method, and a model learning program, wherein the federated learning system has multiple model learning devices, and each model learning device is connected to any one of the other model learning devices via a network. Background Art
[0002] By effectively utilizing emails, purchase histories, research materials within companies, IoT information, and hospital diagnostic information within personal terminals, many advantages such as matching, automatic control, and AI diagnosis and treatment can be expected. However, due to concerns about information security and leakage, effective utilization has not yet been achieved. As a countermeasure to prevent the leakage of confidential information such as personal information, there is a federated learning technology (or distributed learning technology) that analyzes data at the end user terminal, making it possible to effectively utilize only proprietary technology (know-how) (for example, dual variables) without issuing confidential information. In order to solve the drift problem in distributed training of machine learning models, non-patent document 1 reconstructs the update process of ECL (asynchronous distributed deep learning technology (Edge-Consensus Learning)) by optimally selecting the constraint-strength control parameter to implicitly increase the gradient correction of SVR (Stochastic Variance Reduction). Figure 1 The algorithm 2 shown in the non-patent document 1 is shown in FIG. The non-patent document 2 reduces the risk of reproducing the original data by adding noise to the learning model (hereinafter referred to as the “model”). Figure 2 Algorithm 1 and equations 13a and 13b shown in non-patent document 2 are shown in FIG.
[0003] Prior art literature
[0004] Non-patent literature
[0005] Non-patent literature 1: Niwa, K., Zhang, G., Kleijn, WB, Harada, N., Sawada, H., Fujino, A.. (2021). Asynchronous Decentralized Optimization With Implicit StochasticVariance Reduction. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research.
[0006] Non-patent document 2: ZonghanH, et al. "Differentially Private ADMM for ConvexDistributed Learning: Improved Accuracy via Multi-Step Approximation", arXiv: 2005.07890. Summary of the invention
[0007] Problems to be solved by the invention
[0008] However, although federated learning can distribute learning and implement it at high speed, there is a risk of reproducing the original data of the model generated by the user. In addition, it is also unknown to what extent information is sent for model learning. Furthermore, the technology of giving differential privacy to federated learning reduces the risk of reproducing the original data by adding noise to the model generated by users in federated learning. The greater the noise applied, the lower the risk of leakage, but the accuracy of the learned model will decrease. In the existing method, it is necessary to arbitrarily determine the noise applied, and there is still the problem of reducing the risk of leakage and improving the accuracy at the same time.
[0009] In addition, when learning is performed on each user's terminal, the variance of the data held by the user is various. For example, sometimes the attributes (labels) of the data held by each user are different, such as user A only has blood pressure information and user B only has heart rate information. In addition, sometimes the number of data held is different, such as user A holds data for 10 people and user B holds data for 3 people. Furthermore, since the terminal performance and communication environment are different for each user's terminal, there are sometimes large differences in the calculation and communication performance of the user terminal. In the technology of non-patent document 1, when the variance of the dispersedly held data is large, and when the calculation and communication performance of the user terminal are very different, there is also a problem that it is difficult to perform stable federated learning.
[0010] There are several problems in federated learning. In the present invention, the following is first set as a subject: even when the variance of the distributed data is large and when the computing and communication performance of users vary greatly, stable federated learning can be performed.
[0011] Means for solving problems
[0012] The federated learning system of the present invention has a plurality of model learning devices, each of which is connected to any one of the other model learning devices via a network. In the federated learning system of the present invention, the parameters, dual variables, step size, model learning data, and constraint parameters of the model are set to predetermined initial values. Each model learning device has a mini-batch extraction unit, a model parameter update unit, a dual variable calculation and transmission unit, a dual variable receiving unit, and a dual variable setting unit. The mini-batch extraction unit extracts a predetermined amount of data from the model learning data as a mini-batch. The model parameter update unit uses the dual variables, step size, mini-batch of model learning data, constraint parameters, and coefficients γ using a predetermined optimal value η and a predetermined hyperparameter α to learn and update the parameters of the model. The dual variable calculation and transmission unit uses the parameters and coefficients γ of the model updated by the model parameter update unit to calculate and send the dual variables for each other model learning device connected to the model learning device. The dual variable receiving unit receives the dual variables from other model learning devices connected to the model learning device. The dual variable setting unit sets the received dual variables as dual variables used in the next learning.
[0013] Effects of the Invention
[0014] According to the federated learning system of the present invention, the model parameter updating unit of the model learning device uses the coefficient γ using the predetermined optimal value η and the predetermined hyperparameter α for updating. In addition, the dual variable calculation and transmission unit uses the parameters of the updated model and the coefficient γ to calculate and transmit the dual variables for each other model learning device connected to the model learning device. Since the degree of update can be adjusted by the coefficient γ, stable federated learning can be performed even when the variance of the distributed data is large and when the computing and communication performance of the users are very different. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is a diagram showing Algorithm 2 shown in Non-Patent Document 1.
[0016] Figure 2 This is a diagram showing Algorithm 1 and Formulas 13a and 13b shown in Non-Patent Document 2.
[0017] Figure 3 This is a diagram showing a configuration example of a federated learning system according to the first embodiment.
[0018] Figure 4 This is a diagram showing an example in which the model learning devices of the present invention are connected in a ring shape.
[0019] Figure 5 This is a diagram showing an example in which the model learning devices of the present invention are randomly connected.
[0020] Figure 6 This is a diagram showing an example of a process flow of the federated learning system according to the first embodiment.
[0021] Figure 7 This is a diagram showing a configuration example of a federated learning system according to the second embodiment.
[0022] Figure 8 This is a diagram showing an example of a processing flow of the federated learning system according to the second embodiment.
[0023] Fig. 9 This is a diagram showing an algorithm of the federated learning system 20 of the second embodiment using the same description as that of Non-Patent Literature 1.
[0024] Fig.10 It is a diagram showing an example of a functional configuration of a computer. DETAILED DESCRIPTION
[0025] Hereinafter, the embodiments of the present invention will be described in detail. In addition, the same reference numerals are given to components having the same functions, and duplicate descriptions are omitted.
[0026] Example 1
[0027] Figure 3 A configuration example of a federated learning system according to the first embodiment is shown. Figure 4 An example in which the model learning device of the present invention is connected in a ring shape is shown. Figure 5 An example is shown in which the model learning devices of the present invention are randomly connected. Figure 6 1 is a diagram showing an example of a process flow of the federated learning system of the first embodiment. The federated learning system 10 of the first embodiment includes N model learning devices 1001, ..., 100 N , each model learning device 100 i Through the network 900, the model learning device 100 j Here, N is an integer greater than 2, i is an integer greater than 1 and less than N, and j is an integer greater than 1 and less than N other than i. Each model learning device 100 i The system includes an initial setting unit 110 , a small batch extraction unit 130 , a model parameter update unit 140 , a dual variable calculation and transmission unit 150 , a dual variable reception unit 160 , and a dual variable setting unit 170 .
[0028] In each model learning device 1001, ..., 100 N In the initial setting unit 110, the model parameter w i , dual variable z i|j , step size μ, model learning data x i , Constraint Parameter A i|j are set to predetermined initial values (S110). For example, w1, ..., w N are set to the same value, are set to z i|j =0.
[0029] Each model learning device 1001, ..., 100 N The small batch extraction unit 130 extracts the model learning data x i Extract a predetermined amount of data as a mini-batch ξ i r,k (S130). For example, when the number of learning data is 10,000, it is sufficient to extract 500 or 1,000 small batches at random. The i at the lower right of ξ represents the number of the model learning device. The k at the upper right of ξ represents the number of each model learning device 1001, ..., 100 NThe integer representing the number of repetitions of the iterative processing (inner loop process) performed in the federated learning system. In the inner loop process, K repetitions are performed. K is an integer greater than 2, and k is an integer greater than 1 and less than K. The r on the upper right of ξ is an integer representing the number of repetitions of the iterative processing (outer loop process) of the inner loop process in the entire federated learning system. In the outer loop process, R repetitions are performed. R is an integer greater than 2, and r is an integer greater than 1 and less than R. In the following description, i and j on the lower right of the symbol represent the number of model learning devices, r on the upper right of the symbol represents the number of repetitions in the outer loop process, and k on the upper right represents the number of repetitions in the inner loop process. In addition, in the text in the specification, since the lower right symbol and the upper right symbol cannot be recorded in the same position horizontally, they are recorded in an offset manner. On the other hand, since they can be recorded in the mathematical formulas and figures in the specification, they are recorded in the same position horizontally.
[0030] Each model learning device 1001, ..., 100 N The model parameter updating unit 140 uses dual variables, step size, mini-batch of model learning data, constraint parameters, and coefficient γ using a predetermined optimal value η and a predetermined hyperparameter α to learn and update the parameters of the model (S140). Specifically, the model parameter updating unit 140 updates the parameters of the model as follows to obtain the parameters w of the model used in the next processing: i r,k+1 .
[0031] [Mathematical formula 1]
[0032]
[0033] Among them, f - is the cost function or a function that can be replaced with the cost function, u is the parameter of the model before updating, ξ i r,k is a small batch of data for model learning, the coefficient γ is 1+αη, N is the number of model learning devices that constitute the federated learning system, and A i|j is the constraint parameter, z i|j r is the dual variable. i r,k+1 is the parameter of the model, which is used in the k+1th processing of the inner loop. In addition, “f - "of" - " is originally placed above "f", but due to the limitation of the text in the specification, it is recorded as "f - ”.
[0034] The predetermined optimal value η is set to 1 / (μKEi ) i The model learning device 100 i The number of other model learning devices connected. The predetermined hyperparameter α is, for example, α=2, but is not limited to this. α suitable for each federated system is different, so it is a parameter that can be appropriately determined for each federated learning system. Cost function or function f that can be replaced by cost function - , model parameters u, constraint parameters A i|j The same functions and parameters as those in Non-Patent Document 1 may be used. For example, the following function q described in "4.1 Problem definition" of Non-Patent Document 1 is used: i (w i ) as a function f - That's it.
[0035] [Mathematical formula 2]
[0036]
[0037] Among them, g i is the cost function f i The differential function of .
[0038] Each model learning device 1001, ..., 100 N The dual variable calculation and transmission unit 150 is connected to the model learning device 100 according to each i Other connected model learning devices 100 j , the model parameter w updated by the model parameter updating unit 140 i r,k+1 and coefficient γ to find the dual variable y i|j , according to each other model learning device 100 j Specifically, the dual variable calculation and transmission unit 150 obtains the dual variable y as follows: i|j In addition, other model learning devices 200 j There is at least one, but there may be more than 2. In addition, the arrows in the formulas indicate substitution.
[0039] [Mathematical formula 3]
[0040]
[0041] Each model learning device 1001, ..., 100 N The dual variable receiving unit 160 receives the dual variable from the model learning device 100 i Other connected model learning devices 100 j Receive dual variable yj|i (S160). Since the dual variable y received by the dual variable receiving unit 160 j|i By the model learning device 100 j The dual variable calculation and transmission unit 150 calculates and transmits, so the positions of “j” and “i” are consistent with the dual variable y calculated and transmitted by the dual variable calculation and transmission unit 150. i|j on the contrary.
[0042] Each model learning device 1001, ..., 100 N The dual variable setting unit 170 receives the dual variable y j|i Set as the dual variable z used in the next learning i|j r,k+1 (S170) The dual variable setting unit 170 converts the received dual variable y into j|i Set as the dual variable z used in the next learning i|j r,k+1 That's it.
[0043] [Formula 4]
[0044]
[0045] Each model learning device 1001, ..., 100 N Confirm whether the processing of the inner loop process has been completed. If it has not been completed ("No"), continue to repeat the processing. If it has been completed ("Yes"), enter the confirmation of the outer loop process (S180). When the answer of step S180 is "Yes", each model learning device 1001, ..., 100 N It is confirmed whether the processing of the outer loop process has been completed. If it has not been completed (in the case of "No"), the processing is continued repeatedly, and if it has been completed (in the case of "Yes"), the processing is terminated (S190).
[0046] According to the federated learning system 10, the model learning devices 1001, ..., 100 N The model parameter updating unit 140 updates the model parameter w using the coefficient γ using the predetermined optimal value η and the predetermined hyperparameter α. i r,k+1 In addition, the dual variable calculation and transmission unit 150 is connected to the model learning device 100 for each i Other connected model learning devices 100 j, the dual variables are calculated and sent using the updated model parameters and coefficient γ. Since the degree of update can be adjusted by the coefficient γ, stable federated learning can be performed even when the variance of the distributed data is large and when the computing and communication performance of users vary greatly.
[0047] Example 2
[0048] In Example 1, the first problem, "performing stable federated learning even when the variance of the distributed data is large and when the differences in the computing and communication performance of the users are large", is solved. By adding restrictions to the means for solving the first problem, the problem of "reducing the risk of leakage and improving the accuracy at the same time" can also be solved. Therefore, in Example 2, "reducing the risk of leakage and improving the accuracy at the same time" is set as the second problem.
[0049] exist Figure 7 An example of the structure of the federated learning system of Example 2 is shown in FIG. Figure 4 An example in which the model learning device of the present invention is connected in a ring is shown in Figure 5 An example in which the model learning devices of the present invention are randomly connected is shown. Figure 8 20 is a diagram showing an example of a process flow of a federated learning system according to Embodiment 2. The federated learning system 20 according to Embodiment 2 includes N model learning devices 2001, ..., 200 N , each model learning device 200 i Through the network and other model learning devices 200 j Each model learning device 200 i The system includes an initial setting unit 110 , a noise generating unit 220 , a small batch extracting unit 130 , a model parameter updating unit 140 , a dual variable calculating and transmitting unit 250 , a dual variable receiving unit 160 , and a dual variable setting unit 170 .
[0050] Each model learning device 2001, ..., 200 N The processing (S130) of the initial setting unit 110 and the small batch extraction unit 130 is the same as that of the first embodiment. Each model learning device 2001, ..., 200 N The noise generating unit 220 generates the noise n i (S220). Specifically, as long as the variance σ is determined in advance 2 And generate Gaussian noise n i That's it.
[0051] Each model learning device 2001, ..., 200 NThe model parameter updating unit 140 uses dual variables, step size, mini-batch of model learning data, constraint parameters, and coefficient γ using a predetermined optimal value η and a predetermined hyperparameter α to learn and update the parameters of the model (S140). Specifically, the model parameter updating unit 140 updates the parameters of the model as shown in the following formula to obtain the parameters w of the model used in the next processing: i r,k+1 .
[0052] [Mathematical formula 5]
[0053]
[0054] Among them, f - is the cost function or a function that can be replaced with the cost function, u is the parameter of the model before updating, ξ i r,k is a small batch of data for model learning, the coefficient γ is 1+αη, N is the number of model learning devices that constitute the federated learning system, and A i|j is the constraint parameter, z i|j r is the dual variable. i r,k+1 is a parameter of the model and is used in the k+1th processing of the inner loop. The processing of the model parameter updating unit 140 is the same as that of the first embodiment. However, since the dual variable z is updated by the dual variable calculation and transmission unit 250 described later, i|j r Noise is added, so γ also plays a role in adjusting the parameters w of the updated model i r ,k+1 The effect of noise when.
[0055] Each model learning device 2001, ..., 200 N The dual variable calculation and transmission unit 250 is connected to the model learning device 200 according to each i Other model learning devices connected 200 j , the model parameter w updated by the model parameter updating unit 140 i r,k+1 and coefficient γ, find the dual variable y with added noise i|j And learn from other models 200 j In addition, other model learning devices 200 j There is at least one, but there may be two or more. Specifically, the dual variable calculation and transmission unit 250 obtains the dual variable y with added noise as follows: i|j .
[0056] [Mathematical formula 6]
[0057]
[0058] Among them, n i is the noise generated by the noise generation unit 220. γ of the dual variable calculation and transmission unit 250 plays a role in adjusting the influence of noise and preventing information leakage.
[0059] The processing (S160) of the dual variable receiving unit 160 and the processing (S170) of the dual variable setting unit 170 are the same as those in Example 1. Also, the inner loop process is confirmed (S180) and the outer loop process is confirmed (S190) as in Example 1, and the iterative processing is performed.
[0060] Fig. 9 This is a diagram showing an algorithm of the federated learning system 20 of the second embodiment in the same manner as in the non-patent document 1. According to the federated learning system 20, since ξ i r,k No additional noise is added, so the influence of noise on learning can be reduced, so the accuracy of model learning can be improved. On the other hand, since the data sent to other model learning devices 200 j The dual variable y i|j Noise is added, so the risk of information leakage can be reduced. Therefore, in addition to solving the first problem of "performing stable federated learning even when the variance of distributed data is large and the difference in user computing and communication performance is large", it is also possible to solve the problem of reducing the risk of leakage and improving accuracy at the same time.
[0061] [Program, recording medium]
[0062] The above-mentioned various treatments can be carried out by Fig.10 The program for executing each step of the above-mentioned method is read into the recording unit 2020 of the computer 2000 shown, and the control unit 2010, the input unit 2030, the output unit 2040, the display unit 2050, etc. are operated to implement the method.
[0063] The program describing the processing contents can be recorded on a computer-readable recording medium, such as a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other recording medium.
[0064] In addition, the program can be circulated by selling, transferring, renting, etc. a removable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program can be stored in a storage device of a server computer and circulated by forwarding the program from the server computer to other computers via a network.
[0065] The computer executing such a program, for example, first temporarily stores the program recorded in a removable recording medium or the program forwarded from a server computer in its own storage device. Then, when executing the process, the computer reads the program stored in its own recording medium and executes the process in accordance with the read program. In addition, as another execution method of the program, the computer can also directly read the program from the removable recording medium and execute the process in accordance with the program, and further, it can also successively execute the process in accordance with the received program each time the program is forwarded from the server computer to the computer. In addition, it can also be set to a structure that performs the above-mentioned process by not forwarding the program from the server computer to the computer, and only realizing the processing function through the execution instruction and result acquisition, so-called ASP (Application Service Provider) type service. In addition, the program in this method is set to include information for the processing of the electronic computer, that is, information based on the program (although not a direct instruction to the computer, but having the nature of specifying the processing of the computer, etc.).
[0066] In addition, in this embodiment, the present device is configured by causing a predetermined program to be executed on a computer, but at least a part of the processing contents may be realized by hardware.
[0067] Description of Reference Numerals
[0068] 10, 20 federated learning system; 100, 200 model learning device; 110 initial setting unit; 130 small batch extraction unit; 140 model parameter updating unit; 150, 250 dual variable calculation and sending unit; 160 dual variable receiving unit; 170 dual variable setting unit; 220 noise generation unit.
Claims
1. A federated learning system having a plurality of model learning devices, each of which is connected to any of the other model learning devices via a network, wherein the federated learning system is characterized in that: The model parameters, dual variables, step size, model learning data, and constraint parameters are set to predetermined initial values. Each model learning device has: A small batch extraction unit extracts a predetermined amount of data from the model learning data as a small batch; A model parameter updating unit, which uses dual variables, step size, mini-batches of model learning data, constraint parameters, and coefficients γ using a predetermined optimal value η and a predetermined hyperparameter α to learn and update the parameters of the model; a dual variable calculation and transmission unit that calculates and transmits a dual variable for each other model learning device connected to the model learning device using the parameters and coefficient γ of the model updated by the model parameter updating unit; a dual variable receiving unit that receives dual variables from other model learning devices connected to the model learning device; and The dual variable setting unit sets the received dual variable as a dual variable used in the next learning.
2. A federated learning system having a plurality of model learning devices, each of which is connected to any of the other model learning devices via a network, wherein the federated learning system is characterized in that: The model parameters, dual variables, step size, model learning data, and constraint parameters are set to predetermined initial values. Each model learning device has: A noise generating unit that generates noise; A small batch extraction unit extracts a predetermined amount of data from the model learning data as a small batch; A model parameter updating unit, which uses dual variables, step size, mini-batches of model learning data, constraint parameters, and coefficients γ using a predetermined optimal value η and a predetermined hyperparameter α to learn and update the parameters of the model; The dual variable calculation and transmission unit calculates and transmits the dual variable to which the noise is added, using the parameters and coefficient γ of the model updated by the model parameter updating unit, for each other model learning device connected to the model learning device. a dual variable receiving unit that receives dual variables from other model learning devices connected to the model learning device; and The dual variable setting unit sets the received dual variable as a dual variable used in the next learning.
3. A model learning device, wherein the model learning device constitutes a federated learning system, the federated learning system having a plurality of the model learning devices, each of which is connected to any one of the other model learning devices via a network, wherein the model learning device is characterized in that: The model parameters, dual variables, step size, model learning data, and constraint parameters are set to predetermined initial values. The model learning device has: A small batch extraction unit extracts a predetermined amount of data from the model learning data as a small batch; A model parameter updating unit, which uses dual variables, step size, mini-batches of model learning data, constraint parameters, and coefficients γ using a predetermined optimal value η and a predetermined hyperparameter α to learn and update the parameters of the model; a dual variable calculation and transmission unit that calculates and transmits a dual variable for each other model learning device connected to the model learning device using the parameters and coefficient γ of the model updated by the model parameter updating unit; a dual variable receiving unit that receives dual variables from other model learning devices connected to the model learning device; and The dual variable setting unit sets the received dual variable as a dual variable used in the next learning.
4. The model learning device according to claim 3, in, The model parameter updating unit updates the model parameter w as follows: i r,k+1 : [Mathematical formula 7] Among them, w i r，k+1 is the parameter of the model, r is the number of learning iterations in the entire federated learning system, k is the number of learning iterations in the model learning device, i and j are symbols representing the model learning device, and f - is the cost function or a function that can be replaced by the cost function, u is the parameter of the model before updating, ξ i r,k is a small batch of data for model learning, the coefficient γ is 1+αη, N is the number of model learning devices that constitute the federated learning system, and A i|j is the constraint parameter, z i|j r is the dual variable, The dual variable calculation and transmission unit obtains the dual variable as follows: [Mathematical formula 8] Among them, y i|j is the dual variable.
5. A model learning device, wherein the model learning device constitutes a federated learning system, the federated learning system having a plurality of the model learning devices, each of which is connected to any one of the other model learning devices via a network, wherein the model learning device is characterized in that: The model parameters, dual variables, step size, model learning data, and constraint parameters are set to predetermined initial values. The model learning device has: A noise generating unit that generates noise; A small batch extraction unit extracts a predetermined amount of data from the model learning data as a small batch; A model parameter updating unit, which uses dual variables, step size, mini-batches of model learning data, constraint parameters, and coefficients γ using a predetermined optimal value η and a predetermined hyperparameter α to learn and update the parameters of the model; A dual variable calculation and transmission unit, for each other model learning device connected to the model learning device, uses the parameters and coefficient γ of the model updated by the model parameter updating unit to calculate and transmit the dual variable to which the noise is added; a dual variable receiving unit that receives dual variables from other model learning devices connected to the model learning device; and The dual variable setting unit sets the received dual variable as a dual variable used in the next learning.
6. The model learning device according to claim 5, It is characterized in that The model parameter updating unit updates the model parameter w as follows: i r,k+1 : [Mathematical formula 9] Among them, w i r,k+1 is the parameter of the model, r is the number of learning iterations in the entire federated learning system, k is the number of learning iterations in the model learning device, i and j are symbols representing the model learning device, and f - is the cost function or a function that can be replaced with the cost function, u is the parameter of the model before updating, ξ i r,k is a small batch of data for model learning, the coefficient γ is 1+αη, N is the number of model learning devices that constitute the federated learning system, and A i|j is the constraint parameter, z i|j r is the dual variable, The dual variable calculation and transmission unit obtains the dual variable to which noise is added as follows: [Formula 10] Among them, y i|j is the dual variable with added noise, n i is the noise generated by the noise generating unit.
7. A federated learning method, which is a federated learning method using a federated learning system, wherein the federated learning system has a plurality of model learning devices, each of which is connected to any one of the other model learning devices via a network, in which: The model parameters, dual variables, step size, model learning data, and constraint parameters are set to predetermined initial values. The federated learning method has an inner loop process, in which each model learning device executes the following steps a first specified number of times: A model parameter update step, using dual variables, step size, mini-batches of model learning data, constraint parameters, coefficients γ using a predetermined optimal value η and a predetermined hyperparameter α to learn and update the model parameters; A dual variable calculation and transmission step, for each other model learning device connected to the model learning device, using the parameters and coefficient γ of the model updated in the model parameter updating step to calculate and transmit the dual variables; a dual variable receiving step of receiving dual variables from other model learning devices connected to the model learning device; and The dual variable setting step sets the received dual variable as the dual variable used in the next learning. The federated learning method has an outer loop process that executes the inner loop process a second specified number of times. 8 . A model learning program that causes a computer to function as the model learning device according to claim 3 .