Method for determining hyperparameters, method for training federated learning model, and electronic device

Abstract

The present application discloses a method for determining hyperparameters, a method for training a federated learning model, and an electronic device, wherein the method for determining hyperparameters comprises: determining a first time consumption and a second time consumption by training a federated learning model; determining a first hyperparameter and / or a second hyperparameter based on the first time consumption, the second time consumption, a set gradient accuracy corresponding to a set loss function, and a first parameter; wherein the first time consumption represents the time consumed by a training node to calculate the gradient of a sample; the second time consumption represents the time from a first moment to a second moment; the first moment represents the moment when the training node uploads the second parameter to an aggregation node; the second moment represents the moment when the training node successfully downloads the corresponding third parameter after aggregation from the aggregation node; the first parameter represents the parameter used to calculate the convergence upper bound of the federated learning model; the first hyperparameter represents the number of samples used in one iterative training; the second hyperparameter represents the interval between the number of iterations for uploading the second parameter.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for determining hyperparameters, a method for training a federated learning model, and an electronic device. Background Art

[0002] Federated learning aims to establish a federated learning model based on a distributed data set without sharing data samples. In related technologies, when horizontal federated learning is performed based on the FedAvg (Federated Averaging) algorithm, it is necessary to manually adjust the hyperparameters to adjust the training time, but the manually set hyperparameters are inappropriate, resulting in too long training time for the federated learning model. Among them, the training nodes of horizontal federated learning have different data samples. Hyperparameters refer to parameters set manually before model training. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a method for determining hyper-parameters, a method for training a federated learning model, and an electronic device to solve the technical problem in the related art that the training time of the federated learning model is too long due to inappropriate manually set hyper-parameters.

[0004] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0005] The present application embodiment provides a method for determining a hyper-parameter, including:

[0006] By training the federated learning model, the first time consumption and the second time consumption are determined;

[0007] Based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, and the first parameter, a first hyperparameter and / or a second hyperparameter is determined; wherein,

[0008] The first time consumption represents the time consumed by the training node to calculate the gradient of a sample; the second time consumption represents the time from the first moment to the second moment; the first moment represents the moment when the training node uploads the second parameter to the aggregation node; the second moment represents the moment when the training node successfully downloads the corresponding aggregated third parameter from the aggregation node; the first parameter represents the parameter used to calculate the convergence upper bound of the federated learning model; the first hyperparameter represents the number of samples used in one iterative training; the second hyperparameter represents the interval between the number of iterations for uploading the second parameter.

[0009] In the above solution, determining the first hyperparameter and / or the second hyperparameter based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, and the first parameter includes:

[0010] Based on the set gradient accuracy and the first parameter corresponding to the set loss function, determining a restriction condition on the total number of iterations; the restriction condition represents a correlation between the set gradient accuracy, the total number of training nodes participating in training the federated learning model, the first hyperparameter, and the second hyperparameter;

[0011] Based on the determined constraints, and based on the correlation between the total training time and the first time, the second time, the total number of iterations, the first hyperparameter and the second hyperparameter, determine the first hyperparameter and / or second hyperparameter corresponding to the minimum total training time.

[0012] In the above scheme, the method further comprises:

[0013] The first super parameter and / or the second super parameter are adjusted by setting the first coefficient to obtain an adjusted first super parameter and / or an adjusted second super parameter.

[0014] When the first super parameter and / or the second super parameter is adjusted by setting the first coefficient, the method further includes:

[0015] Retraining the federated learning model based on the adjusted first hyperparameter and / or the adjusted second hyperparameter;

[0016] When the federated learning model is retrained but fails to converge and satisfies a first set condition, the first coefficient is adjusted; the first set condition represents reaching a set total number of iterations or reaching a set total training time.

[0017] In the above solution, when the first super parameter and / or the second super parameter are determined, the method further includes:

[0018] Based on at least one first hyperparameter and / or second hyperparameter corresponding to the first total training time, determine the first hyperparameter and / or second hyperparameter corresponding to the minimum first total training time; wherein,

[0019] The first total training time represents the total training time corresponding to the convergence of the federated learning model.

[0020] The present application also provides a method for training a federated learning model, including:

[0021] Based on the first hyperparameter and / or the second hyperparameter, the federated learning model is trained; wherein,

[0022] The first super-parameter and / or the second super-parameter is a super-parameter determined by the method for determining a super-parameter described in any of the above technical solutions.

[0023] The present application also provides an electronic device, including:

[0024] A first determining unit, configured to determine a first time consumption and a second time consumption by training a federated learning model;

[0025] A second determining unit is configured to determine a first hyperparameter and / or a second hyperparameter based on the first time consumption, the second time consumption, a set gradient accuracy corresponding to a set loss function, and a first parameter; wherein,

[0026] The first time consumption represents the time consumed by the training node to calculate the gradient of a sample; the second time consumption represents the time from the first moment to the second moment; the first moment represents the moment when the training node uploads the second parameter to the aggregation node; the second moment represents the moment when the training node successfully downloads the corresponding aggregated third parameter from the aggregation node; the first parameter represents the parameter used to calculate the convergence upper bound of the federated learning model; the first hyperparameter represents the number of samples used in one iterative training; the second hyperparameter represents the interval between the number of iterations for uploading the second parameter.

[0027] The present application also provides an electronic device, including:

[0028] A training unit is used to train a federated learning model based on a first hyperparameter and / or a second hyperparameter; wherein the first hyperparameter and / or the second hyperparameter are hyperparameters determined by the method for determining hyperparameters described in any of the above technical solutions.

[0029] The present application also provides an electronic device, including: a processor and a memory for storing a computer program that can be run on the processor.

[0030] Wherein, when the processor is used to run the computer program, at least one of the following is performed:

[0031] The steps of the method for determining hyper-parameters provided by any of the above technical solutions;

[0032] The steps of the above method for training a federated learning model.

[0033] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, at least one of the following is achieved:

[0034] The steps of the method for determining hyper-parameters provided by any of the above technical solutions;

[0035] The steps of the above method for training a federated learning model.

[0036] In an embodiment of the present application, the first time consumption and the second time consumption are determined by training the federated learning model; based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, and the first parameter, the first hyperparameter and / or the second hyperparameter are determined. Since the first time consumption and the second time consumption are related to the training environment (network environment and hardware environment of the training node, etc.) of the federated learning model, and the first hyperparameter or the second hyperparameter is a key hyperparameter that affects the training speed, the first hyperparameter and / or the second hyperparameter are determined based on the first time consumption and the second time consumption, and the influence of the training environment and the key hyperparameter on the training speed of the federated learning model is comprehensively considered. Compared with the method of manually adjusting the hyperparameter in the prior art, this solution can determine the optimal hyperparameter corresponding to the minimum total training time, improve the efficiency of finding the optimal hyperparameter, and when the optimal hyperparameter is used for training, the training time can be shortened and the training speed can be accelerated. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 System architecture diagram of horizontal federated learning for related technologies;

[0038] Figure 2 A schematic diagram of the implementation flow of the method for determining hyper-parameters provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of an implementation flow of adjusting a set first coefficient in a method for determining a hyper-parameter provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of the implementation flow of the method for training a federated learning model provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application;

[0043] Figure 7 Schematic diagram of the hardware structure of the electronic device according to the embodiment of the present application. DETAILED DESCRIPTION

[0044] In the related art, according to the different data distributions among multiple training nodes, federated learning is divided into three categories: horizontal federated learning, vertical federated learning, and federated transfer learning. Among them, horizontal federated learning is also called sample-based federated learning, which is applied to scenarios where the data sets of each training node have the same feature space and different sample spaces (different users), for example, the training nodes correspond to banks in different regions. Vertical federated learning has the same sample space and different feature spaces in the data set, and the essence of vertical federated learning is the union of features. Federated transfer learning is applied to scenarios where both feature space and sample space have little overlap.

[0045] Figure 1 A system architecture diagram of horizontal federated learning provided by related technologies is shown, such as Figure 1 As shown in Figure 1, the federated learning system includes at least two training nodes and one aggregation node.

[0046] The training nodes use the federated averaging algorithm to train the federated model based on their respective private data sets. After every several iterations of training, the latest parameters are uploaded to the aggregation node.

[0047] When receiving the latest parameters uploaded by all training nodes, the aggregation node aggregates all the latest parameters, for example, calculates the mean or weighted average of all the latest parameters to obtain the global parameters.

[0048] The training node downloads the corresponding global parameters from the aggregation node and continues training based on the downloaded global parameters. The training node uses the global parameters aggregated by the aggregation node based on the parameters uploaded by each training node when the set model convergence conditions are met as the final parameters of the model, and the federated learning model training is completed.

[0049] When the training nodes perform horizontal federated learning based on the federated average algorithm, it is necessary to manually adjust the multiple hyperparameters involved in the federated average algorithm to adjust the training duration in order to determine the optimal hyperparameters corresponding to the shortest training duration. However, the related technology does not provide recommended settings for the hyperparameters. After manually fine-tuning the hyperparameters for multiple times, the optimal hyperparameters still cannot be obtained, which results in the federated learning model training time being too long.

[0050] The applicant has found that by reasonably setting the hyperparameter batchsize or the interval of iterations between two adjacent uploads of the latest parameters, the training speed of the federated learning model can be improved. Based on this, an embodiment of the present application provides a method for determining hyperparameters, including: determining a first time consumption and a second time consumption by training the federated learning model; determining a first hyperparameter and / or a second hyperparameter based on the first time consumption, the second time consumption, a set gradient accuracy corresponding to a set loss function, and a first parameter; wherein the first time consumption represents the time consumed by the training node to calculate the gradient of a sample; the second time consumption represents the time from the first moment to the second moment; the first moment represents the moment when the training node uploads the second parameter to the aggregation node; the second moment represents the moment when the training node successfully downloads the corresponding aggregated third parameter from the aggregation node; the first parameter represents the parameter used to calculate the convergence upper bound of the federated learning model; the first hyperparameter represents the number of samples used in one iterative training; the second hyperparameter represents the interval of iterations for uploading the second parameter.

[0051] In the above scheme, since the first time consumption and the second time consumption are related to the training environment of the federated learning model (network environment and hardware environment of the training node, etc.), and the first hyperparameter or the second hyperparameter is a key hyperparameter that affects the training speed, the first hyperparameter and / or the second hyperparameter is determined based on the first time consumption and the second time consumption, and the influence of the training environment and the key hyperparameters on the training speed of the federated learning model is comprehensively considered. Compared with the method of manually adjusting the hyperparameters in the prior art, this scheme can determine the optimal hyperparameter corresponding to the minimum total training time, thereby improving the efficiency of finding the optimal hyperparameter. When the optimal hyperparameter is used for training, the training time can be shortened and the training speed can be accelerated.

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Figure 2 The following is a schematic diagram of the implementation process of the method for determining hyperparameters provided in an embodiment of the present application. The execution subject of the process is a training node, which is an electronic device such as a terminal or a server. Figure 2 As shown, the method for determining the hyperparameters includes:

[0054] Step 201: Determine a first time consumption and a second time consumption by training a federated learning model; wherein the first time consumption represents the time consumed by the training node to calculate the gradient of a sample; the second time consumption represents the time from the first moment to the second moment; the first moment represents the moment when the training node uploads the second parameter to the aggregation node; the second moment represents the moment when the training node successfully downloads the corresponding aggregated third parameter from the aggregation node.

[0055] Here, the training node uses the federated average algorithm to perform multiple iterations of training on the federated learning model based on samples in the private data set. During the multiple iterations of training, the first time consumption and the second time consumption corresponding to the training node are determined. The first time consumption is related to the data processing capability of the training node, and the second time consumption is related to the data transmission capability of the network, the data processing capability of other training nodes, and the data processing capability of the aggregation node.

[0056] When the federated learning model is trained for the first time using the federated averaging algorithm, multiple hyperparameters involved in the federated averaging algorithm are set according to the set initialization values.

[0057] When the training node iteratively trains the federated learning model, it inputs a batch of samples and the calibration value corresponding to each sample into the federated learning model, and obtains the predicted value corresponding to each sample in the batch of samples output by the federated learning model; uses the set loss function to calculate the loss value corresponding to each sample based on the predicted value and calibration value corresponding to each sample; calculates the gradient corresponding to each sample according to the chain rule; calculates the gradient corresponding to the federated learning model based on the gradient corresponding to each sample in the batch of samples, and updates the weight parameters of each layer of the federated learning model along the descending direction of the gradient.

[0058] The first time consumption can represent the average time consumed by the training node to calculate the gradient of each sample during one or more iterative training processes. For example, when a training node uses a batch of samples for training, it can calculate the total time consumed to calculate the gradients of all samples in the batch of samples based on the time consumed to calculate the gradient corresponding to each sample in the batch of samples, and determine the quotient between the total time and the total number of samples included in the batch of samples as the first time consumption. In actual application, the training node can perform multiple iterative trainings on the same batch of samples to calculate the first time consumption; it can also perform multiple iterative trainings on samples from different batches to calculate the first time consumption.

[0059] During multiple iterations of training, the training node uploads the second parameter to the aggregation node at least once and successfully downloads the corresponding third parameter after aggregation. The corresponding third parameter after aggregation is a parameter obtained by the aggregation node based on the second parameters uploaded by all training nodes for the kth time, where k is a positive integer. In actual application, the second parameter includes at least one of the latest model parameters and the gradient calculated by the training node, and the third parameter corresponds to the second parameter.

[0060] The second duration may be a first duration determined based on a first moment and a corresponding second moment, or an average duration determined based on at least two first durations, wherein the first duration refers to the duration from a first moment to the corresponding second moment.

[0061] For example, the first moment may be the moment when the training node first uploads the second parameter to the aggregation node, and the corresponding second moment may be the moment when the corresponding third parameter is first successfully downloaded from the aggregation node. The third parameter refers to a parameter obtained by the aggregation node based on the second parameters first uploaded by all training nodes.

[0062] Step 202: Determine a first hyperparameter and / or a second hyperparameter based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, and the first parameter; wherein the first parameter represents the parameter used to calculate the convergence upper bound of the federated learning model; the first hyperparameter represents the number of samples used in one iterative training; the second hyperparameter represents the iteration interval for uploading the second parameter.

[0063] Here, the training node determines at least one of the first super parameter and the second super parameter based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, the first parameter, the first set expression, the second set expression, and the third set expression.

[0064] The first setting expression is the expression corresponding to the total training time. The first setting expression is:

[0065]

[0066] Among them, T represents the total training time, which refers to the time from the start of training the federated learning model to the end of training the federated learning model; K represents the total number of iterations; B is the first hyperparameter, which represents the number of samples used in one iterative training of a single training node, that is, the batch size corresponding to a single iteration of the training node, and B is a positive integer; t1 represents the first time; τ is the second hyperparameter, which represents the iteration interval for uploading the second parameter, that is, the number of iterative training performed by the training node within the time of two adjacent uploads of the second parameter; t2 represents the second time.

[0067] The second setting expression is the expression of the convergence upper bound of the federated learning model. The second setting expression is:

[0068]

[0069] in, It refers to the binary norm of the gradient of the federated learning model, which represents the convergence upper bound of the federated learning model; C is a constant; W represents the total number of training nodes involved in training the federated learning model. For example, when the federated learning model is jointly trained by 4 training nodes, W is equal to 4. The first parameter includes W, B, K and τ, and may also include C. It should be noted that in each embodiment of the present application, the federated learning model is trained using the federated average algorithm, and the federated average algorithm is applied to horizontal federated learning. Therefore, the federated learning model in each embodiment of the present application refers to a horizontal federated learning model.

[0070] The third setting expression is:

[0071]

[0072] Among them, ∈ represents the set gradient accuracy corresponding to the set loss function, that is, the target gradient accuracy of the set loss function.

[0073] The training node determines the calculation formula of the first hyperparameter and the calculation formula of the second hyperparameter based on the first setting expression, the second setting expression and the third setting expression; substitutes the first time consumption t1, the second time consumption t2, and the set gradient accuracy ∈ corresponding to the set loss function into the calculation formula of the first hyperparameter to obtain the first hyperparameter B, and substitutes t1, t2 and ∈ into the calculation formula of the second hyperparameter to obtain the second hyperparameter τ. Among them, the calculation formula of the first hyperparameter and the calculation formula of the second hyperparameter both represent the correlation between W, t1, t2 and ∈. In actual application, the calculation formula of the first hyperparameter is: The calculation formula for the second hyperparameter is: When B is minimum and τ is maximum, the total training time is minimum.

[0074] In some embodiments, determining the first hyperparameter and / or the second hyperparameter based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, and the first parameter includes:

[0075] Based on the set gradient accuracy and the first parameter corresponding to the set loss function, determining a restriction condition on the total number of iterations; the restriction condition represents a correlation between the set gradient accuracy, the total number of training nodes participating in training the federated learning model, the first hyperparameter, and the second hyperparameter;

[0076] Based on the determined constraints, and based on the correlation between the total training time and the first time, the second time, the total number of iterations, the first hyperparameter and the second hyperparameter, determine the first hyperparameter and / or second hyperparameter corresponding to the minimum total training time.

[0077] Here, the training node determines the implementation process of the restriction condition of the total number of iterations based on the set gradient accuracy and the first parameter corresponding to the set loss function as follows:

[0078] The training node determines the Based on Determine the following relationship:

[0079]

[0080]

[0081]

[0082] The training nodes transform equations 4 to 6 respectively to obtain:

[0083]

[0084]

[0085]

[0086] The training node determines the corresponding total number of iterations when equations 7 to 9 are satisfied:

[0087] Since B is a positive integer, the training node simplifies the restriction on the total number of iterations to:

[0088]

[0089] When the training node determines Formula 10, based on the determined restriction conditions and the correlation between the total training time and the first time, the second time, the total number of iterations, the first super parameter and the second super parameter in Formula 1, the first super parameter and / or the second super parameter corresponding to the minimum total training time are determined. The implementation process of determining the first super parameter and / or the second super parameter corresponding to the minimum total training time based on Formula 1 and Formula 10 is as follows:

[0090] Because when hour, when hour, Therefore, the training node determines the conditions 1 and 2 corresponding to the total number of iterations by formula 10; based on Condition 1 and the above formula 1, determine the first hyperparameter and / or second hyperparameter corresponding to the minimum total training time; based on Condition 2 and the above formula 1 determine the first hyperparameter and / or second hyperparameter corresponding to the minimum total training time. Among them, condition 1 is Simplifying condition 1, we get Condition 2 is Simplifying condition 2, we get

[0091] The following are respectively introduced in and In both cases, the implementation process of determining the first hyperparameter and / or second hyperparameter corresponding to the minimum total training time is as follows:

[0092] Case 1: In In the case of Substituting into the above formula 1, we get:

[0093]

[0094] Since τ is maximized when t1, t2, W and ∈ are known, the total training time T can be minimized. Therefore, when τ is maximized, the simplified condition 1 is: Can be determined The training node will Substituting into the above formula 11, we get:

[0095]

[0096] Training nodes based on mean inequality The above formula 12 is processed to obtain:

[0097]

[0098] From formula 13, we can see that when and When the total training time T is the smallest, and Simplifying, we get Will Substitution get Will and Substituting into the above formula 11, we get

[0099] Among them, it can be seen from the above formula 1 that T is positively correlated with B, and T is negatively correlated with τ, that is, when τ remains unchanged, the smaller B is, the smaller T is; when B remains unchanged, the larger τ is, the smaller T is; therefore, the first hyperparameter B takes the minimum value The second hyperparameter τ takes the maximum value When , the total training time T can be minimized, that is, the first hyperparameter corresponding to the minimum total training time T The second super parameter

[0100] Case 2: In In the case of Substituting into the above formula 1, we get:

[0101]

[0102] Since when t1, t2, W and ∈ are known, the total training time T can be minimized only when τ is minimized. Therefore, when τ is minimized, the simplified condition 2 is: Can be determined The training node will Substituting into the above formula 14, we get:

[0103]

[0104] Training nodes based on mean inequality The above formula 15 is processed to obtain:

[0105]

[0106] From formula 16, we can see that when and When the total training time T is the smallest, and Simplifying, we get Will Substitution get Will and Substituting into the above formula 15, we get Therefore, the first hyperparameter corresponding to the minimum total training time T The second super parameter

[0107] Combining the calculation results of the above cases 1 and 2, we can see that the calculation results of the two cases are the same. The first hyperparameter corresponding to the minimum total training time T is The second super parameter That is to say, and Both are optimal hyperparameters.

[0108] The training node substitutes the value of W, the set gradient accuracy corresponding to the set loss function, and the first time t1 and the second time t2 determined in step 201 into and Get the first hyperparameter and / or the second hyperparameter.

[0109] It should be noted that, in some embodiments, the training node may store a calculation formula for the first hyperparameter: And the calculation formula of the second hyperparameter Where W and ∈ are set values. When t1 and t2 are determined, the training node substitutes W, ∈, t1 and t2 into the calculation formula of the first hyperparameter The value of B can be obtained; substitute W,∈, t1 and t2 into the calculation formula of the second hyperparameter The value of τ can be obtained.

[0110] Considering that the theoretical operating environment and the actual operating environment of the training node have some deviations, in order to obtain more accurate optimal hyperparameters so as to minimize the total training time of the federated learning model, in some examples, the method further includes:

[0111] The first super parameter and / or the second super parameter are adjusted by setting the first coefficient to obtain an adjusted first super parameter and / or an adjusted second super parameter.

[0112] Here, the first coefficient set represents the tuning parameter, and the adjusted first hyperparameter is: The adjusted second hyperparameter is: α is the first coefficient that is set, and α is greater than zero.

[0113] In this embodiment, the first hyper-parameter and the second hyper-parameter can be adjusted simultaneously by setting the first coefficient, which can reduce the time consumed in adjusting the hyper-parameters, thereby improving the efficiency of hyper-parameter optimization.

[0114] In order to more accurately determine the first hyperparameter and the second hyperparameter corresponding to the minimum total training time, in some embodiments, when adjusting the first hyperparameter and / or the second hyperparameter by the set first coefficient, the method further includes:

[0115] Retraining the federated learning model based on the adjusted first hyperparameter and / or the adjusted second hyperparameter;

[0116] When the federated learning model is retrained but fails to converge and satisfies a first set condition, the first coefficient is adjusted; the first set condition indicates that a set total number of iterations is reached or a set total training time is exceeded.

[0117] Please refer to Figure 3, the training node determines the first time consumption and the second time consumption by training the federated learning model; when the first time consumption and the second time consumption are determined, the first hyperparameter and / or the second hyperparameter are determined based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, and the first parameter; the determined first hyperparameter and / or the second hyperparameter are adjusted by the set first coefficient; the federated learning algorithm is used to train the federated learning model again based on the adjusted first hyperparameter and / or the second hyperparameter; during the training process, it is determined whether the federated learning model converges to obtain a first judgment result; and it is determined whether the first set condition is met to obtain a second judgment result. Among them, the first set condition represents reaching the set total number of iterations or exceeding the set total training time.

[0118] When the first judgment result indicates that the federated learning model has converged, and the second judgment result indicates that the first set condition is not met, the training of the federated learning model ends, and the total time consumed for the first training and the corresponding values ​​of the first hyperparameter and the second hyperparameter are recorded. In other words, when the federated learning model converges and the total number of iterations is less than or equal to the set total number of iterations, the training of the federated learning model ends; or, when the federated learning model converges and the total training time is less than or equal to the set total number of iterations, the training of the federated learning model ends.

[0119] When the first judgment result indicates that the federated learning model has not converged and does not meet the first set condition, iterative training continues.

[0120] When the first judgment result indicates that the federated learning model has not converged and satisfies the first set condition, the iterative training is stopped, and the set first coefficient is adjusted, so that the first hyperparameter and / or the second hyperparameter are adjusted again through the adjusted set first coefficient, and subsequent steps are performed. In actual application, the training node can adjust the set first coefficient based on the set first coefficient with a set step value. The set first coefficient can be 1; the set step value can be 0.1 or -0.1.

[0121] In this embodiment, when the federated learning model is re-trained and fails to converge and meets the first set condition, the set first coefficient is adjusted, and the first hyperparameter and / or the second hyperparameter is further adjusted by the adjusted first coefficient to more accurately determine the first hyperparameter and / or the second hyperparameter corresponding to the minimum total training time.

[0122] Considering that the first total time consumption corresponding to each convergence of the federated learning model may be different, in order to accurately determine the first hyperparameter and / or the second hyperparameter corresponding to the minimum total training time consumption; in some embodiments, when determining the first hyperparameter and / or the second hyperparameter, the method further includes:

[0123] Based on at least one first hyperparameter and / or second hyperparameter corresponding to the first total training time, determine the first hyperparameter and / or second hyperparameter corresponding to the minimum first total training time; wherein,

[0124] The first total training time represents the total training time corresponding to the convergence of the federated learning model.

[0125] Here, the training node determines the minimum first training total time consumption from all recorded first training total time consumptions, and determines the first super-parameters and the second super-parameters corresponding to the minimum first training total time consumption as the final first super-parameters and / or second super-parameters.

[0126] In the embodiment of the present application, the first time consumption and the second time consumption are determined by training the federated learning model; based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function and the first parameter, the first hyperparameter and / or the second hyperparameter are determined. Since the first time consumption and the second time consumption are related to the training environment (network environment and hardware environment of the training node, etc.) of the federated learning model, and the first hyperparameter or the second hyperparameter is a key hyperparameter that affects the training speed, the first hyperparameter and / or the second hyperparameter are determined based on the first time consumption and the second time consumption, and the influence of the training environment and the key hyperparameter on the training speed of the federated learning model is comprehensively considered. Compared with the method of manually adjusting the hyperparameter in the prior art, this solution can determine the optimal hyperparameter corresponding to the minimum total training time, improve the efficiency of finding the optimal hyperparameter, and when the optimal hyperparameter is used for training, the training time can be shortened and the training speed can be accelerated.

[0127] As another embodiment of the present application, Figure 4 The following is a schematic diagram of the implementation process of the method for training a federated learning model provided in an embodiment of the present application. The execution subject of the process is a training node, such as Figure 4 As shown, the method for training the federated learning model includes:

[0128] Step 401: training the federated learning model based on the first hyperparameter and / or the second hyperparameter; wherein:

[0129] The first super-parameter and / or the second super-parameter is a super-parameter determined by the method for determining a super-parameter provided in any of the above embodiments.

[0130] Here, when the training node determines the first hyperparameter and / or the second hyperparameter by using the method for determining the hyperparameter provided in any of the above embodiments, it initializes the hyperparameters other than the first hyperparameter and / or the second hyperparameter, and trains the federated learning model based on the first hyperparameter and / or the second hyperparameter by using the federated average algorithm; when the set model convergence condition is met, the federated learning model training ends, and the global parameters obtained by the aggregation node based on the parameters uploaded by each training node when the set model convergence condition is met are used as the final parameters of the model. In actual application, the set model convergence condition can be that the gradient corresponding to the loss function is less than or equal to the set gradient accuracy ∈.

[0131] Among them, in the process of training the federated learning model, the training node determines the number of samples corresponding to the first hyperparameter from the private data set, and inputs the determined samples into the federated learning model for iterative training; when the number of iterative training reaches the number of iterations represented by the second hyperparameter, the latest parameters after iterative update are uploaded to the aggregation node; the aggregation node aggregates the parameters uploaded by N training nodes for the kth time to obtain the global parameters; the training node downloads the corresponding global parameters from the aggregation node, and continues to perform iterative training based on the global parameters until the federated learning model converges.

[0132] In this embodiment, the training node trains the federated learning model based on the first hyperparameter and / or the second hyperparameter corresponding to the minimum total training consumption, which can shorten the training time of the federated learning model and speed up the training.

[0133] In order to implement the method for determining the hyper-parameters in the embodiment of the present application, the embodiment of the present application also provides an electronic device, such as Figure 5 As shown, the electronic device includes:

[0134] A first determining unit 51 is used to determine a first time consumption and a second time consumption by training a federated learning model;

[0135] The second determining unit 52 is used to determine the first hyperparameter and / or the second hyperparameter based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, and the first parameter; wherein,

[0136] The first time consumption represents the time consumed by the training node to calculate the gradient of a sample; the second time consumption represents the time from the first moment to the second moment; the first moment represents the moment when the training node uploads the second parameter to the aggregation node; the second moment represents the moment when the training node successfully downloads the corresponding aggregated third parameter from the aggregation node; the first parameter represents the parameter used to calculate the convergence upper bound of the federated learning model; the first hyperparameter represents the number of samples used in one iterative training; the second hyperparameter represents the interval between the number of iterations for uploading the second parameter.

[0137] In some embodiments, the second determining unit 52 is configured to:

[0138] Based on the set gradient accuracy and the first parameter corresponding to the set loss function, determining a restriction condition on the total number of iterations; the restriction condition represents a correlation between the set gradient accuracy, the total number of training nodes participating in training the federated learning model, the first hyperparameter, and the second hyperparameter;

[0139] Based on the determined constraints, and based on the correlation between the total training time and the first time, the second time, the total number of iterations, the first hyperparameter and the second hyperparameter, determine the first hyperparameter and / or second hyperparameter corresponding to the minimum total training time.

[0140] In some embodiments, the electronic device further comprises:

[0141] The first adjustment unit is used to adjust the first super parameter and / or the second super parameter by setting a first coefficient to obtain an adjusted first super parameter and / or an adjusted second super parameter.

[0142] In some embodiments, the electronic device further comprises:

[0143] A training unit, configured to retrain the federated learning model based on the adjusted first hyperparameter and / or the adjusted second hyperparameter;

[0144] The second adjustment unit is used to adjust the first coefficient when the federated learning model is re-trained but fails to converge and satisfies a first set condition; the first set condition represents reaching a set total number of iterations or reaching a set total training time.

[0145] In some embodiments, the second determination unit 52 is used to: determine the first hyperparameter and / or second hyperparameter corresponding to the minimum first total training time based on at least one first total training time; wherein the first total training time represents the corresponding total training time when the federated learning model converges.

[0146] In actual application, the above-mentioned units can be implemented by a processor in an electronic device, such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU) or a field-programmable gate array (FPGA).

[0147] It should be noted that: when the electronic device provided in the above embodiment determines the hyper-parameters, only the division of the above-mentioned program modules is used as an example. In practical applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the electronic device provided in the above embodiment and the method embodiment for determining the hyper-parameters belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0148] In order to implement the method for training a federated learning model in the embodiment of the present application, the embodiment of the present application also provides an electronic device, such as Figure 6 As shown, the electronic device includes:

[0149] The training unit 61 is used to train the federated learning model based on the first hyperparameter and / or the second hyperparameter; wherein the first hyperparameter and / or the second hyperparameter are hyperparameters determined by the method for determining the hyperparameter provided in any of the above embodiments.

[0150] In actual application, the training unit 61 can be implemented by a processor in the electronic device, such as a central processing unit CPU, DSP, MCU or FPGA.

[0151] It should be noted that: when the electronic device provided in the above embodiment trains the federated learning model, only the division of the above program modules is used as an example. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the electronic device provided in the above embodiment and the method embodiment for training the federated learning model belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0152] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Figure 7 Schematic diagram of the hardware structure of the electronic device of the present application embodiment. Figure 7 As shown, the electronic device 7 includes:

[0153] Communication interface 71, capable of exchanging information with other devices such as network devices;

[0154] The processor 72 is connected to the communication interface 71 to implement information exchange with other devices, and is used to execute the method for determining the hyper-parameter provided by one or more of the above technical solutions, or the method for training the federated learning model provided by the above technical solutions when running the computer program. The computer program is stored in the memory 73.

[0155] Of course, in actual application, the various components in the electronic device 7 are coupled together through the bus system 74. It can be understood that the bus system 74 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 74 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 74 .

[0156] The memory 73 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 7. Examples of such data include: any computer program used to operate on the electronic device 7.

[0157] It can be understood that the memory 73 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 73 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memory.

[0158] The method disclosed in the above embodiment of the present application can be applied to the processor 72, or implemented by the processor 72. The processor 72 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 72 or the instruction in the form of software. The above processor 72 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 72 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 73, and the processor 72 reads the program in the memory 73 and completes the steps of the above method in combination with its hardware.

[0159] Optionally, when the processor 72 executes the program, it implements the corresponding processes implemented by the terminal in each method of the embodiments of the present application, which will not be described here for the sake of brevity.

[0160] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, including a first memory 73 storing a computer program, and the computer program can be executed by a processor 72 of the terminal to complete the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0162] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0163] In addition, all functional units in the embodiments of the present application may be integrated into one processing module, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0164] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0165] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0166] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining hyperparameters, characterized in that: include: By training the federated learning model, the first time consumption and the second time consumption are determined; Based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, and the first parameter, a first hyperparameter and / or a second hyperparameter is determined; wherein, The first time consumption represents the time consumed by the training node to calculate the gradient of a sample; the second time consumption represents the time from the first moment to the second moment; the first moment represents the moment when the training node uploads the second parameter to the aggregation node; the second moment represents the moment when the training node successfully downloads the corresponding aggregated third parameter from the aggregation node; the first parameter represents the parameter used to calculate the convergence upper bound of the federated learning model; the first hyperparameter represents the number of samples used in one iterative training; the second hyperparameter represents the interval between the number of iterations for uploading the second parameter; The determining of the first hyperparameter and / or the second hyperparameter based on the first time consumption, the second time consumption, the set gradient accuracy corresponding to the set loss function, and the first parameter includes: Based on the set gradient accuracy and the first parameter corresponding to the set loss function, determining a restriction condition on the total number of iterations; the restriction condition represents a correlation between the set gradient accuracy, the total number of training nodes participating in training the federated learning model, the first hyperparameter, and the second hyperparameter; Based on the determined constraints, and based on the correlation between the total training time and the first time, the second time, the total number of iterations, the first hyperparameter and the second hyperparameter, determine the first hyperparameter and / or second hyperparameter corresponding to the minimum total training time.

2. The method according to claim 1, characterized in that The method further comprises: The first super parameter and / or the second super parameter are adjusted by setting the first coefficient to obtain an adjusted first super parameter and / or an adjusted second super parameter.

3. The method according to claim 2, characterized in that When the first super parameter and / or the second super parameter is adjusted by setting the first coefficient, the method further includes: Retraining the federated learning model based on the adjusted first hyperparameter and / or the adjusted second hyperparameter; When the federated learning model is retrained but fails to converge and satisfies a first set condition, the first coefficient is adjusted; the first set condition indicates that a set total number of iterations is reached or a set total training time is exceeded.

4. The method according to claim 3, characterized in that When the first super parameter and / or the second super parameter are determined, the method further includes: Based on at least one first hyperparameter and / or second hyperparameter corresponding to the first total training time, determine the first hyperparameter and / or second hyperparameter corresponding to the minimum first total training time; wherein, The first total training time represents the total training time corresponding to the convergence of the federated learning model.

5. A method for training a federated learning model, characterized in that: include: Based on the first hyperparameter and / or the second hyperparameter, the federated learning model is trained; wherein, The first super-parameter and / or the second super-parameter is a super-parameter determined by the method for determining a super-parameter according to any one of claims 1 to 4.

6. An electronic device, characterized in that: include: A first determining unit, configured to determine a first time consumption and a second time consumption by training a federated learning model; A second determining unit is configured to determine a first hyperparameter and / or a second hyperparameter based on the first time consumption, the second time consumption, a set gradient accuracy corresponding to a set loss function, and a first parameter; wherein, The first time consumption represents the time consumed by the training node to calculate the gradient of a sample; the second time consumption represents the time from the first moment to the second moment; the first moment represents the moment when the training node uploads the second parameter to the aggregation node; the second moment represents the moment when the training node successfully downloads the corresponding aggregated third parameter from the aggregation node; the first parameter represents the parameter used to calculate the convergence upper bound of the federated learning model; the first hyperparameter represents the number of samples used in one iterative training; the second hyperparameter represents the interval between the number of iterations for uploading the second parameter; Among them, the second determination unit is specifically used to determine the restriction condition of the total number of iterations based on the set gradient accuracy corresponding to the set loss function and the first parameter; the restriction condition represents the set gradient accuracy, the total number of training nodes involved in training the federated learning model, the correlation between the first hyperparameter and the second hyperparameter; based on the determined restriction condition, and based on the correlation between the total training time and the first time, the second time, the total number of iterations, the first hyperparameter and the second hyperparameter, determine the first hyperparameter and / or second hyperparameter corresponding to the minimum total training time.

7. An electronic device, characterized in that: include: A training unit, used to train a federated learning model based on a first hyperparameter and / or a second hyperparameter; wherein the first hyperparameter and / or the second hyperparameter is a hyperparameter determined by the method for determining a hyperparameter as described in any one of claims 1 to 4.

8. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, at least one of the following is performed: The steps of the method for determining hyper-parameters according to any one of claims 1 to 4; The steps of the method for training a federated learning model as described in claim 5.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, at least one of the following is achieved: The steps of the method for determining hyper-parameters according to any one of claims 1 to 4; The steps of the method for training a federated learning model as described in claim 5.

Images