A federated learning time consumption prediction method, device, storage medium and program product

Abstract

The application discloses a federated learning time consumption prediction method and device, a storage medium and a program product, relates to the technical field of computers, and comprises the following steps: determining the calculation delay of local federated learning model training corresponding to each participant node, determining the communication delay of the model parameter update process between the participant nodes, and dividing the node clusters according to the calculation delay and the communication delay. The consistency of the time intervals corresponding to the participant nodes of the same layer in the divided node clusters is improved, the participant nodes with similar calculation capabilities and communication delays are divided into the same node cluster for federated learning training, the influence of the heterogeneity of the participant nodes is greatly reduced, the communication amount of the federated learning is reduced, the technical problems of low training efficiency of the federated learning and low accuracy of federated learning time consumption prediction are solved, and the technical effects of improving the training efficiency of the federated learning and improving the accuracy of the federated learning time consumption prediction are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and particularly relates to a federated learning time consumption prediction method, device, storage medium and program product. BACKGROUND

[0002] The core idea of federated learning is to distribute the model training process to multiple clients, each of which trains the model on local data and sends the updated model parameters to the central server for aggregation.

[0003] The commonly used aggregation algorithm is the federal average algorithm (Fed Avg), which can only aggregate model parameters according to the size of the data of each participant at the central parameter server, without considering the heterogeneity of the participant devices, so that the central parameter server cannot simultaneously or timely receive the model parameter update information of all participants when updating the parameters each round, which leads to the central parameter server being unable to perform the federated average algorithm model parameter aggregation process, and further leads to low training efficiency of federated learning and low accuracy of federated learning time consumption prediction. SUMMARY

[0004] The present application provides a federated learning time consumption prediction method, device, storage medium and program product to at least solve the problems of low training efficiency of federated learning and low accuracy of federated learning time consumption prediction in related technologies.

[0005] The present application provides a federated learning time consumption prediction method, comprising:

[0006] determining the computing delay of the local federated learning model training of each participant node respectively;

[0007] determining the communication delay of the model parameter update process between each participant node;

[0008] determining the time interval corresponding to the model parameter aggregation between each participant node according to each computing delay and each communication delay;

[0009] dividing the participant nodes into node clusters according to each time interval, to obtain each node cluster;

[0010] predicting the federated learning time consumption according to each node cluster.

[0011] The present application also provides an electronic device comprising a memory for storing a computer program and a processor for executing the computer program to implement the steps of any of the above federated learning time consumption prediction methods.

[0012] The application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program.

[0013] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of any of the federated learning time consumption prediction methods.

[0014] According to the application, the consistency of the time intervals corresponding to the same layer of the participant nodes in the divided node cluster is improved by determining the calculation delay of the local federated learning model training of each participant node and the communication delay of the model parameter update process between the participant nodes, and the participant nodes with similar calculation capability and communication delay are divided into the same node cluster for federated learning training, so that the influence of the heterogeneity of the participant nodes is greatly reduced, and the communication amount of the federated learning is reduced, thereby the technical problems of low training efficiency of the federated learning and low accuracy of the federated learning time consumption prediction can be solved, and the technical effects of improving the training efficiency of the federated learning and improving the accuracy of the federated learning time consumption prediction are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0016] Figure 1 An implementation flowchart of a federated learning time consumption prediction method provided for the embodiments of the application is provided.

[0017] Figure 2 An implementation flowchart of another federated learning time consumption prediction method provided for the embodiments of the application is provided.

[0018] Figure 3 A hierarchical federated learning framework diagram based on a trusted execution environment provided for the embodiments of the application is provided.

[0019] Figure 4 A parameter update schematic diagram based on the similarity of communication delay and calculation capability provided for the embodiments of the application is provided.

[0020] Figure 5 A structural block diagram of a federated learning time consumption prediction device provided for the embodiments of the application is provided. DETAILED DESCRIPTION

[0021] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described in order to make the technical solutions in the embodiments of the present application apparent to one of ordinary skill in the art. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by one of ordinary skill in the art without creative work shall fall within the protection scope of the present application.

[0022] It should be noted that, in the description of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover the non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0023] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0024] The embodiments of the present application provide a federated learning time consumption prediction method, and the method is described in detail in combination with the execution process of the federated learning time consumption prediction method.

[0025] Referring to Figure 1 , Figure 1 An implementation flowchart of a federated learning time consumption prediction method provided by the embodiments of the present application, which can include the following steps.

[0026] S101: Determine the computing delay of the local federated learning model training corresponding to each participant node.

[0027] The central parameter server and the plurality of participant nodes jointly participate in federated learning, and each participant node will perform local federated learning model training using the data set stored locally. The computing delay of the local federated learning model training corresponding to each participant node is determined. For example, the computing delay of the local federated learning model training corresponding to each participant node can be calculated according to the number of iterations of a single local federated learning model training, the number of central processing unit (CPU) cycles required for the participant node to train 1 bit of data, the amount of data involved in a single local federated learning model training update, and the device computing capability of the participant node for a single local federated learning model training, thereby realizing accurate calculation of the computing delay of the local federated learning model training corresponding to each participant node.

[0028] A layered federated learning framework is composed of a central parameter aggregation server and N participant nodes, and each participant node constitutes a set Any bottommost participant node is only associated with an intermediate layer participant node The local data set owned by the bottommost participant node is The local data set owned by the intermediate layer participant node is .

[0029] The calculation formula of the calculation delay of the local federated learning model training of each participant node is as follows:

[0030] ;

[0031] Wherein, is the iteration number of single local federated learning model training of the participant node , is the number of central processor cycles required for training 1 bit of data at the participant node, is the data volume involved in single local federated learning model training update, is the device computing power of single local federated learning model training of the participant node , is the calculation delay of the local federated learning model training of the participant node .

[0032] S102: Determine the communication delay of the model parameter update process between each participant node.

[0033] There is communication delay in the model parameter update between each participant node, and the communication delay of the model parameter update process between each participant node is determined. For example, the model size uploaded by the participant node model parameter update and the transmission rate between each participant node can be used to calculate the communication delay of the model parameter update process between each participant node.

[0034] The calculation formula of the communication delay of the model parameter update process between each participant node is as follows:

[0035] ;

[0036] Wherein, is the model size uploaded by the model parameter update of the participant node i, is the transmission rate between the participant node and the participant node , is the communication delay of the model parameter update process between the participant node Communication delay of model parameter update process between participant nodes .

[0037] S103: Determine time intervals corresponding to model parameter aggregation between participant nodes respectively according to each calculation delay and each communication delay.

[0038] After determining the calculation delay of local federated learning model training of each participant node respectively and determining the communication delay of model parameter update process between each participant node, determine time intervals corresponding to model parameter aggregation between participant nodes respectively according to each calculation delay and each communication delay.

[0039] Participant node Relative participant node The calculation delay matrix of local federated learning model training can be expressed as:

[0040] .

[0041] The communication delay matrix between participant nodes can be expressed as:

[0042] .

[0043] Participant node Relative participant node Time interval matrix of single execution of model aggregation is:

[0044] .

[0045] S104: Cluster division of participant nodes according to each time interval, to obtain each node cluster.

[0046] After determining the time intervals corresponding to model parameter aggregation between each participant node respectively, cluster division of participant nodes is performed according to each time interval, to obtain each node cluster. For example, participant nodes with equivalent time intervals can be divided into the same node cluster, so as to ensure that the time delay of each participant node in a single node cluster has high consistency, thereby improving the training efficiency of federated learning.

[0047] S105: Time consumption prediction of federated learning according to each node cluster.

[0048] After the node clusters are divided, federated learning time consumption prediction is performed according to the node clusters. The federated learning time consumption prediction is realized by performing federated learning on the divided node clusters, and performing time consumption statistics on the federated learning process. By dividing the participant nodes with similar computing capability and communication delay into the same node cluster for federated learning training, the influence of the heterogeneity of the participant nodes is greatly reduced, the communication amount of the federated learning is reduced, the training efficiency of the federated learning is improved, and the federated learning time consumption prediction accuracy is improved.

[0049] According to the present application, by determining the computing delay of the local federated learning model training of each participant node, determining the communication delay of the model parameter update process between the participant nodes, and dividing the node clusters according to the computing delay and the communication delay, the consistency of the time intervals of the participant nodes of the same layer in the divided node clusters is improved, so that the participant nodes with similar computing capability and communication delay are divided into the same node cluster for federated learning training, the influence of the heterogeneity of the participant nodes is greatly reduced, and the communication amount of the federated learning is reduced. Therefore, the technical problems of low training efficiency of federated learning and low federated learning time consumption prediction accuracy can be solved, and the technical effects of improving the training efficiency of federated learning and improving the federated learning time consumption prediction accuracy are achieved.

[0050] Referring to Figure 2 , Figure 2 The embodiment of the present application provides another federated learning time consumption prediction method, which can include the following steps.

[0051] S201: Determine the computing delay of the local federated learning model training of each participant node.

[0052] Considering that the central parameter server has strong computing capability, and the actual scene is that the downlink channel bandwidth is often much higher than the uplink channel bandwidth, the computing delay of the server and the downlink transmission delay are small, so the system time delay can be ignored, and only the communication delay of the participant node model parameter update upload process and the computing delay of the participant node local federated learning model training are considered. At this time, the system time delay mainly includes the computing delay of the participant node local federated learning model training and the communication delay of the participant node model parameter update upload process.

[0053] S202: Determine the communication delay of the model parameter update process between the participant nodes.

[0054] S203: Determine the time interval corresponding to the model parameter aggregation between the participant nodes according to each computing delay and each communication delay.

[0055] S204: Select the current minimum value from the current time interval set formed by the time intervals.

[0056] After determining the time intervals corresponding to the model parameter aggregation between each participant, a current minimum value is selected from the current time interval set composed of each time interval.

[0057] Taking an example of the step S103, assuming that the minimum value is selected from the time interval matrix .

[0058] S205: Determine the current minimum value as the current minimum time interval.

[0059] After selecting the current minimum value from the current time interval set composed of each time interval, the current minimum value is determined as the current minimum time interval.

[0060] S206: Obtain the participant node corresponding to the current minimum time interval, and determine the participant node corresponding to the current minimum time interval as the current node cluster head.

[0061] After determining the current minimum time interval, the participant node corresponding to the current minimum time interval is obtained, and the participant node corresponding to the current minimum time interval is determined as the current node cluster head.

[0062] Taking an example of the step S204, the minimum value corresponding participant node m is determined as the current node cluster head.

[0063] S207: Select each communication delay for model parameter update to the current node cluster head from each communication delay, and sort each selected communication delay in size to obtain a sorting result.

[0064] After determining the current node cluster head, each communication delay for model parameter update to the current node cluster head is selected from each communication delay, and each selected communication delay is sorted in size to obtain a sorting result.

[0065] S208: Construct the current node cluster according to the sorting result.

[0066] After sorting each selected communication delay in size to obtain a sorting result, the current node cluster is constructed according to the sorting result, such as selecting a certain number of communication delays from the smaller end of the communication delay, determining the participant nodes corresponding to each communication delay, and constructing the current node cluster with the current node cluster head. By selecting the participant nodes according to the communication delay sorting result, the consistency of the communication delays corresponding to each participant node selected into the same node cluster is improved, the influence of the heterogeneity of the participant nodes is reduced, and the communication amount of the federated learning is reduced.

[0067] In one specific embodiment of the present application, step S208 can include the following steps:

[0068] Step one: selecting the first preset number of communication delays from the end with small communication delays according to the sorting result, and determining the selected first preset number of communication delays as each target delay;

[0069] Step two: selecting the time interval corresponding to the maximum value of each target delay from each time interval, and determining the selected time interval as the target time interval;

[0070] Step three: determining whether the target time interval is less than or equal to the preset time threshold, if yes, executing step four, if not, executing step five;

[0071] Step four: constructing the current node cluster according to the current node cluster head and the participant nodes corresponding to each target delay;

[0072] Step five: selecting the minimum value of the time intervals other than the current minimum time interval from the current time interval set;

[0073] Step six: determining the selected minimum value as the new current minimum value, and returning to execute step S205.

[0074] For convenience of description, the above six steps can be combined for description.

[0075] According to the sorting result, the first preset number of communication delays are selected from the end with small communication delays, and the selected first preset number of communication delays are determined as each target delay. The time interval corresponding to the maximum value of each target delay is selected from each time interval, and the selected time interval is determined as the target time interval. It is determined whether the target time interval is less than or equal to the preset time threshold. If yes, it indicates that the current node cluster head and the participant nodes corresponding to each target delay meet the node cluster construction condition. The current node cluster is constructed according to the current node cluster head and the participant nodes corresponding to each target delay. If not, it indicates that the current node cluster head and the participant nodes corresponding to each target delay do not meet the node cluster construction condition. The minimum value of the time intervals other than the current minimum time interval is selected from the current time interval set. The selected minimum value is determined as the new current minimum value. Thus, a new round of current node cluster construction is performed based on the new current minimum value. By setting the time threshold as the node cluster construction condition, the consistency of the time delay of each participant node in the same node cluster is improved. The influence of the heterogeneity of the participant nodes is further reduced. The communication amount of federated learning is reduced. Thus, the model training convergence speed is improved. The model robustness is improved. The overall federated learning training accuracy is improved.

[0076] It should be noted that the preset time threshold can be set and adjusted according to actual conditions, and the embodiment of the application does not limit this.

[0077] The example of the step S206 is to select the minimum value from the time interval matrix . Then, K-1 minimum values of the time intervals of the participant nodes are selected . If the maximum difference of the K time intervals is less than t, the K participant nodes form a node cluster with the participant node m, where the participant node m is the current cluster head. If the maximum difference of the K time intervals of the participant nodes selected above is greater than t, the second smallest value is selected from the time interval matrix . K-1 minimum values of the time intervals of the participant nodes are selected . . If the maximum difference of the K time intervals is less than t, the K participant nodes form a node cluster with the participant node m, where the participant node m is the current cluster head. If the maximum difference of the K time intervals of the participant nodes selected above is greater than t, the second smallest value is selected from the time interval matrix . .

[0078] S209: Remove the time intervals corresponding to each participant node in the current node cluster and the current cluster head from the current time interval set.

[0079] After the current node cluster is constructed, the time intervals corresponding to each participant node in the current node cluster and the current cluster head are removed from the current time interval set, so that the node cluster division is performed on the participant nodes that have not formed a cluster. By selecting the participant nodes according to the communication delay sorting result, the influence of the heterogeneity of the participant nodes in the node cluster obtained by the division is further reduced.

[0080] The example of the step S208 is that the K+1 participant nodes in the current node cluster are , and the time intervals related to the participant nodes are deleted from the time interval matrix .

[0081] S210: Obtain the number of time intervals currently remaining in the current time interval set.

[0082] After the time intervals corresponding to each participant node in the current node cluster and the current cluster head are removed from the current time interval set, the number of time intervals currently remaining in the current time interval set is obtained, so that the number of time intervals currently remaining in the current time interval set is counted.

[0083] S211: Determine whether the number of time intervals is greater than or equal to the first preset number, if yes, return to step S204, if no, execute step S212.

[0084] After obtaining the number of time intervals currently remaining in the current time interval set, it is determined whether the number of time intervals is greater than or equal to the first preset number, if yes, it returns to step S204 to perform the next round of node cluster division, if no, it means that the number of currently remaining unclustered participant nodes does not meet the number of nodes to form a node cluster, and step S212 is executed.

[0085] S212: When the number of time intervals currently remaining in the current time interval set is not 0, determine the unclustered participant nodes according to each time interval currently remaining in the current time interval set, and adjust the respective calculation delay of each unclustered participant node to join the corresponding node cluster to obtain each node cluster.

[0086] When it is determined that the number of time intervals is less than the first preset number, it means that the number of currently remaining unclustered participant nodes does not meet the number of nodes to form a node cluster, when the number of time intervals currently remaining in the current time interval set is not 0, the unclustered participant nodes are determined according to each time interval currently remaining in the current time interval set, and the respective calculation delay of each unclustered participant node is adjusted, so that the time interval of the unclustered participant node approaches the time interval of the clustered participant node, and then the adjusted each unclustered participant node is added to the corresponding node cluster to obtain each node cluster, further reducing the influence of the heterogeneity of each participant node in the divided node cluster.

[0087] In one specific embodiment of the application, adjusting the respective calculation delay of each unclustered participant node can include the following steps:

[0088] Adjusting the local data set size of each unclustered participant node corresponding to the single local federated learning model training and / or the iteration number of each unclustered participant node corresponding to the single local federated learning model training.

[0089] When adjusting the computing delays of the participant nodes corresponding to each unclustered cluster, the local dataset size of the single local federated learning model training corresponding to each unclustered participant node can be adjusted to adjust the computing delays of the participant nodes corresponding to each unclustered cluster. The number of iterations of the single local federated learning model training corresponding to each unclustered participant node can also be adjusted to adjust the computing delays of the participant nodes corresponding to each unclustered cluster. The local dataset size of the single local federated learning model training and the number of iterations of the single local federated learning model training corresponding to each unclustered participant node can also be adjusted to adjust the computing delays of the participant nodes corresponding to each unclustered cluster. By adjusting one or both of the local dataset size of the single local federated learning model training and the number of iterations of the single local federated learning model training corresponding to each unclustered participant node, the computing delays of the participant nodes corresponding to each unclustered cluster are adjusted, so that the adjusted participant nodes of each unclustered cluster are added to the corresponding node cluster to obtain each node cluster, and the influence of the heterogeneity of the participant nodes in the divided node clusters is further reduced.

[0090] Referring to Figure 3 , Figure 3 A hierarchical federated learning framework based on a trusted execution environment is provided for an embodiment of the present application. The hierarchical federated learning framework includes a central parameter server and participant nodes.

[0091] The central parameter server is located at the top layer of the hierarchical federated learning framework and performs the overall federated learning aggregation process. The adaptive aggregation algorithm is used. The adaptive aggregation algorithm automatically adjusts according to the data volume and the number of iterations of each node cluster. The adaptive aggregation algorithm runs in a trusted execution environment, ensuring that the plaintext of the model update parameters only appears in the trusted execution environment, and the plaintext of the model update cannot be obtained outside, thereby preventing the model update parameters from being reverse-engineered to obtain the participant data and preventing the participant data from being leaked. Each time, the update parameters of the participants are aggregated according to the specified threshold, rather than waiting for all participants to send the update parameters before aggregation.

[0092] The participant nodes are divided into two categories, namely bottom layer participant nodes and intermediate layer participant nodes. The bottom layer participant nodes only perform local federated learning model training, and send model update parameters to the trusted execution environment of the intermediate layer participant nodes in the node cluster, and perform aggregation algorithms. In addition to performing the local federated learning training process, the intermediate layer participant nodes also have the function of aggregating the model update parameters of the nodes in the cluster. The aggregation algorithm runs in the trusted execution environment, ensuring that the plaintext of the model update parameters only appears in the trusted execution environment, and the plaintext of the model update cannot be obtained outside, thereby preventing the model update parameters from being reverse-engineered to obtain the participant node data, and further preventing the participant node data from being leaked. The intermediate layer participant nodes also use adaptive aggregation algorithms. If the intermediate layer participant nodes and the bottom layer participant nodes have high security requirements, the local federated learning training process can also be placed in the trusted execution environment to ensure the safety of the entire federated learning process, including the training process, parameter aggregation process, parameter transmission process, and data storage process. At the same time, the trusted execution environment can be composed of only the trusted execution environment of the CPU, or a combination of the CPU and the Graphics Processing Unit (GPU) to accelerate the execution efficiency of the local federated learning training process. In addition, the intermediate layer participant nodes only aggregate the update parameters of the bottom layer participant nodes according to the specified threshold, rather than waiting for all the bottom layer participant nodes to send the update parameters before aggregating.

[0093] The embodiment of the present application constructs a federated learning framework in layers, divides the layers according to the similarity of the data volume, communication ability, and computing ability of the federated learning participant nodes, forms node clusters, and is divided into at least three layers. The top layer only contains a central parameter server, and the bottom layer only contains federated learning participant nodes and only has the function of local federated learning model training. The intermediate layer is composed of participant nodes in federated learning, which not only perform the function of local federated learning model training, but also perform the function of aggregation. At the same time, the intermediate layer can have multiple layers, at least one layer, so that multiple rounds of model parameter updates can be performed within the cluster before uploading to the upper layer.

[0094] Referring to Figure 4 , Figure 4 A parameter update schematic diagram based on communication delay and computing ability similarity is provided for the embodiment of the present application. One or both of the local data set size of single local federated learning model training and the iteration number of single local federated learning model training of each unclustered participant node are adjusted, such as the iteration number of single local federated learning model training of participant node 1 , and the iteration number of single local federated learning model training of participant node K , and the adjusted participant nodes of each un-clustered cluster are added to the corresponding node cluster.

[0095] S213: A global model is initialized by using the central parameter server, initial model parameters obtained by the initialization are determined as current model parameters, and the current model parameters are distributed to each participant node.

[0096] After the adjusted participant nodes of each un-clustered cluster are added to the corresponding node cluster, a global model is initialized by using the central parameter server, initial model parameters obtained by the initialization are determined as current model parameters, and the current model parameters are distributed to each participant node, so that each participant node contains the current model parameters.

[0097] S214: Each participant node updates a local model according to the current model parameters and a local data set to obtain each local model parameter.

[0098] After the current model parameters are distributed to each participant node, each participant node updates a local model according to the current model parameters and a local data set to obtain each local model parameter.

[0099] S215: Each local model parameter is aggregated in each node cluster to obtain updated intra-cluster local model parameters.

[0100] After obtaining each local model parameter, each local model parameter is aggregated in each node cluster to obtain updated intra-cluster local model parameters, thereby realizing the aggregation of each local model parameter in the node cluster. Since the time delay similarity of each participant node in a single node cluster is high, the aggregation efficiency of the model parameters in the node cluster is improved.

[0101] S216: Each intermediate layer participant node in a node cluster containing the central parameter server is obtained, and each intermediate layer participant node is uploaded to the central parameter server as the target intra-cluster local model parameter corresponding to the node cluster head.

[0102] The intermediate layer participant node can only contain one layer, or can contain multiple layers. After aggregating each local model parameter in each node cluster to obtain updated intra-cluster local model parameters, each intermediate layer participant node in a node cluster containing the central parameter server is obtained, and each intermediate layer participant node is uploaded to the central parameter server as the target intra-cluster local model parameter corresponding to the node cluster head.

[0103] S217: The central parameter server aggregates each target intra-cluster local model parameter to obtain updated global model parameters.

[0104] After uploading the local model parameters of each intermediate layer participant node in each node cluster corresponding to each target cluster to the central parameter server as the node cluster head, the central parameter server is used to aggregate the local model parameters in each target cluster to obtain the updated global model parameters, thereby completing the iteration of the current round.

[0105] In an embodiment of the present application, aggregating the local model parameters in each node cluster can include the following steps:

[0106] The local model parameters in each node cluster are aggregated in the intermediate layer trusted execution environment by the intermediate layer participant node;

[0107] Correspondingly, aggregating the local model parameters in each target cluster by the central parameter server can include the following steps:

[0108] The local model parameters in each target cluster are aggregated in the top layer trusted execution environment by the central parameter server.

[0109] For convenience of description, the above steps can be combined for description.

[0110] When aggregating the model parameters, the local model parameters in each node cluster can be aggregated in the intermediate layer trusted execution environment by the intermediate layer participant node, and the local model parameters in each target cluster can be aggregated in the top layer trusted execution environment by the central parameter server. By aggregating the model parameters in the intermediate layer trusted execution environment and the top layer trusted execution environment, the security of the model parameter aggregation is greatly improved, and the problem of data leakage of the participant node due to leakage of model parameter update information is prevented.

[0111] In an embodiment of the present application, aggregating the local model parameters in each node cluster can include the following steps:

[0112] When it is detected that the second preset number of local model parameters are updated, the second preset number of local model parameters in the corresponding node cluster are aggregated; wherein the second preset number is less than the first preset number;

[0113] Correspondingly, aggregating the local model parameters in each target cluster by the central parameter server can include the following steps:

[0114] When it is detected that the second preset number of local model parameters in the target cluster are uploaded to the central parameter server, the second preset number of local model parameters in the target cluster are aggregated by the central parameter server.

[0115] For convenience of description, the above steps can be combined for description.

[0116] The second preset quantity is less than the first preset quantity, and when it is detected that the second preset quantity of local model parameter updates are completed, the second preset quantity of local model parameters that are updated are aggregated in the corresponding node cluster, and when it is detected that the second preset quantity of target cluster local model parameters are uploaded to the central parameter server, the second preset quantity of target cluster local model parameters are aggregated by the central parameter server. By selecting only the second preset quantity of corresponding model parameters for aggregation when aggregating local model parameters and target cluster local model parameters, the model parameter aggregation efficiency is improved, the federated learning efficiency is improved, the utilization rate of system resources is improved, and the federated learning time consumption is reduced.

[0117] In one specific embodiment of the present application, aggregating each local model parameter in each node cluster can include the following steps:

[0118] According to the data size and the local model iteration number corresponding to each local model parameter, each local model parameter is aggregated in each node cluster.

[0119] Correspondingly, aggregating each target cluster local model parameter by the central parameter server can include the following steps:

[0120] According to the data size and the local model iteration number corresponding to each target cluster local model parameter, each target cluster local model parameter is aggregated by the central parameter server.

[0121] For convenience of description, the above steps can be combined for description.

[0122] According to the data size and the local model iteration number corresponding to each local model parameter, each local model parameter is aggregated in each node cluster, and according to the data size and the local model iteration number corresponding to each target cluster local model parameter, each target cluster local model parameter is aggregated by the central parameter server.

[0123] Taking step S209 as an example, each participant node calculates the gradient using the local data set and uploads it to the upper participant, the gradient of each participant node is , a single node cluster is set to have at most K lower participant nodes for a single intermediate participant node, and when aggregating model parameters, there are K+1 participant nodes including the intermediate participant node, and the threshold value is selected; the central parameter server has at most K lower participant nodes, and when aggregating model parameters, the threshold value The average gradient in the cluster is calculated:

[0124] ;

[0125] wherein, or represents the data participating in the aggregation, the size of which is . represents the model parameter sharing value basis of the participant node. It is assumed that the benchmark value of the iteration number of the local federated learning model training of the participant node once is , , is the iteration number of the single local federated learning model training.

[0126] The model parameter aggregation can be performed according to the data size corresponding to the model parameter and the iteration number of the local model by the following formula:

[0127] ;

[0128] wherein, represents the training number, represents the learning rate, represents the weight vector obtained by the model iteration parameter aggregation for the t-th time.

[0129] By performing the model parameter aggregation according to the data size corresponding to the model parameter and the iteration number of the local model, the aggregation accuracy is ensured.

[0130] S218: The updated global model parameters are distributed to each participant node by the central parameter server.

[0131] After the local model parameters in each target cluster are aggregated by the central parameter server to obtain the updated global model parameters, the updated global model parameters are distributed to each participant node by the central parameter server, so that each participant node obtains the latest global model parameters obtained by aggregation.

[0132] S219: It is judged whether the current model converges and / or reaches the preset iteration round, if yes, step S220 is performed, and if no, step S214 is returned.

[0133] After the updated global model parameters are distributed to each participant node by the central parameter server, it is judged whether the current model converges and / or reaches the preset iteration round, that is, it is judged whether any one or both of the current model converges and reaches the preset iteration round, if yes, it indicates that the model corresponding to the updated global model parameters has met the requirements, step S220 is performed, and if no, it indicates that the model corresponding to the updated global model parameters still does not meet the requirements, step S214 is returned. ​

[0134] S220: determining the model corresponding to the updated global model parameters as the target global model.

[0135] When it is determined that the current model converges and / or reaches the preset iteration round, it indicates that the model corresponding to the updated global model parameters has met the requirements, and the model corresponding to the updated global model parameters is determined as the target global model.

[0136] S221: determining the time consumption of training the target global model from the initialized global model as the target federated learning time consumption.

[0137] After the target global model is determined, the time consumption of training the target global model from the initialized global model is determined as the target federated learning time consumption.

[0138] In one specific embodiment of the present application, after adjusting the computing delay of each un-clustered participant node corresponding to the respective node cluster, so that after the adjusted each un-clustered participant node joins the corresponding node cluster, the method can further include the following steps:

[0139] Step one: count the number of each current layer cluster head obtained by dividing the current layer node cluster;

[0140] Step two: determine whether the number of each current layer cluster head is greater than or equal to the first preset number, if yes, execute step three, if not, execute step five;

[0141] Step three: select the time interval between each current layer cluster head from each time interval, and determine each selected time interval as a new current time interval set;

[0142] Step four: determine each current layer cluster head as each new participant node, and return to execute step S204;

[0143] Step five: obtain a central parameter server, and construct a top layer node cluster by using the central parameter server and each current layer cluster head.

[0144] For convenience of description, the above five steps can be combined for description.

[0145] The number of each current layer node cluster head obtained by the current layer node cluster division is counted, and it is determined whether the number of each current layer node cluster head is greater than or equal to a first preset number. If yes, it is indicated that a new layer of node cluster division can be performed, the time intervals between each current layer node cluster head are selected from each time interval, and each selected time interval is determined as a new current time interval set, each current layer node cluster head is determined as each new participant node, and step S204 is returned to be executed, so that a new layer of node cluster division is performed. If not, it is indicated that a new layer of node cluster division is not needed, a central parameter server is obtained, and a top layer node cluster is constructed by using the central parameter server and each current layer node cluster head. By comparing the number of each current layer node cluster head with the first preset number, it is determined whether a new layer of node cluster division is performed according to the comparison result, and the accuracy of node cluster division is improved.

[0146] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platforms, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment.

[0147] Embodiments of the application also provide a federated learning time consumption prediction device.

[0148] Referring to Figure 5 , Figure 5 A structural block diagram of a federated learning time consumption prediction device provided by an embodiment of the application is provided, and the device can include:

[0149] A calculation delay determination module 51 is configured to determine the calculation delay of the local federated learning model training of each participant node respectively;

[0150] A communication delay determination module 52 is configured to determine the communication delay of the model parameter update process between each participant node;

[0151] A time interval determination module 53 is configured to determine the time interval corresponding to the model parameter aggregation between each participant node according to each calculation delay and each communication delay;

[0152] A node cluster acquisition module 54 is configured to divide the node cluster of each participant node according to each time interval, and obtain each node cluster;

[0153] A time consumption prediction module 55 is configured to predict the federated learning time consumption according to each node cluster.

[0154] By the present application, since the computing delay of the local federated learning model training of each participant node is determined, the communication delay of the model parameter update process between the participant nodes is determined, the node cluster is divided according to the computing delay and the communication delay, the consistency of the time intervals corresponding to the participant nodes of the same layer in the divided node cluster is improved, the participant nodes with similar computing capability and communication delay are divided into the same node cluster for federated learning training, the influence of the heterogeneity of the participant nodes is greatly reduced, and the communication amount of the federated learning is reduced, so that the technical problems of low training efficiency of the federated learning and low time consumption prediction accuracy of the federated learning can be solved, and the technical effects of improving the training efficiency of the federated learning and improving the time consumption prediction accuracy of the federated learning are achieved.

[0155] In an embodiment of the present application, the node cluster obtaining module 54 can include:

[0156] The current minimum value selection submodule is configured to select a current minimum value from the current time interval set composed of the time intervals.

[0157] The current minimum time interval determination submodule is configured to determine the current minimum value as a current minimum time interval.

[0158] The current node cluster head determination submodule is configured to obtain the participant node corresponding to the current minimum time interval, and determine the participant node corresponding to the current minimum time interval as a current node cluster head.

[0159] The sorting result obtaining submodule is configured to select the communication delays of the model parameter update to the current node cluster head from the communication delays, and sort the selected communication delays in size to obtain a sorting result.

[0160] The current node cluster construction submodule is configured to construct a current node cluster according to the sorting result.

[0161] The time interval elimination submodule is configured to eliminate the time intervals corresponding to the participant nodes and the current node cluster head in the current node cluster from the current time interval set, and return to execute the step of selecting the minimum value from the current time interval set composed of the time intervals until the node cluster division of each participant node is completed.

[0162] In an embodiment of the present application, the current node cluster construction submodule can include:

[0163] The target delay determination unit is configured to select the first preset number of communication delays from one end of the communication delays according to the sorting result, and determine the selected first preset number of communication delays as target delays.

[0164] The target time interval determination unit is configured to select a time interval corresponding to a maximum value in the target delays from the time intervals, and determine the selected time interval as the target time interval.

[0165] The first determination unit is configured to determine whether the target time interval is less than or equal to a preset time threshold.

[0166] The current node cluster construction unit is configured to construct a current node cluster according to the current node cluster head and the participant nodes corresponding to the target delays when it is determined that the target time interval is less than or equal to the preset time threshold.

[0167] In an embodiment of the present application, the device can further include:

[0168] The minimum value selection module is configured to select a minimum value from the time intervals other than the current minimum time interval in the current time interval set when it is determined that the target time interval is greater than the preset time threshold.

[0169] The first return execution module is configured to determine the selected minimum value as a new current minimum value, and return to execute the step of determining the current minimum time interval as the current minimum time interval.

[0170] In an embodiment of the present application, the time interval elimination submodule can include:

[0171] The time interval number acquisition unit is configured to acquire a number of time intervals currently remaining in the current time interval set.

[0172] The second determination unit is configured to determine whether the number of time intervals is greater than or equal to a first preset number.

[0173] The first return execution unit is configured to return to execute the step of selecting a minimum value from the current time interval set composed of the time intervals when it is determined that the number of time intervals is greater than or equal to the first preset number.

[0174] The calculation delay adjustment unit is configured to determine the participant nodes not yet in a cluster according to the time intervals currently remaining in the current time interval set when it is determined that the number of time intervals is less than the first preset number and the number of time intervals currently remaining in the current time interval set is not 0, and adjust the calculation delays corresponding to the participant nodes not yet in a cluster respectively, so that the participant nodes not yet in a cluster after adjustment join the corresponding node clusters.

[0175] In an embodiment of the present application, the calculation delay adjustment unit is specifically a unit for adjusting a local data set size of a single local federated learning model training and / or an iteration number of a single local federated learning model training corresponding to the participant nodes not yet in a cluster respectively.

[0176] In an embodiment of the present application, the device can further comprise:

[0177] a quantity counting module configured to adjust the calculation delay of each un-clustered participant node respectively, and count the number of each current layer cluster head obtained by the current layer node cluster division after each un-clustered participant node joins the corresponding node cluster after adjustment;

[0178] a judging module configured to judge whether the number of each current layer cluster head is greater than or equal to a first preset number;

[0179] a new current time interval set determining module configured to, when it is determined that the number of each current layer cluster head is greater than or equal to the first preset number, select the time interval between each current layer cluster head from each time interval, and determine each selected time interval as a new current time interval set;

[0180] a second return execution module configured to determine each current layer cluster head as each new participant node, and return to execute the step of selecting the current minimum value from the current time interval set composed of each time interval;

[0181] a top layer node cluster constructing module configured to, when it is determined that the number of each current layer cluster head is less than the first preset number, acquire a central parameter server, and construct a top layer node cluster by using the central parameter server and each current layer cluster head.

[0182] In an embodiment of the present application, the time-consuming prediction module 55 can comprise:

[0183] a current model parameter issuing sub-module configured to initialize a global model by using a central parameter server, determine the initial model parameter obtained by the initialization as a current model parameter, and issue the current model parameter to each participant node;

[0184] a local model parameter obtaining sub-module configured to respectively obtain each local model parameter by using each participant node to perform local model updating according to the current model parameter and a local data set;

[0185] an updated intra-cluster local model parameter obtaining sub-module configured to aggregate each local model parameter in each node cluster to obtain an updated intra-cluster local model parameter;

[0186] an intra-cluster local model parameter uploading sub-module configured to acquire each intermediate layer participant node in a node cluster containing the central parameter server, and upload each target intra-cluster local model parameter corresponding to each intermediate layer participant node to the central parameter server as a node cluster head;

[0187] The updated global model parameter obtaining submodule is configured to aggregate the local model parameters in each target cluster by using the central parameter server to obtain the updated global model parameters.

[0188] The updated global model parameter issuing submodule is configured to issue the updated global model parameters to each participant node by using the central parameter server.

[0189] The target federated learning time consumption determining submodule is configured to determine the updated global model parameters as new current model parameters, and return to execute the step of performing local model updating by each participant node according to the current model parameters and the local data set until a target global model is trained, and determine the time consumption of training the target global model from the initialized global model as the target federated learning time consumption.

[0190] In an embodiment of the present application, the target federated learning time consumption determining submodule can include:

[0191] The third determining unit is configured to determine whether the current model converges and / or reaches a preset iteration round;

[0192] The target global model determining unit is configured to determine the model corresponding to the updated global model parameters as the target global model when it is determined that the current model converges and / or reaches the preset iteration round.

[0193] The first returning execution unit is configured to return to execute the step of performing local model updating by each participant node according to the current model parameters and the local data set when it is determined that the current model does not converge and does not reach the preset iteration round.

[0194] In an embodiment of the present application, the updated intra-cluster local model parameter obtaining submodule is specifically a module for aggregating each local model parameter in each node cluster by each intermediate layer participant node in the intermediate layer trusted execution environment.

[0195] The updated global model parameter obtaining submodule is specifically a module for aggregating the local model parameters in each target cluster by using the central parameter server in the top layer trusted execution environment.

[0196] In an embodiment of the present application, the updated intra-cluster local model parameter obtaining submodule is specifically a module for aggregating the second preset number of updated local model parameters in the corresponding node cluster when it is detected that the second preset number of local model parameters are updated; wherein the second preset number is less than the first preset number.

[0197] The updated global model parameter obtaining submodule specifically comprises a module for aggregating the second preset number of target intra-cluster local model parameters by using the central parameter server when it is detected that the uploading of the second preset number of target intra-cluster local model parameters to the central parameter server is completed.

[0198] In one specific embodiment of the present application, the updated intra-cluster local model parameter obtaining submodule specifically comprises a module for aggregating each local model parameter in each node cluster according to the data size and the local model iteration number corresponding to each local model parameter;

[0199] The updated global model parameter obtaining submodule specifically comprises a module for aggregating each target intra-cluster local model parameter by using the central parameter server according to the data size and the local model iteration number corresponding to each target intra-cluster local model parameter.

[0200] The description of the features in the embodiments of the federated learning time consumption prediction device can refer to the related description of the embodiments of the federated learning time consumption prediction method, which will not be repeated here.

[0201] The embodiments of the present application also provide an electronic device comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-mentioned federated learning time consumption prediction method embodiments.

[0202] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above-mentioned federated learning time consumption prediction method embodiments when running.

[0203] In one exemplary embodiment, the above-mentioned computer readable storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0204] The embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned federated learning time consumption prediction method embodiments.

[0205] The embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned federated learning time consumption prediction method embodiments.

[0206] Those skilled in the art will further appreciate that the units and algorithms described in connection with the examples disclosed herein can be implemented in electronic hardware, computer software, or both. As described above, the disclosure is directed to each individual feature, hardware and software, and method steps of the various examples. The disclosure is directed to each individual feature or individual algorithm step of each example, and every combination or permutation of the individual features and algorithm steps. Each individual feature or individual algorithm step that is disclosed can be implemented in electronic hardware, computer software, or both. To clearly illustrate this interchangeability of hardware and software, various examples can be described generally in terms of their functionality, without reference to the details of their technical structure. The particular implementation is illustrative for one of ordinary skill in the art to provide a computer program product that implements the various examples. Those skilled in the art will appreciate that the features and steps of the disclosed examples can be implemented in hardware, software, or both.

[0207] The above provides a detailed introduction to the federated learning time-consuming prediction method, device, storage medium and program product provided by the present application. The principles and implementation modes of the present application are described herein by applying specific examples. The above description of the examples is only applicable to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways. These improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A federated learning time consumption prediction method, characterized in that, The method comprises the following steps: determining the computing delay of each participant node corresponding to the local federated learning model training respectively; determining the communication delay of each participant node in the model parameter updating process; determining the time interval corresponding to the model parameter aggregation between each participant node according to the computing delay and the communication delay; dividing the participant nodes into node clusters according to the time intervals to obtain each node cluster; predicting the time consumption of federated learning according to each node cluster; wherein, dividing the participant nodes into node clusters according to the time intervals comprises: selecting the current minimum value from the current time interval set composed of each time interval; determining the current minimum value as the current minimum time interval; obtaining the participant node corresponding to the current minimum time interval, and determining the participant node corresponding to the current minimum time interval as the current node cluster head; selecting each communication delay for model parameter updating to the current node cluster head from each communication delay, and sorting each selected communication delay to obtain a sorting result; constructing a current node cluster according to the sorting result; eliminating each time interval corresponding to each participant node and the current node cluster head in the current node cluster from the current time interval set, and returning to execute the step of selecting the minimum value from the current time interval set composed of each time interval until the node cluster division of each participant node is completed; constructing a current node cluster according to the sorting result, comprising: selecting the first preset number of communication delays from the small end of the communication delay according to the sorting result, and determining the selected first preset number of communication delays as each target delay; selecting the time interval corresponding to the maximum value in each target delay from each time interval, and determining the selected time interval as the target time interval; determining whether the target time interval is less than or equal to a preset time threshold; if yes, constructing a current node cluster according to the participant nodes corresponding to the current node cluster head and each target delay; further comprising: when it is determined that the target time interval is greater than the preset time threshold, selecting the minimum value from other time intervals in the current time interval set except the current minimum time interval; determining the selected minimum value as a new current minimum value, and returning to execute the step of determining the current minimum value as the current minimum time interval; returning to execute the step of selecting the minimum value from the current time interval set composed of each time interval until the node cluster division of each participant node is completed, comprising: obtaining the number of time intervals currently remaining in the current time interval set; determining whether the number of time intervals is greater than or equal to the first preset number; if yes, returning to execute the step of selecting the minimum value from the current time interval set composed of each time interval; If not, when the number of time intervals currently remaining in the current time interval set is not 0, each unclustered participant node is determined according to each time interval currently remaining in the current time interval set, and the computing delay of each unclustered participant node is adjusted respectively, so that each adjusted unclustered participant node joins the corresponding node cluster.

2. The federated learning latency prediction method of claim 1, wherein, The adjusting of the computing delay of each unclustered participant node respectively comprises: The adjusting of the computing delay of each unclustered participant node respectively comprises:

3. The federated learning latency prediction method of claim 1, wherein, The adjusting of the computing delay of each unclustered participant node respectively comprises: After the adjusting of the computing delay of each unclustered participant node respectively, so that each adjusted unclustered participant node joins the corresponding node cluster, the method further comprises: Counting the number of each current layer cluster head obtained by the current layer node cluster division; Determining whether the number of each current layer cluster head is greater than or equal to the first preset number; If yes, selecting the time interval between each current layer cluster head from each time interval, and determining each selected time interval as a new current time interval set; Determining each current layer cluster head as each new participant node, and returning to execute the step of selecting the current minimum value from the current time interval set composed of each time interval; 4. The federated learning time consumption prediction method according to any one of claims 1 to 3, characterized in that, If not, obtaining a central parameter server, and constructing a top layer node cluster by using the central parameter server and each current layer cluster head. The federated learning time consumption prediction according to each node cluster comprises: Initializing a global model by using a central parameter server, and determining the initial model parameter obtained by the initialization as a current model parameter, and distributing the current model parameter to each participant node; Updating a local model by using each participant node according to the current model parameter and a local data set respectively, to obtain each local model parameter; Aggregating each local model parameter in each node cluster to obtain an updated cluster-local model parameter; Obtaining each intermediate layer participant node in the node cluster containing the central parameter server, and uploading each target cluster-local model parameter corresponding to each intermediate layer participant node to the central parameter server as a node cluster head; Aggregating each target cluster-local model parameter by using the central parameter server to obtain an updated global model parameter; Distributing the updated global model parameter to each participant node by using the central parameter server; 5. The federated learning latency prediction method of claim 4, wherein, Determining the updated global model parameter as a new current model parameter, and returning to execute the step of updating a local model by using each participant node according to the current model parameter and a local data set respectively, until a target global model is trained, and the training time consumption of the target global model obtained by initializing the global model is determined as a target federated learning time consumption. Determining the updated global model parameter as a new current model parameter, and returning to execute the step of updating a local model by using each participant node according to the current model parameter and a local data set respectively, until a target global model is trained, comprises: determining whether the current model converges and / or reaches a preset iteration round; if yes, determining the model corresponding to the updated global model parameter as a target global model; if no, returning to performing the step of updating the local model by each participant node according to the current model parameter and the local data set.

6. The federated learning latency prediction method of claim 4, wherein, aggregating each local model parameter in each node cluster, including: aggregating each local model parameter in each node cluster by each intermediate layer participant node in the intermediate layer trusted execution environment; correspondingly, aggregating each target cluster local model parameter by the central parameter server, including: aggregating each target cluster local model parameter by the central parameter server in the top layer trusted execution environment.

7. The federated learning latency prediction method of claim 4, wherein, aggregating each local model parameter in each node cluster, including: when detecting that there are a second preset number of local model parameters updated, aggregating the second preset number of local model parameters in the corresponding node cluster; wherein the second preset number is less than the first preset number; correspondingly, aggregating each target cluster local model parameter by the central parameter server, including: when detecting that there are the second preset number of target cluster local model parameters uploaded to the central parameter server, aggregating the second preset number of target cluster local model parameters by the central parameter server.

8. The federated learning latency prediction method of claim 4, wherein, aggregating each local model parameter in each node cluster, including: aggregating each local model parameter in each node cluster according to the data size and the local model iteration number corresponding to each local model parameter; correspondingly, aggregating each target cluster local model parameter by the central parameter server, including: aggregating each target cluster local model parameter by the central parameter server according to the data size and the local model iteration number corresponding to each target cluster local model parameter.

9. An electronic device, comprising: including: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the federated learning time consumption prediction method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the steps of the federated learning time consumption prediction method according to any one of claims 1 to 8.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the federated learning time consumption prediction method according to any one of claims 1 to 8.

Images