Federal learning time consumption prediction method and device, storage medium and program product

Abstract

The invention discloses a federated learning time consumption prediction method and device, a storage medium and a program product, and relates to the technical field of computers, and the method comprises the steps: determining the calculation delay of local federated learning model training corresponding to each participant node, and determining the communication delay of a model parameter updating process between the participant nodes; the node cluster division is performed according to the calculation delay and the communication delay, so that the consistency of the time intervals corresponding to the participant nodes in the same layer in the node cluster obtained by division is improved, and the participant nodes with similar calculation capability and communication delay are divided into the same node cluster for federated learning training. According to the method, the heterogeneous influence of participant nodes is greatly reduced, the communication traffic of federated learning is reduced, the technical problems of low training efficiency of federated learning and low time consumption prediction accuracy of federated learning are solved, and the technical effects of improving the training efficiency of federated learning and improving the time consumption prediction accuracy of federated learning are achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium, and program product for predicting the time consumption of federated learning. Background Art

[0002] The core idea of ​​federated learning is to distribute the model training process to multiple clients. Each client trains the model on local data and sends the updated model parameters to the central server for aggregation.

[0003] The commonly used aggregation algorithm is the Federal Average Algorithm (Fed Avg). The Fed Avg algorithm can only aggregate model parameters on the central parameter server based on the amount of data from each participant. It does not take into account the heterogeneity of the participants' devices. As a result, during each round of parameter updates, the central parameter server cannot receive model parameter update information from all participants simultaneously or in a timely manner. As a result, the central parameter server cannot execute the Fed Avg algorithm model parameter aggregation process, which in turn leads to low training efficiency of federated learning, low prediction accuracy due to the time consumption of federated learning. Summary of the Invention

[0004] The present invention provides a method, device, storage medium and program product for predicting the time consumption of federated learning, so as to at least solve the problems of low training efficiency of federated learning and low accuracy of predicting the time consumption of federated learning in related technologies.

[0005] The present invention provides a method for predicting the time consumption of federated learning, comprising: Determine the computational delay of the local federated learning model training corresponding to each participant's node; Determine the communication delay between participating nodes in the model parameter update process; Determine the time intervals corresponding to the aggregation of model parameters between the nodes of each participant according to each calculation delay and each communication delay; Divide the nodes of each participant into node clusters according to each time interval to obtain each node cluster; Predict the time consumption of federated learning based on each node cluster.

[0006] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned federated learning time-consuming prediction methods when executing the computer program.

[0007] The present invention also provides a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned federated learning time-consuming prediction methods are implemented.

[0008] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned federated learning time-consuming prediction methods.

[0009] Through the present invention, by determining the computational delay of the local federated learning model training corresponding to each participant node and determining the communication delay of the model parameter update process between each participant node, node clustering is performed according to the computational delay and communication delay, thereby improving the consistency of the time intervals corresponding to the participant nodes of the same layer in the divided node cluster, so that the participant nodes with similar computing power and communication delay are divided into the same node cluster for federated learning training, greatly reducing the heterogeneity effect of the participant nodes and reducing the communication volume of federated learning. Therefore, the technical problems of low training efficiency of federated learning and low accuracy of prediction of federated learning time consumption can be solved, thereby achieving the technical effect of improving the training efficiency of federated learning and improving the accuracy of prediction of federated learning time consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 A flowchart of a method for predicting the time consumption of federated learning provided by an embodiment of the present invention; Figure 2 A flowchart of another method for predicting the time consumption of federated learning provided by an embodiment of the present invention; Figure 3 A diagram of a hierarchical federated learning framework based on a trusted execution environment provided by an embodiment of the present invention; Figure 4 A schematic diagram of parameter updating based on communication delay and computing power similarity provided by an embodiment of the present invention; Figure 5 A structural block diagram of a federated learning time-consuming prediction device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0013] It should be noted that, in the description of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence.

[0014] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0015] An embodiment of the present invention provides a method for predicting the time consumption of federated learning. The method is described in detail in conjunction with the execution process of the method for predicting the time consumption of federated learning.

[0016] See also Figure 1 , Figure 1 A flowchart of an implementation method for predicting the time consumption of federated learning provided in an embodiment of the present invention is provided. The method may include the following steps.

[0017] S101: Determine the computational delay of the local federated learning model training corresponding to each participant node.

[0018] The central parameter server and multiple participant nodes jointly participate in federated learning. Each participant node uses its own locally stored data set to perform local federated learning model training. The computational delay of local federated learning model training corresponding to each participant node is determined. For example, the computational delay of local federated learning model training corresponding to each participant node can be calculated based on the number of iterations of a single local federated learning model training, the number of central processing unit (CPU) cycles required to train 1 bit of data at the participant node, the amount of data involved in a single local federated learning model training update, and the computing power of the device used for a single local federated learning model training at the participant node. This allows for accurate calculation of the computational delay of local federated learning model training corresponding to each participant node.

[0019] Assume that the hierarchical federated learning framework consists of a central parameter aggregation server and N participant nodes, and each participant node constitutes a set , any bottom-level participant node Only with the middle-layer participant nodes Association. The lowest level participant node The local datasets owned are , middle-layer participant nodes The local datasets owned are .

[0020] The calculation formula for the computational delay of the local federated learning model training corresponding to each participant's node is as follows: ; in, For participating nodes The number of iterations of a single local federated learning model training, The number of CPU cycles required to train 1 bit of data on the participating nodes, The amount of data involved in a single local federated learning model training update, For participating nodes The computing power of the device for a single local federated learning model training, For participating nodes The corresponding computational latency of local federated learning model training.

[0021] S102: Determine the communication delay of the model parameter update process between the participating nodes.

[0022] There is a communication delay between participating nodes when updating model parameters. The communication delay during the model parameter update process between participating nodes is determined. For example, the communication delay during the model parameter update process between participating nodes can be calculated based on the size of the model uploaded by the participating nodes and the transmission rate between participating nodes.

[0023] The calculation formula for the communication delay in the model parameter update process between the participating nodes is as follows: ; in, Update the uploaded model size for the model parameters corresponding to participant node i, For participating nodes With the participating nodes The transmission rate between For participating nodes With the participating nodes Communication delay in the model parameter update process.

[0024] S103: Determine the time intervals corresponding to the model parameter aggregation between the nodes of each participant according to each calculation delay and each communication delay.

[0025] After determining the computational delay of the local federated learning model training corresponding to each participating node and the communication delay of the model parameter update process between each participating node, the time interval corresponding to the model parameter aggregation between each participating node is determined based on each computational delay and each communication delay.

[0026] Participant Node Relative participant node The computational delay matrix for local federated learning model training can be expressed as: .

[0027] The communication delay matrix between participating nodes can be expressed as: .

[0028] Participant Node With the participating nodes Time interval matrix for single execution model aggregation for: .

[0029] S104: Divide the nodes of each participant into node clusters according to each time interval to obtain node clusters.

[0030] After determining the time intervals for model parameter aggregation between participating nodes, the participating nodes are divided into clusters based on these time intervals. For example, by grouping participating nodes with similar time intervals into the same cluster, the latency of participating nodes within a single cluster can be highly consistent, thereby improving the training efficiency of federated learning.

[0031] S105: Predict the time consumption of federated learning based on each node cluster.

[0032] After partitioning the node clusters, the federated learning time consumption is predicted for each cluster. By using each partitioned node cluster for federated learning and calculating the time consumption of the federated learning process, the federated learning time consumption prediction is achieved. By grouping participating nodes with similar computing power and communication latency into the same node cluster for federated learning training, the heterogeneity of participating nodes is significantly reduced, the communication volume of federated learning is reduced, the training efficiency of federated learning is improved, and the accuracy of federated learning time consumption prediction is improved.

[0033] Through the present invention, by determining the computational delay of the local federated learning model training corresponding to each participant node and determining the communication delay of the model parameter update process between each participant node, node clustering is performed according to the computational delay and communication delay, thereby improving the consistency of the time intervals corresponding to the participant nodes of the same layer in the divided node cluster, so that the participant nodes with similar computing power and communication delay are divided into the same node cluster for federated learning training, greatly reducing the heterogeneity effect of the participant nodes and reducing the communication volume of federated learning. Therefore, the technical problems of low training efficiency of federated learning and low accuracy of prediction of federated learning time consumption can be solved, thereby achieving the technical effect of improving the training efficiency of federated learning and improving the accuracy of prediction of federated learning time consumption.

[0034] See also Figure 2 , Figure 2 This is a flowchart of another method for predicting the time consumption of federated learning provided in an embodiment of the present invention. The method may include the following steps.

[0035] S201: Determine the computational delay of the local federated learning model training corresponding to each participant node.

[0036] Considering the strong computing power of the central parameter server and the fact that in real-world scenarios the downlink channel bandwidth is often much higher than the uplink channel bandwidth, the server's computational latency and downlink transmission latency are relatively small. Therefore, the computational system latency can be ignored, and only the communication latency during the participant node model parameter update and upload process and the computational latency during the participant node's local federated learning model training process are considered. In this case, the system latency primarily includes the computational latency during the participant node's local federated learning model training process and the communication latency during the participant node model parameter update and upload process.

[0037] S202: Determine the communication delay of the model parameter update process between the participating nodes.

[0038] S203: Determine the time intervals corresponding to the model parameter aggregation between the nodes of each participant according to each calculation delay and each communication delay.

[0039] S204: Select a current minimum value from the current time interval set consisting of the time intervals.

[0040] After determining the time intervals corresponding to the aggregation of model parameters between the participants, a current minimum value is selected from the current time interval set composed of the time intervals.

[0041] Following the example of step S103, assuming that the time interval matrix Select the minimum value .

[0042] S205: Determine the current minimum value as the current minimum time interval.

[0043] After selecting the current minimum value from the current time interval set consisting of the time intervals, the current minimum value is determined as the current minimum time interval.

[0044] S206: Acquire the participant node corresponding to the current minimum time interval, and determine the participant node corresponding to the current minimum time interval as the current node cluster head.

[0045] After determining the current minimum time interval, the participant node corresponding to the current minimum time interval is acquired, and the participant node corresponding to the current minimum time interval is determined as the current node cluster head.

[0046] Following the example of step S204, the minimum value The corresponding participant node m is determined as the current node cluster head.

[0047] S207: Selecting communication delays for updating model parameters to the current node cluster head from the communication delays, and sorting the selected communication delays by size to obtain a sorting result.

[0048] After the current node cluster head is determined, each communication delay for updating the model parameters to the current node cluster head is selected from each communication delay, and the selected communication delays are sorted by size to obtain a sorting result.

[0049] S208: Construct the current node cluster according to the sorting result.

[0050] After sorting the selected communication delays, the current node cluster is constructed based on the sorted results. For example, a certain number of communication delays are selected from the smaller end of the communication delay spectrum, and the corresponding participant nodes are determined. Each participant node is then combined with the current node cluster head to form the current node cluster. By selecting participant nodes based on the communication delay sorting results, the consistency of the communication delays corresponding to the participant nodes selected in the same node cluster is improved, the impact of the heterogeneity of the participant nodes is reduced, and the communication volume of federated learning is reduced.

[0051] In a specific embodiment of the present invention, step S208 may include the following steps: Step 1: selecting a first preset number of communication delays from the end with the smaller communication delay according to the sorting result, and determining the selected first preset number of communication delays as each target delay; Step 2: Select the time interval corresponding to the maximum value of each target delay from each time interval, and determine the selected time interval as the target time interval; Step 3: Determine whether the target time interval is less than or equal to the preset time threshold. If so, proceed to step 4; if not, proceed to step 5. Step 4: Construct the current node cluster based on the current node cluster head and the participant nodes corresponding to each target delay; Step 5: Select the minimum value of other time intervals except the current minimum time interval from the current time interval set; Step 6: Determine the selected minimum value as the new current minimum value, and return to step S205.

[0052] For the convenience of description, the above six steps can be combined for explanation.

[0053] Based on the sorting results, the first preset number of communication delays are selected from the end with the smallest communication delay, and the selected first preset number of communication delays are determined as the target delays. The time interval corresponding to the maximum value of each target delay is selected from each time interval, and the selected time interval is determined as the target time interval. It is determined whether the target time interval is less than or equal to the preset time threshold. If so, it means that the current node cluster head and the participating nodes corresponding to each target delay meet the node cluster construction conditions. The current node cluster is constructed based on the current node cluster head and the participating nodes corresponding to each target delay. If not, it means that the current node cluster head and the participating nodes corresponding to each target delay do not meet the node cluster construction conditions. The minimum value of the time intervals other than the current minimum time interval is selected from the current time interval set, and the selected minimum value is determined as the new current minimum value. A new round of current node cluster construction is then carried out based on the new current minimum value. By setting the time threshold as the node cluster construction condition, the consistency of the delays of each participating node in the same node cluster is improved, the influence of the heterogeneity of the participating nodes is further reduced, and the communication volume of federated learning is reduced, thereby improving the convergence speed of model training, improving the robustness of the model, and improving the overall federated learning training accuracy.

[0054] It should be noted that the preset time threshold can be set and adjusted according to actual conditions, and the embodiment of the present invention does not limit this.

[0055] Following the example of step S206, from the time interval matrix Select the minimum value Then, select K-1 nodes with the minimum value of m , If the maximum difference of the K time intervals is less than t, then the K participant nodes and the participant node m form a node cluster, where the participant node m is the current node cluster head. If the maximum difference of the time intervals of the K participant nodes selected above is greater than t, then select the node from the time interval matrix Select the second smallest value , select K-1 and participants Minimum , If the maximum difference between these K time intervals is less than t, then these K participants and participant Form a node cluster where the participating nodes It is the middle-layer participant node.

[0056] S209: Eliminate each participant node in the current node cluster and the current node cluster head corresponding to each time interval from the current time interval set.

[0057] After the current node cluster is constructed, each participant node and the current node cluster head in the current node cluster are removed from the current time interval set for each corresponding time interval, thereby dividing the remaining unclustered participant nodes into node clusters. By selecting participant nodes for node cluster division based on the communication delay sorting results, the heterogeneity of the participant nodes in the divided node clusters is further reduced.

[0058] Following the example of step S208, it is assumed that the K+1 participant nodes in the current node cluster are constructed as , from the time interval matrix Delete the node with the participant The relevant time interval.

[0059] S210: Obtain the number of time intervals currently remaining in the current time interval set.

[0060] After removing the time intervals corresponding to the participant nodes and the current node cluster head in the current node cluster from the current time interval set, the number of time intervals remaining in the current time interval set is obtained to achieve statistics on the number of time intervals remaining in the current time interval set.

[0061] S211: Determine whether the number of time intervals is greater than or equal to a first preset number. If so, return to step S204; if not, execute step S212.

[0062] After obtaining the number of time intervals currently remaining in the current time interval set, determine whether the number of time intervals is greater than or equal to the first preset number. If so, return to execute step S204 to perform the next round of node cluster division. If not, it means that the number of currently remaining non-clustered participant nodes does not meet the number of nodes that constitute a node cluster, and execute step S212.

[0063] S212: When the number of time intervals currently remaining in the current time interval set is not 0, each non-clustered participant node is determined according to each time interval currently remaining in the current time interval set, and the calculation delay corresponding to each non-clustered participant node is adjusted so that the adjusted non-clustered participant nodes are added to the corresponding node cluster to obtain each node cluster.

[0064] When it is determined that the number of time intervals is less than the first preset number, it means that the number of currently remaining non-clustered participant nodes does not meet the number of nodes that constitute a node cluster. When the number of currently remaining time intervals in the current time interval set is not 0, each non-clustered participant node is determined according to each currently remaining time interval in the current time interval set, and the calculation delay corresponding to each non-clustered participant node is adjusted, so that the time interval corresponding to the non-clustered participant node is closer to the time interval of the clustered participant node, and then the adjusted non-clustered participant nodes are added to the corresponding node cluster to obtain each node cluster, further reducing the heterogeneity effect of each participant node in the divided node cluster.

[0065] In a specific embodiment of the present invention, adjusting the computation delay corresponding to each non-clustered participant node may include the following steps: The local data set size and / or the number of iterations of a single local federated learning model training corresponding to each non-clustered participant node are adjusted.

[0066] When adjusting the computing delay corresponding to each non-clustered participant node, the computing delay corresponding to each non-clustered participant node can be adjusted only by adjusting the local data set size of the single local federated learning model training corresponding to each non-clustered participant node. The computing delay corresponding to each non-clustered participant node can also be adjusted only by adjusting the number of iterations of the single local federated learning model training corresponding to each non-clustered participant node. The computing delay corresponding to each non-clustered participant node can also be adjusted by adjusting both the local data set size and the number of iterations of the single local federated learning model training corresponding to each non-clustered participant node. By adjusting one or both of the local data set size and the number of iterations of a single local federated learning model training corresponding to each non-clustered participant node, the calculation delay corresponding to each non-clustered participant node is adjusted, so that the adjusted non-clustered participant nodes are added to the corresponding node cluster to obtain each node cluster, further reducing the heterogeneity effect of each participant node in the divided node cluster.

[0067] See also Figure 3 , Figure 3 A diagram of a hierarchical federated learning framework based on a trusted execution environment (TEE) provided by an embodiment of the present invention. The hierarchical federated learning framework includes a central parameter server and participant nodes.

[0068] The central parameter server, located at the top level of the hierarchical federated learning framework, executes the overall federated learning aggregation process using an adaptive aggregation algorithm. This algorithm automatically adjusts based on the amount of data and number of iterations corresponding to each node cluster. The adaptive aggregation algorithm runs within a trusted execution environment (TEE), ensuring that the plaintext of model update parameters appears only within the TEE and is inaccessible externally. This prevents the inference of participant data from model update parameters, thereby preventing data leakage. Each time, participant update parameters are aggregated only according to a specified threshold, rather than waiting for all participants to send their updated parameters before aggregating.

[0069] Participant nodes are divided into two categories: bottom-tier and middle-tier nodes. Bottom-tier nodes only perform local federated learning model training and send updated model parameters to the trusted execution environment (TEE) of middle-tier nodes within the node cluster, where they execute the aggregation algorithm. In addition to performing local federated learning training, middle-tier nodes also aggregate model update parameters for nodes within the cluster. Their aggregation algorithm runs within the TEE, ensuring that the plaintext model update parameters remain within the TEE and are inaccessible to external parties. This prevents the reverse inference of participant node data based on model update parameters, thereby preventing data leakage. Middle-tier nodes also utilize an adaptive aggregation algorithm. For middle-tier and bottom-tier nodes with high security requirements, their local federated learning training can also be performed within the TEE, ensuring the security of the entire federated learning process—training, parameter aggregation, parameter transmission, and data storage. The trusted execution environment (TEE) can consist solely of the CPU or a combination of the CPU and a graphics processing unit (GPU), accelerating the efficiency of local federated learning training. Furthermore, intermediate-layer nodes aggregate updated parameters from underlying nodes only according to a specified threshold, rather than waiting for all underlying nodes to send updated parameters before aggregating them.

[0070] The embodiment of the present invention constructs a federated learning framework in a hierarchical manner, and divides the hierarchical structure into nodes according to the similarity of indicators such as the amount of data, communication capabilities, and computing capabilities of the federated learning participant nodes to form a node cluster, which is divided into at least three layers. The top layer only contains the central parameter server, and the bottom layer only contains the federated learning participant nodes, and only has the local federated learning model training function. The middle layer is composed of the participant nodes in the federated learning, which not only performs the local federated learning model training function, but also performs the aggregation function. At the same time, the middle layer can have multiple layers, at least one layer, so that multiple rounds of model parameter updates can be performed within the cluster and then uploaded to the upper layer.

[0071] See also Figure 4 , Figure 4 A schematic diagram of parameter update based on communication delay and computing power similarity provided by an embodiment of the present invention. By adjusting one or both of the local data set size and the number of iterations of a single local federated learning model training corresponding to each non-clustered participant node, for example, the number of iterations of a single local federated learning model training of participant node 1 is , the number of iterations of a single local federated learning model training of participant node K is , which realizes the adjustment of the computing delay corresponding to each non-clustered participant node, so that each non-clustered participant node after adjustment is added to the corresponding node cluster.

[0072] S213: Initialize the global model using the central parameter server, determine the initial model parameters obtained by initialization as the current model parameters, and send the current model parameters to each participating node.

[0073] After adding the adjusted non-clustered participant nodes to the corresponding node clusters, the global model is initialized using the central parameter server, and the initial model parameters obtained by initialization are determined as the current model parameters, and the current model parameters are sent to each participant node, so that each participant node contains the current model parameters.

[0074] S214: Utilize each participant node to update the local model according to the current model parameters and the local data set to obtain local model parameters.

[0075] After the current model parameters are sent to each participant node, each participant node is used to update the local model according to the current model parameters and the local data set to obtain the local model parameters.

[0076] S215: Aggregate each local model parameter in each node cluster to obtain updated local model parameters within the cluster.

[0077] After obtaining the local model parameters, the local model parameters are aggregated within each node cluster to obtain the updated local model parameters within the cluster, thereby realizing the aggregation of the local model parameters within the node cluster. Since the time delay similarity of each participating node in a single node cluster is relatively high, the aggregation efficiency of the model parameters within the node cluster is improved.

[0078] S216: Acquire each intermediate-layer participant node in the node cluster including the central parameter server, and upload each intermediate-layer participant node as a local model parameter in each target cluster corresponding to the node cluster head to the central parameter server.

[0079] It may include only one layer of intermediate layer participant nodes or multiple layers of intermediate layer participant nodes. After aggregating the local model parameters in each node cluster to obtain the updated local model parameters within the cluster, the intermediate layer participant nodes in the node cluster including the central parameter server are obtained, and the local model parameters of each target cluster corresponding to each intermediate layer participant node as the node cluster head are uploaded to the central parameter server.

[0080] S217: Utilize the central parameter server to aggregate the local model parameters in each target cluster to obtain updated global model parameters.

[0081] After uploading the local model parameters of each target cluster corresponding to each intermediate layer participant node as the node cluster head to the central parameter server, the central parameter server is used to aggregate the local model parameters in each target cluster to obtain the updated global model parameters, thereby completing the current round of iteration.

[0082] In a specific embodiment of the present invention, aggregating local model parameters within each node cluster may include the following steps: Utilize each middle-layer participant node to aggregate each local model parameter within each node cluster in each middle-layer trusted execution environment; Accordingly, using the central parameter server to aggregate the local model parameters in each target cluster can include the following steps: A central parameter server is used to aggregate local model parameters within each target cluster in a top-level trusted execution environment.

[0083] For the convenience of description, the above steps can be combined for explanation.

[0084] When performing model parameter aggregation, each intermediate-layer participant node can aggregate local model parameters within each node cluster in each intermediate-layer trusted execution environment, and a central parameter server can aggregate local model parameters within each target cluster in the top-level trusted execution environment. By aggregating model parameters in both the intermediate-layer trusted execution environment and the top-level trusted execution environment, the security of model parameter aggregation is greatly improved, preventing the leakage of participant node data due to the leakage of model parameter update information.

[0085] In a specific embodiment of the present invention, aggregating local model parameters within each node cluster may include the following steps: When it is detected that a second preset number of local model parameters have been updated, aggregating the second preset number of local model parameters that have been updated in the corresponding node cluster; wherein the second preset number is smaller than the first preset number; Accordingly, using the central parameter server to aggregate the local model parameters in each target cluster can include the following steps: When it is detected that a second preset number of local model parameters in the target cluster have been uploaded to the central parameter server, the central parameter server is used to aggregate the second preset number of local model parameters in the target cluster.

[0086] For the convenience of description, the above steps can be combined for explanation.

[0087] A second preset number is pre-set to be smaller than the first preset number. When it is detected that the second preset number of local model parameters have been updated, the second preset number of local model parameters that have been updated are aggregated within the corresponding node cluster. When it is detected that the second preset number of local model parameters within the target cluster have been uploaded to the central parameter server, the second preset number of local model parameters within the target cluster are aggregated using the central parameter server. By selecting only the second preset number of corresponding model parameters for aggregation when aggregating local model parameters and aggregating local model parameters within the target cluster, there is no need to wait until all the first preset number of model parameters have been iterated before aggregating, thereby improving the efficiency of model parameter aggregation, thereby improving the efficiency of federated learning, improving the utilization of system resources, and reducing the time consumption of federated learning.

[0088] In a specific embodiment of the present invention, aggregating local model parameters within each node cluster may include the following steps: Aggregate each local model parameter within each node cluster according to the data size and the number of local model iterations corresponding to each local model parameter; Accordingly, using the central parameter server to aggregate the local model parameters in each target cluster can include the following steps: The central parameter server is used to aggregate the local model parameters in each target cluster according to the data size and the number of local model iterations corresponding to the local model parameters in each target cluster.

[0089] For the convenience of description, the above steps can be combined for explanation.

[0090] The local model parameters are aggregated in each node cluster according to the data size and the number of local model iterations corresponding to each local model parameter. The local model parameters in each target cluster are aggregated using the central parameter server according to the data size and the number of local model iterations corresponding to each local model parameter in each target cluster.

[0091] Following the example of step S209, each participant node calculates the gradient using the local data set and uploads it to its upper-layer participant. The gradient of each participant node is In a single node cluster, a single middle-layer participant node is set to have a maximum of K lower-layer participant nodes. When aggregating model parameters, there are a total of K+1 participant nodes, including the middle-layer participant nodes. The threshold is selected The central parameter server has a maximum of K lower-level participant nodes. When aggregating model parameters, it aggregates the model parameters of K intermediate-level participant nodes and selects the threshold . Compute the average gradient within a cluster: ; in, Or it represents the data involved in the aggregation, whose size is . Indicates the cardinality of model parameter sharing values ​​of participating nodes. Assume that the baseline value of the number of iterations of a local federated learning model training of a participating node is , , The number of iterations for a single local federated learning model training.

[0092] The following formula can be used to aggregate model parameters based on the data size and the number of local model iterations corresponding to the model parameters: ; in, represents the number of training times, represents the learning rate, Indicates the The weight vector obtained by aggregating the model iteration parameters.

[0093] Aggregation accuracy is ensured by aggregating model parameters according to the data size and local model iteration number corresponding to each model parameter.

[0094] S218: Use the central parameter server to send the updated global model parameters to each participating node.

[0095] After using the central parameter server to aggregate the local model parameters in each target cluster to obtain the updated global model parameters, the central parameter server is used to send the updated global model parameters to each participating node, so that each participating node can obtain the latest aggregated global model parameters.

[0096] S219: Determine whether the current model has converged and / or reached the preset iteration rounds. If so, execute step S220; if not, return to execute step S214.

[0097] After using the central parameter server to send the updated global model parameters to each participating node, determine whether the current model converges and / or reaches the preset iteration rounds, that is, determine whether any one or both of the current model convergence and the preset iteration rounds are met. If so, it means that the model corresponding to the updated global model parameters has met the requirements, and execute step S220. If not, it means that the model corresponding to the updated global model parameters still does not meet the requirements, and return to execute step S214.

[0098] S220: Determine the model corresponding to the updated global model parameters as the target global model.

[0099] When it is determined that the current model converges and / or reaches the preset iteration rounds, it means that the model corresponding to the updated global model parameters has met the requirements, and the model corresponding to the updated global model parameters is determined as the target global model.

[0100] S221: The time consumed by training the initialized global model to obtain the target global model is determined as the target federated learning time.

[0101] After the target global model is determined, the time consumed by training the initialized global model to obtain the target global model is determined as the target federated learning time.

[0102] In a specific embodiment of the present invention, after adjusting the computation delays corresponding to the non-clustered participant nodes so that the adjusted non-clustered participant nodes are added to the corresponding node cluster, the method may further include the following steps: Step 1: Count the number of cluster heads of each current layer node obtained by dividing the current layer node cluster; Step 2: Determine whether the number of cluster heads of each current layer node is greater than or equal to a first preset number, if so, execute step 3, if not, execute step 5; Step 3: Select the time intervals between the cluster heads of the nodes in each current layer from the time intervals, and determine the selected time intervals as the new current time interval set; Step 4: Determine the cluster heads of each current layer node as each new participant node, and return to step S204; Step 5: Obtain the central parameter server and use it to build the top-level node cluster with the central parameter server and the cluster heads of each current layer node.

[0103] For the convenience of description, the above five steps can be combined for explanation.

[0104] The number of node cluster heads of each current layer obtained by dividing the node cluster of the current layer is counted, and it is determined whether the number of node cluster heads of each current layer is greater than or equal to a first preset number. If so, it indicates that a new layer of node cluster division can be performed. The time intervals between the node cluster heads of each current layer are selected from each time interval, and each selected time interval is determined as a new current time interval set. Each node cluster head of each current layer is determined as a new participating node, and the process returns to step S204 to perform a new layer of node cluster division. If not, it indicates that a new layer of node cluster division is not required, a central parameter server is obtained, and a top-level node cluster is constructed using the central parameter server and each node cluster head of the current layer. By comparing the number of node cluster heads of each current layer with the first preset number and determining whether to perform a new layer of node cluster division based on the comparison result, the accuracy of the node cluster division is improved.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0106] An embodiment of the present invention also provides a device for predicting the time consumption of federated learning.

[0107] See also Figure 5 , Figure 5 This is a structural block diagram of a federated learning time-consuming prediction device provided in an embodiment of the present invention. The device may include: A computation delay determination module 51 is configured to determine the computation delay of the local federated learning model training corresponding to each participant node; A communication delay determination module 52 is used to determine the communication delay of the model parameter update process between the participating nodes; A time interval determination module 53 is used to determine the time intervals corresponding to the model parameter aggregation between the nodes of each participant according to each calculation delay and each communication delay; A node cluster acquisition module 54 is used to divide each participant's node into node clusters according to each time interval to obtain each node cluster; The time consumption prediction module 55 is used to predict the time consumption of federated learning based on each node cluster.

[0108] Through the present invention, by determining the computational delay of the local federated learning model training corresponding to each participant node and determining the communication delay of the model parameter update process between each participant node, node clustering is performed according to the computational delay and communication delay, thereby improving the consistency of the time intervals corresponding to the participant nodes of the same layer in the divided node cluster, so that the participant nodes with similar computing power and communication delay are divided into the same node cluster for federated learning training, greatly reducing the heterogeneity effect of the participant nodes and reducing the communication volume of federated learning. Therefore, the technical problems of low training efficiency of federated learning and low accuracy of prediction of federated learning time consumption can be solved, thereby achieving the technical effect of improving the training efficiency of federated learning and improving the accuracy of prediction of federated learning time consumption.

[0109] In a specific embodiment of the present invention, the node cluster acquisition module 54 may include: A current minimum value selection submodule is used to select a current minimum value from a current time interval set consisting of various time intervals; A current minimum time interval determination submodule, configured to determine the current minimum value as the current minimum time interval; The current node cluster head determination submodule is used to obtain the participant node corresponding to the current minimum time interval and determine the participant node corresponding to the current minimum time interval as the current node cluster head; The sorting result obtaining submodule is used to select each communication delay for updating the model parameters to the current node cluster head from each communication delay, and sort the selected communication delays by size to obtain the sorting result; The current node cluster construction submodule is used to construct the current node cluster according to the sorting results; The time interval elimination submodule is used to eliminate each participant node and the current node cluster head in the current node cluster from the current time interval set corresponding to each time interval, and return to execute the step of selecting the minimum value from the current time interval set composed of each time interval until the node cluster division of each participant node is completed.

[0110] In a specific embodiment of the present invention, the current node cluster construction submodule may include: a target delay determining unit, configured to select a first preset number of communication delays from the end with the smaller communication delay according to the sorting result, and determine the selected first preset number of communication delays as each target delay; a target time interval determining unit, configured to select a time interval corresponding to a maximum value among the target delays from the time intervals, and determine the selected time interval as the target time interval; A first judging unit, configured to judge whether the target time interval is less than or equal to a preset time threshold; The current node cluster construction unit is used to construct the current node cluster according to the current node cluster head and the participant nodes corresponding to each target delay when it is determined that the target time interval is less than or equal to a preset time threshold.

[0111] In a specific embodiment of the present invention, the device may further include: A minimum value selection module is configured to select the minimum value of other time intervals except the current minimum time interval from the current time interval set when it is determined that the target time interval is greater than a preset time threshold; The first return execution module is used to determine the selected minimum value as the new current minimum value, and return to execute the step of determining the current minimum value as the current minimum time interval.

[0112] In a specific embodiment of the present invention, the time interval elimination submodule may include: A time interval number obtaining unit, used to obtain the number of time intervals currently remaining in the current time interval set; A second judging unit, configured to judge whether the number of time intervals is greater than or equal to a first preset number; a first return execution unit, configured to, when it is determined that the number of time intervals is greater than or equal to a first preset number, return to the step of selecting a minimum value from a current time interval set consisting of the time intervals; A calculation delay adjustment unit is used to determine each non-clustered participant node according to each currently remaining time interval in the current time interval set when it is determined that the number of time intervals is less than a first preset number and the number of time intervals currently remaining in the current time interval set is not 0, and adjust the calculation delay corresponding to each non-clustered participant node so that each adjusted non-clustered participant node is added to the corresponding node cluster.

[0113] In a specific embodiment of the present invention, the calculation delay adjustment unit is specifically a unit that adjusts the local data set size and / or the number of iterations of a single local federated learning model training corresponding to each non-clustered participant node.

[0114] In a specific embodiment of the present invention, the device may further include: A number counting module is used to adjust the calculation delay corresponding to each non-clustered participant node so that after each adjusted non-clustered participant node is added to the corresponding node cluster, the number of cluster heads of each current layer node obtained by dividing the current layer node cluster is counted; A judging module, configured to judge whether the number of cluster heads of nodes in each current layer is greater than or equal to a first preset number; a new current time interval set determination module, configured to select time intervals between cluster heads of nodes in each current layer from the time intervals when it is determined that the number of cluster heads of nodes in each current layer is greater than or equal to a first preset number, and determine the selected time intervals as a new current time interval set; The second return execution module is used to determine the cluster heads of the nodes in each current layer as the new participant nodes, and return to execute the step of selecting the current minimum value from the current time interval set composed of the time intervals; The top-level node cluster construction module is used to obtain a central parameter server when it is determined that the number of node cluster heads of each current layer is less than a first preset number, and to construct a top-level node cluster using the central parameter server and the node cluster heads of each current layer.

[0115] In a specific embodiment of the present invention, the time consumption prediction module 55 may include: The current model parameter sending submodule is used to initialize the global model using the central parameter server, determine the initial model parameters obtained by initialization as the current model parameters, and send the current model parameters to each participating node; The local model parameter acquisition submodule is used to use each participant's node to update the local model according to the current model parameters and local data set to obtain the local model parameters; The updated in-cluster local model parameter acquisition submodule is used to aggregate the local model parameters in each node cluster to obtain the updated in-cluster local model parameters; The cluster local model parameter upload submodule is used to obtain the intermediate layer participant nodes in the node cluster containing the central parameter server, and upload the local model parameters of each target cluster corresponding to each intermediate layer participant node as the node cluster head to the central parameter server; The updated global model parameter acquisition submodule is used to aggregate the local model parameters in each target cluster using the central parameter server to obtain the updated global model parameters; The updated global model parameter distribution submodule is used to distribute the updated global model parameters to each participant node using the central parameter server; The target federated learning time determination submodule is used to determine the updated global model parameters as the new current model parameters, and return to execute the steps of using each participating node to update the local model according to the current model parameters and local data set until the target global model is trained. The statistical time consumption of training the target global model from the initialized global model is determined as the target federated learning time consumption.

[0116] In a specific embodiment of the present invention, the target federated learning time-consuming determination submodule may include: A third judgment unit is used to judge whether the current model has converged and / or reached a preset iteration round; a target global model determining unit, configured to determine the model corresponding to the updated global model parameters as the target global model when it is determined that the current model has converged and / or a preset number of iterations has been reached; The first return execution unit is used to return to the step of updating the local model using each participant node according to the current model parameters and the local data set when it is determined that the current model has not converged and the preset iteration rounds have not been reached.

[0117] In a specific embodiment of the present invention, the updated in-cluster local model parameter obtaining submodule is specifically a module for aggregating each local model parameter within each node cluster using each intermediate layer participant node in each intermediate layer trusted execution environment; The updated global model parameter acquisition submodule is specifically a module that uses the central parameter server to aggregate the local model parameters in each target cluster in the top-level trusted execution environment.

[0118] In a specific embodiment of the present invention, the updated in-cluster local model parameter obtaining submodule is specifically a module that aggregates the second preset number of updated local model parameters in the corresponding node cluster when it is detected that the second preset number of updated local model parameters has been completed; wherein the second preset number is less than the first preset number; The updated global model parameter acquisition submodule is specifically a module that uses the central parameter server to aggregate the second preset number of local model parameters in the target cluster when it is detected that the local model parameters in the second preset number of target clusters have been uploaded to the central parameter server.

[0119] In a specific embodiment of the present invention, the updated in-cluster local model parameter acquisition submodule is specifically a module for aggregating each local model parameter in each node cluster according to the data amount and the number of local model iterations corresponding to each local model parameter; The updated global model parameter acquisition submodule is specifically a module that uses the central parameter server to aggregate the local model parameters in each target cluster according to the data size and the number of local model iterations corresponding to the local model parameters in each target cluster.

[0120] For the description of the features in the embodiment corresponding to the federated learning time consumption prediction device, please refer to the relevant description of the embodiment corresponding to the federated learning time consumption prediction method, and will not be repeated here.

[0121] An embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the federated learning time consumption prediction method.

[0122] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned federated learning time consumption prediction method embodiments when running.

[0123] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0124] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned federated learning time-consuming prediction method embodiments.

[0125] An embodiment of the present invention also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned federated learning time consumption prediction method embodiments.

[0126] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0127] The above describes in detail the method, device, storage medium, and program product for predicting the time consumption of federated learning provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the method and core ideas of the present invention. It should be noted that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for predicting the time consumption of federated learning, characterized in that: include: Determine the computational delay of the local federated learning model training corresponding to each participant's node; Determine the communication delay between participating nodes in the model parameter update process; Determine the time intervals corresponding to the aggregation of model parameters between the nodes of each participant according to each calculation delay and each communication delay; Divide the nodes of each participant into node clusters according to each time interval to obtain each node cluster; Predict the time consumption of federated learning based on each node cluster.

2. The method for predicting the time consumption of federated learning according to claim 1, characterized in that: The nodes of each participant are divided into node clusters according to each time interval, including: Selecting a current minimum value from a current time interval set consisting of each time interval; Determine the current minimum value as the current minimum time interval; Acquire the participant node corresponding to the current minimum time interval, and determine the participant node corresponding to the current minimum time interval as the current node cluster head; Selecting each communication delay for updating the model parameters to the cluster head of the current node from each communication delay, and sorting the selected communication delays by size to obtain a sorting result; Building a current node cluster according to the sorting result; Eliminate each participant node and the current node cluster head in the current node cluster from the current time interval set corresponding to each time interval, and return to the step of selecting the minimum value from the current time interval set composed of each time interval until all participant nodes are divided into node clusters.

3. The method for predicting the time consumption of federated learning according to claim 2, characterized in that: Building a current node cluster according to the sorting result includes: Selecting a first preset number of communication delays from the end with smaller communication delays according to the sorting result, and determining the selected first preset number of communication delays as each target delay; Selecting a time interval corresponding to a maximum value among the target delays from the time intervals, and determining the selected time interval as the target time interval; Determining whether the target time interval is less than or equal to a preset time threshold; If so, a current node cluster is constructed according to the current node cluster head and the participant nodes corresponding to each target delay.

4. The method for predicting the time consumption of federated learning according to claim 3, characterized in that: Also includes: When it is determined that the target time interval is greater than the preset time threshold, selecting a minimum value of other time intervals except the current minimum time interval from the current time interval set; The selected minimum value is determined as the new current minimum value, and the process returns to the step of determining the current minimum value as the current minimum time interval.

5. The method for predicting the time consumption of federated learning according to claim 3, characterized in that: Return to the step of selecting the minimum value from the current time interval set composed of each time interval until all participating nodes are divided into node clusters, including: Obtain the number of time intervals currently remaining in the current time interval set; Determining whether the number of time intervals is greater than or equal to the first preset number; If so, return to the step of selecting the minimum value from the current time interval set consisting of each time interval; If not, when the number of time intervals currently remaining in the current time interval set is not 0, each non-clustered participant node is determined according to each time interval currently remaining in the current time interval set, and the calculation delay corresponding to each non-clustered participant node is adjusted so that the adjusted non-clustered participant node is added to the corresponding node cluster.

6. The method for predicting the time consumption of federated learning according to claim 5, characterized in that: Adjust the computational delays of each non-clustered participant node, including: The local data set size and / or the number of iterations of a single local federated learning model training corresponding to each non-clustered participant node are adjusted.

7. The method for predicting the time consumption of federated learning according to claim 5, characterized in that: After adjusting the computation delays corresponding to the non-clustered participant nodes so that the adjusted non-clustered participant nodes are added to the corresponding node cluster, the method further includes: Count the number of node cluster heads of each current layer obtained by dividing the current layer node cluster; Determining whether the number of cluster heads of each current layer node is greater than or equal to the first preset number; If so, the time intervals between the cluster heads of the nodes in each current layer are selected from the time intervals, and the selected time intervals are determined as the new current time interval set; Determine the cluster heads of the nodes in each current layer as the new participant nodes, and return to execute the step of selecting the current minimum value from the current time interval set composed of the time intervals; If not, a central parameter server is obtained, and a top-level node cluster is constructed using the central parameter server and the cluster heads of each current layer node.

8. The method for predicting the time consumption of federated learning according to any one of claims 1 to 7, characterized in that: Predict the duration of federated learning based on each node cluster, including: Initialize the global model using the central parameter server, determine the initial model parameters obtained by initialization as the current model parameters, and send the current model parameters to each participating node; Using each participant node to update the local model according to the current model parameters and the local data set to obtain local model parameters; Aggregate each local model parameter within each node cluster to obtain the updated local model parameters within the cluster; Obtaining each intermediate-layer participant node in the node cluster including the central parameter server, and uploading each intermediate-layer participant node as a local model parameter in each target cluster corresponding to the node cluster head to the central parameter server; aggregating the local model parameters in each target cluster using the central parameter server to obtain updated global model parameters; Using the central parameter server to send the updated global model parameters to each participant node; The updated global model parameters are determined as the new current model parameters, and the process returns to executing the steps of updating the local model using each participating node according to the current model parameters and the local data set, until the target global model is trained. The time consumed by training the initialized global model to obtain the target global model is determined as the target federated learning time.

9. The method for predicting the time consumption of federated learning according to claim 8, characterized in that: Determining the updated global model parameters as new current model parameters, and returning to execute the step of using each participant node to perform local model update according to the current model parameters and the local data set until the target global model is obtained through training, including: Determine whether the current model has converged and / or reached the preset iteration rounds; If yes, the model corresponding to the updated global model parameters is determined as the target global model; If not, return to the step of using each participant node to update the local model according to the current model parameters and the local data set.

10. The method for predicting time consumption of federated learning according to claim 8, characterized in that: Aggregate each local model parameter within each node cluster, including: Utilize each middle-layer participant node to aggregate each local model parameter within each node cluster in each middle-layer trusted execution environment; Accordingly, the central parameter server is used to aggregate the local model parameters in each target cluster, including: The central parameter server is used to aggregate local model parameters in each target cluster in a top-level trusted execution environment.

11. The method for predicting time consumption of federated learning according to claim 8, characterized in that: Aggregate each local model parameter within each node cluster, including: When it is detected that a second preset number of local model parameters have been updated, aggregating the second preset number of local model parameters that have been updated in the corresponding node cluster; wherein the second preset number is smaller than the first preset number; Accordingly, the central parameter server is used to aggregate the local model parameters in each target cluster, including: When it is detected that the second preset number of local model parameters in the target cluster have been uploaded to the central parameter server, the second preset number of local model parameters in the target cluster are aggregated using the central parameter server.

12. The method for predicting time consumption of federated learning according to claim 8, characterized in that: Aggregate each local model parameter within each node cluster, including: Aggregate each local model parameter within each node cluster according to the data size and the number of local model iterations corresponding to each local model parameter; Accordingly, the central parameter server is used to aggregate the local model parameters in each target cluster, including: The central parameter server is used to aggregate the local model parameters in each target cluster according to the data amount and the number of local model iterations corresponding to the local model parameters in each target cluster.

13. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the method for predicting the time consumption of federated learning as described in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for predicting the time consumption of federated learning according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for predicting the time consumption of federated learning as described in any one of claims 1 to 12 are implemented.

Images