Multi-task federated learning method and system for large-scale federated learning tasks

Abstract

The application relates to a multi-task federated learning method and system for a large-scale federated learning task. The system comprises a center node, a participant node and a remote storage unit, wherein: the center node comprises a user service layer, a task pool, a global scheduler and a plurality of task execution units; the participant node comprises a task operation interface service, a local scheduling unit and a plurality of computing nodes; and the remote storage unit is an AWS S3, an OSS, a cloud database or the like. The application adopts a global and local scheduler combination and a storage-computation separation architecture to schedule and efficiently execute the federated learning task. The global scheduler makes a decision on the task entering an execution state, balances the fairness and average running time of the task, the local scheduler makes a task admission decision according to local computing resources, and the storage-computation separation structure enables the resources of the system to be more focused on the computing process and is beneficial to independent expansion and contraction of the participant computing nodes.

Description

Technical Field

[0001] This invention relates to a federated learning method and system, and more particularly to a multi-task federated learning method and system for large-scale federated learning tasks. Background Technology

[0002] The continuous development of computer technology and artificial intelligence has led to an explosive growth of data in society, but it has also raised concerns about personal data privacy. Federated learning technology has been proposed and rapidly developed to address data silos and data privacy issues. However, current federated learning frameworks such as PySyft, FATE, and FedML are still largely focused on single-center research-style single-task implementations or optimization for large datasets. How to efficiently execute large-scale federated learning tasks in multi-task and multi-center research scenarios has become a pressing problem. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention proposes a multi-task federated learning method and system for large-scale federated learning tasks.

[0004] In a first aspect, the present invention provides a multi-task federated learning system, the system comprising: a central node, participating nodes, and remote storage, wherein: The central node includes a user service layer, a task pool, a global scheduler, and several task execution units.

[0005] The participating nodes include a task operation interface service, a local scheduler, and several computing nodes.

[0006] The remote storage mentioned refers to cloud storage services such as AWS S3, OSS, and cloud databases.

[0007] The user service layer is used to provide users with gRPC interfaces such as task submission, task status query, task result query, and task runtime data acquisition, and to package the tasks obtained from users and send them to the task pool.

[0008] The task pool receives tasks from the user service layer.

[0009] The global scheduler pulls currently scheduled tasks from the task pool, calculates and sorts the scheduling parameters of the tasks, and schedules the tasks in order according to the calculation results.

[0010] The task execution units receive individual tasks scheduled by the global scheduler and monitor the actual execution process of the tasks, as well as centrally aggregate tasks.

[0011] The operation interface service of the participating node is used to communicate with the task execution unit of the central node and receive the computation tasks of a single participating party.

[0012] The local scheduler of the participating node receives requests to acquire computing resources and makes admission decisions.

[0013] The participating nodes are federated learning devices compatible with different computing frameworks.

[0014] The cloud storage service mentioned above utilizes a remote cloud storage service to store task data and calculation result data files.

[0015] In one implementation, the global scheduler of the central node includes a task scheduling unit and a task assignment unit.

[0016] In one implementation, the task scheduling unit includes a task running parameter estimation unit, a task execution parameter estimation unit, and a comprehensive priority calculation unit.

[0017] In one implementation, the task execution parameter estimation unit calculates the fairness of task execution.

[0018] In one implementation, the task execution parameter estimation unit calculates the estimated time consumption and resource consumption for task execution.

[0019] In one implementation, the comprehensive priority calculation unit calculates the scheduling priority based on the calculation results of the two estimation units mentioned above, and arranges all tasks to be scheduled in descending order of scheduling priority.

[0020] In one implementation, the task assignment unit receives the task arrangement from the comprehensive priority calculation unit and assigns the tasks sequentially to the task execution unit of the central node.

[0021] In one implementation, the multi-task execution unit of the central node initializes the system by starting several independent gRPC processes according to configuration parameters, receiving individual tasks from the task dispatch unit and monitoring their entire execution process, while simultaneously performing aggregate computations in the federated learning process.

[0022] In one implementation, when a participating node's local scheduling unit receives a resource acquisition request from a task execution unit, it decides whether to participate in the task's computation based on its local resource availability and the amount of resources requested.

[0023] In one implementation scheme, participating nodes communicate only with the task execution unit of the central node, which controls the collection, aggregation, and distribution of computational parameters for the current federated learning task.

[0024] In one implementation, both the central node and the participants periodically check the liveness of their local execution units via heartbeat information, and attempt to restart new gRPC processes to supplement the execution units when there are insufficient live execution units.

[0025] In one implementation, the remote storage utilizes object storage cloud services to store the datasets and computation results of each participant, and uses relational database cloud services to store task data.

[0026] Secondly, this invention provides a multi-task federated learning method for large-scale federated learning. This method uses the system for multi-task federated learning of this invention, and the method includes the following steps: 1) The central node service layer receives the task execution parameters submitted by the user, packages the task execution parameters into a Python object for program execution, and puts it into the task pool; 2) The global scheduler accesses the task pool and pulls the set of tasks to be scheduled through mutual exclusion locks. The pulled task set is then sorted in descending order according to the calculation results by the global scheduler's parameter calculation and comprehensive scheduling priority calculation. 3) The task dispatcher assigns the tasks in the set to the task execution units one by one according to the sorting result; 4) After receiving a single task to be run, the task execution unit attempts to obtain computing resources from the participants according to the task's configuration information for the participants; 5) Participants decide whether the resources for the task are easily available through local scheduling units; 6) After the task execution unit obtains a certain percentage of successful resource acquisition results, it starts the task execution process. If the percentage is not reached, the task is marked as failed and the task is put back into the scheduling pool. 7) After the task starts execution, the task execution unit sends the calculation task to the execution units of the participating parties according to the configuration information requirements, obtains the results, performs aggregation, and performs a new round of parameter distribution, etc. 8) The execution unit of the participating party receives the task of the current round, obtains the data through remote scheduling according to the dataset path of the task parameters, and performs calculations according to the specified algorithm; 9) After the task is completed, the calculation results will be stored in a remote storage service via remote call.

[0027] In one implementation, the remote storage call is made via HTTPS.

[0028] The beneficial effects of this invention are: 1. In multi-task federated learning scenarios, the execution efficiency of multi-task federated learning can be significantly improved by parallel execution of multiple task execution units; 2. For large-scale federated learning tasks, the global scheduler of the central node can comprehensively consider the waiting time and computational resources and time consumed by the current tasks to be scheduled, and execute multiple tasks in a more efficient order. 3. This system adopts a storage-compute separation architecture, which eliminates the need for central nodes and participating nodes to allocate additional operating resources for storage services, thereby improving computing efficiency and facilitating the horizontal expansion of computing resources by central nodes and participating nodes, thus increasing the system's computing power. Attached Figure Description

[0029] Figure 1 The example illustrates the overall architecture of the system.

[0030] Figure 2 The example illustrates the central node architecture.

[0031] Figure 3 The example illustrates the process of system creation and initial startup.

[0032] Figure 4 The example illustrates a sample flow of the scheduled execution of a federated learning task. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of protection of the invention.

[0034] Figure 1 This diagram illustrates the overall architecture of a multi-task federated learning system provided in an embodiment of this application. The system includes: a central node, participating node nodes, and remote storage, wherein: The central node includes a user service layer, a task pool, a global scheduler, and several task execution units; The participating nodes include a task operation interface service, a local scheduler, and several computing nodes; The remote storage mentioned above is cloud storage, including AWS S3, OSS, and cloud databases; The user service layer is used to provide users with gRPC interfaces for task submission, task status query, task result query, and task runtime data acquisition, and to package the acquired tasks from users and send them to the task pool. The task pool receives tasks from the user service layer; The global scheduler pulls currently scheduled tasks from the task pool, calculates and sorts the scheduling parameters of the tasks, and schedules the tasks in order according to the calculation results. The task execution unit receives a single task scheduled by the global scheduler and monitors the actual execution process of the task as well as centrally aggregates tasks. The operation interface service of the participating node is used to communicate with the task execution unit of the central node and receive the computing tasks of a single participating party. The local scheduler of the participating node receives requests to acquire computing resources and makes admission decisions; The participating nodes are federated learning devices compatible with different computing frameworks. The cloud storage mentioned above utilizes remote cloud storage services to store task data and calculation result data files.

[0035] Users interact only with the central node, making task operation API calls. Both the central node and participating nodes can access the cluster's remote storage services via HTTPS.

[0036] Figure 2 The example illustrates the central node architecture. The central node includes a user service layer, a task pool, a global scheduler, and several task execution units. The user service layer provides users with gRPC interfaces for task submission, task status query, task result query, and task runtime data retrieval, and packages the tasks obtained from users and sends them to the task pool.

[0037] The task pool receives tasks from the user service layer.

[0038] The global scheduler pulls currently scheduled tasks from the task pool, calculates and sorts the scheduling parameters of the tasks, and schedules the tasks in order according to the calculation results; the global scheduler includes a task scheduling unit and a task assignment unit.

[0039] The task scheduling unit includes a task operation parameter estimation unit, a task execution parameter estimation unit, and a comprehensive priority calculation unit.

[0040] The task execution parameter estimation unit calculates the fairness of task execution; the task execution parameter estimation unit calculates the estimated time consumption and resource consumption of task execution.

[0041] The comprehensive priority calculation unit calculates the scheduling priority based on the calculation results of the two estimation units mentioned above, and arranges all tasks to be scheduled in descending order of scheduling priority; the task assignment unit receives the task arrangement from the comprehensive priority calculation unit and assigns the tasks to the task execution unit of the central node in sequence.

[0042] The task execution unit receives individual tasks scheduled by the global scheduler and monitors the actual execution process of the tasks, as well as centrally aggregates tasks.

[0043] Figure 3 The system startup process is shown, during which system creation and initialization work are performed.

[0044] During initialization, the central node starts the corresponding number of task execution units according to the given cluster parameter configuration during service startup. Each task execution unit is a separate gRPC service process after startup, occupying one socket resource of the central node. The central node stores the socket information of all task execution units locally in an active list and periodically checks the liveness status of task execution units via heartbeat messages. For task execution units that return a dead status or have not returned for a long time, the central node will remove them from the active list and retry starting new task execution units to reach the number of task execution units specified in the cluster configuration.

[0045] During initialization, participants first start a given number of computational entity services according to the specified local computing framework type (Spark, MapReduce, Ray, or no framework) and the number of computational entity services. Each computational entity service is a gRPC service process and occupies independent socket resources. After the computational entity services start, participants start the task operation interface layer service, which is also a gRPC service process. This layer service maintains a list of sockets for all computational entity services and periodically checks the liveness status of the computational entity services via heartbeat information. For computational entity services that return a dead status or have not returned for an extended period, the central node will remove them from the active list and retry starting new computational entity services to reach the number of computational entity services specified in the cluster configuration.

[0046] After both the central node and participating parties are started, the participating parties apply to join the cluster by calling the central node's service interface. The central node records the participating party information of the current cluster and stores this information in a remote relational database via remote call.

[0047] Figure 4 This diagram illustrates an example flow of the federated learning system receiving and executing federated learning tasks. In a multi-task scenario, each incoming federated learning task is scheduled and executed according to this flow, and parallelization of multi-task execution is achieved through multiple task execution units. The specific task scheduling steps for multi-task federated learning execution are as follows: 1. Users can concurrently submit several federated learning tasks by calling the task submission interface of the central node. Each submission will include relevant parameters of the federated learning task, including the federated learning algorithm to be executed, the expected machine learning model algorithm to be trained, the initialization parameters of the algorithm, the identifiers of the participants who can participate in the federated learning task, the dataset paths of the participants, and the minimum number of participants that the task can accept in each round.

[0048] 2. The user service layer of the central node receives the configuration parameters, converts them into Python objects, acquires the mutex lock of the task pool, and then puts the task objects into the task pool of the central node.

[0049] 3. The global scheduler of the central node periodically attempts to acquire the mutex lock of the task pool and pulls a set of tasks to be scheduled from the task pool. The number of tasks in the pulled task set is the smaller of twice the number of idle task execution units of the current central node and the total number of tasks.

[0050] 4. After the global scheduling unit retrieves the tasks, it calculates the running wait score and execution consumption score of each task in the task set through the task running parameter estimation unit and the task execution parameter estimation unit, respectively. Then, it inputs the two scores of the task into the comprehensive priority calculation unit to calculate the scheduling priority score of the task.

[0051] 5. After calculating the scheduling priority scores for all tasks, sort the tasks in the set in descending order of their scheduling priority scores.

[0052] 6. Select tasks in the sorted order that are at most equal to the number of idle task execution units of the current central node, and put the other tasks back into the task pool.

[0053] 7. Assign the retrieved tasks to idle task execution units.

[0054] After task scheduling is completed, each individual task is assigned to a task execution unit and then enters the running state. The execution steps of a single task are as follows: 1. The task execution unit sends computing resource request requests to all participants in the task according to the task configuration parameters, and at the same time sends the task parameters and minimum resource requirements.

[0055] 2. After receiving the request, the local scheduler decides whether to accept the task based on available resources. If the minimum resource requirements cannot be met, the request is rejected; otherwise, success is returned along with the socket information of the computing entity service that the participant is involved in for the task.

[0056] 3. The task execution unit enters the task execution state after receiving more than a predetermined number of success messages from participants; otherwise, it will retry the above process.

[0057] 4. After the task enters the actual execution process, the task execution unit sends the parameters of the computation task to the participating party's computation entity service in each round. The participating party retrieves the dataset used through remote storage calls, performs local computation according to the given algorithm, and returns the computation results. The task execution unit aggregates the results to generate parameters for the next round of computation, and repeats the above computation process until the computation ends. During the computation process, the participating party's computation entity service stores the results of each round of local computation to remote storage through remote calls. After the computation is completed, the computation results of the task execution unit are also stored to remote storage through remote calls.

[0058] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, all of which fall within the protection scope of the present invention. The protection scope of the present invention is defined by the appended claims and any equivalent technical solutions.

Claims

1. A multi-task federated learning system for large-scale federated learning tasks, characterized in that, The system includes: a central node, participating node, and remote storage, wherein: The central node includes a user service layer, a task pool, a global scheduler, and several task execution units; The participating nodes include a task operation interface service, a local scheduler, and several computing nodes; The remote storage mentioned refers to AWS S3, OSS, and cloud databases; The user service layer is used to provide users with gRPC interfaces for task submission, task status query, task result query, and task runtime data acquisition, and to package the acquired tasks from users and send them to the task pool. The task pool receives tasks from the user service layer; The global scheduler pulls currently scheduled tasks from the task pool, calculates and sorts the scheduling parameters of the tasks, and schedules the tasks in order according to the calculation results. The task execution unit receives a single task scheduled by the global scheduler and monitors the actual execution process of the task as well as centrally aggregates tasks. The operation interface service of the participating node is used to communicate with the task execution unit of the central node and receive the computing tasks of a single participating party. The local scheduler of the participating node receives requests to acquire computing resources and makes admission decisions; The participating nodes are federated learning devices compatible with different computing frameworks. The remote storage utilizes remote cloud storage services to store task data and calculation result data files; After task scheduling is completed, each individual task is assigned to a task execution unit and then enters the running state. The execution steps of a single task are as follows: 1) The task execution unit sends computing resource request requests to all participants in the task according to the task configuration parameters, and at the same time sends the task parameters and minimum resource requirements; 2) After receiving the request, the local scheduler decides whether to accept the task based on the availability of idle resources. If the minimum resource requirements cannot be met, the request is rejected; otherwise, success is returned along with the socket information of the computing entity service that the participant is involved in. 3) The task execution unit enters the task execution state after receiving success messages from more than a predetermined number of participants; otherwise, it will retry the above process. 4) After the task enters the actual execution process, the task execution unit sends the parameters of the computing task to the computing entity service of the participants in each round. The participants obtain the dataset used by calling the remote storage, perform local calculations according to the given algorithm and return the calculation results. The task execution unit aggregates the results to generate the parameters for the next round of calculations, and repeats the above calculation process until the calculation ends.

2. The multi-task federated learning system according to claim 1, characterized in that, The global scheduler of the central node includes a task scheduling unit and a task assignment unit. The task scheduling unit includes a task running parameter estimation unit, a task execution parameter estimation unit, and a comprehensive priority calculation unit; The task execution parameter estimation unit calculates the fairness of task execution. The task execution parameter estimation unit calculates the estimated time consumption and resource consumption for task execution; The comprehensive priority calculation unit calculates the scheduling priority based on the calculation results of the two estimation units mentioned above, and arranges all tasks to be scheduled in descending order of scheduling priority. The task assignment unit receives the task arrangement from the comprehensive priority calculation unit and assigns the tasks to the task execution unit of the central node in sequence.

3. The multi-task federated learning system according to claim 1, characterized in that, The task execution unit of the central node consists of several independent gRPC processes that are initialized according to the configuration parameters of the system. It receives individual tasks from the task dispatch unit and monitors their entire execution process, while also performing aggregate computations in the federated learning process.

4. The multi-task federated learning system according to claim 1, characterized in that, When a local scheduling unit of a participating node receives a resource acquisition request from a task execution unit, it decides whether to participate in the calculation of the task based on its local resource situation and the amount of resources requested.

5. The multi-task federated learning system according to claim 1, characterized in that, The participating nodes communicate only with the task execution unit of the central node, which controls the collection, aggregation, and distribution of calculation parameters for the current federated learning task.

6. The multi-task federated learning system according to claim 1, characterized in that, The central node and participating parties will periodically check the liveness status of their local execution units through heartbeat information, and attempt to restart new gRPC processes to supplement the execution units when there are insufficient live execution units.

7. The multi-task federated learning system according to claim 1, characterized in that, The remote storage utilizes object storage cloud services to store the datasets and computation results of each participant, and uses relational database cloud services to store task data.

8. A multi-task federated learning method for large-scale federated learning, characterized in that, The method uses the system according to any one of claims 1-7, and the method includes the following steps: 1) The central node service layer receives the task execution parameters submitted by the user, packages the task execution parameters into a Python object for program execution, and puts it into the task pool; 2) The global scheduler accesses the task pool and pulls the set of tasks to be scheduled through mutual exclusion locks. The pulled task set is then sorted in descending order according to the calculation results by the global scheduler's parameter calculation and comprehensive scheduling priority calculation. 3) The task dispatcher assigns the tasks in the set to the task execution units one by one according to the sorting result; 4) After receiving a single task to be run, the task execution unit attempts to obtain computing resources from the participants according to the task's configuration information for the participants; 5) Participants decide whether computing resources are readily available for the task through local scheduling units; 6) After the task execution unit obtains a certain percentage of successful resource acquisition results, it starts the task execution process. If the percentage is not reached, the task is marked as failed and the task is put back into the scheduling pool. 7) After the task starts execution, the task execution unit sends the calculation task to the execution units of the participating parties according to the configuration information requirements, obtains the results, aggregates them, and performs a new round of parameter distribution. 8) The execution unit of the participating party receives the task of the current round, obtains the data through remote call according to the dataset path of the task parameters, and performs calculations according to the specified algorithm; 9) After the task is completed, the calculation results will be stored in a remote storage service via remote call.

9. The method according to claim 8, characterized in that, The remote call described in step 8) is an HTTPS call.

Images