Federal learning task execution method and device, equipment and medium
By receiving and parsing task descriptions based on protocol templates, it determines and runs the target federated learning framework, which solves the difficulty of task execution caused by differences in different frameworks and achieves more efficient task execution.
Patent Information
- Application Number
- CN202411514859.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-01-24
AI Technical Summary
There are differences between different federated learning frameworks, which leads to business personnel spending a lot of effort on analysis and research when preparing data and converting results, affecting the ease of task execution.
Provide a federal learning task execution method. By receiving task descriptions based on protocol templates, analysis of task descriptions to determine the target federal learning framework, and pull up these frameworks to perform tasks. This method also includes dynamic deployment and monitoring operating status to ensure the smooth implementation of the task.
The differences between different federal learning frameworks have been eliminated, and the easy degree of task execution is improved, so that business personnel can focus on business implementation without having to pay too much attention to the framework differences.
Smart Images
Figure CN119358022A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data security technology, and in particular to a method, device, equipment and medium for executing a federated learning task. Background Art
[0002] Federated learning is a distributed machine learning technology that has emerged in recent years. All participants can jointly train models without centralizing data. This decentralized training model can not only avoid the privacy leakage risks caused by centralized data storage on central servers, but also make better use of decentralized data resources. With the development of big data and machine learning in recent years, federated learning frameworks are also emerging. These frameworks provide a secure computing framework to support the federated AI (Artificial Intelligence) ecosystem, allowing enterprises and institutions to collaborate on data while protecting data security and data privacy, and only need to focus on the implementation of the scenario itself, without paying too much attention to the implementation of the infrastructure. However, these federated learning frameworks are different in many aspects such as storage, communication transmission, resource usage, and security mechanisms. The data input and output format requirements of different frameworks are inconsistent. Business personnel need to analyze and study different frameworks in the steps of preparing data and converting results. This will cause business personnel to spend too much energy on learning and researching the framework instead of focusing on the implementation of the business.
[0003] In summary, how to eliminate the differences between different federated learning frameworks to improve the ease of executing federated learning tasks is a problem to be solved in this field. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for executing a federated learning task, which can eliminate the differences between different federated learning frameworks to improve the ease of executing the federated learning task. The specific scheme is as follows:
[0005] In a first aspect, the present application discloses a method for executing a federated learning task, which is applied to a federated learning management platform, including:
[0006] Receive a task description of a federated learning task to be executed based on the protocol template;
[0007] Parsing the task description to determine each target federated learning framework;
[0008] Launch and run each of the target federated learning frameworks to execute the to-be-executed federated learning tasks.
[0009] Optionally, the federated learning management platform includes container images corresponding to each preset federated learning framework generated based on business logic.
[0010] Optionally, the receiving a task description of a to-be-executed federated learning task arranged based on a protocol template includes:
[0011] Receive a task description of a federated learning task to be executed that is arranged based on a protocol template and a preset data exchange format; wherein the preset data exchange format is any one of a YAML format, an XML format, and a JOSN format.
[0012] Optionally, parsing the task description to determine each target federated learning framework includes:
[0013] Parsing the task description to determine each target federated learning framework and the running order of each target federated learning framework;
[0014] Accordingly, launching and running each of the target federated learning frameworks to execute the to-be-executed federated learning tasks includes:
[0015] Based on the running order, the algorithm components in each of the target federated learning frameworks are pulled up and run to execute the to-be-executed federated learning tasks.
[0016] Optionally, the process of launching and running each of the target federated learning frameworks to execute the to-be-executed federated learning tasks further includes:
[0017] Monitoring the running status of each of the target federated learning frameworks;
[0018] The identity of the target federated learning framework whose running status is ended is added to the to-be-deleted list, so as to clear the target federated learning framework corresponding to the identity stored in the to-be-deleted list.
[0019] Optionally, launching and running each of the target federated learning frameworks includes:
[0020] The current target federated learning framework is pulled up in a dynamic deployment manner, and an identity identifier of the current target federated learning framework is created, and then the current target federated learning framework is run.
[0021] Optionally, the federated learning management platform is an application developed based on a B / S architecture;
[0022] Before receiving the task description of the to-be-executed federated learning task arranged based on the protocol template, the method further includes:
[0023] When there is a federated learning task to be executed, the DNS management function of K8S is called to configure the surrounding environment of the federated learning management platform corresponding to the federated learning task to be executed; wherein the surrounding environment includes a network environment and a storage environment.
[0024] In a second aspect, the present application discloses a federated learning task execution device, which is applied to a federated learning management platform, including:
[0025] A task description receiving module, used to receive a task description of a to-be-executed federated learning task orchestrated based on a protocol template;
[0026] A target framework acquisition module, used to parse the task description to determine each target federated learning framework;
[0027] The learning task execution module is used to pull up and run each of the target federated learning frameworks to execute the federated learning tasks to be executed.
[0028] In a third aspect, the present application discloses an electronic device, comprising:
[0029] Memory, used to store computer programs;
[0030] A processor is used to execute the computer program to implement the steps of the aforementioned disclosed method for executing federated learning tasks.
[0031] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed federated learning task execution method are implemented.
[0032] The beneficial effects of this application are: this application is applied to a federated learning management platform, including: receiving a task description of a to-be-executed federated learning task arranged based on a protocol template; parsing the task description to determine each target federated learning framework; pulling up and running each of the target federated learning frameworks to execute the to-be-executed federated learning task. It can be seen that the task description received by this application is arranged based on a protocol template, that is, when arranging the task description, business personnel do not need to pay attention to the differences between the various target federated learning frameworks, that is, this application can eliminate the differences between different federated learning frameworks to improve the ease of executing federated learning tasks. Furthermore, the task description received by the federated learning management platform is also more standardized, and it is easier to parse out the various target federated learning frameworks required for the to-be-executed federated learning task. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0034] Figure 1 A flowchart of a method for executing a federated learning task disclosed in this application;
[0035] Figure 2 A flowchart of a specific method for executing a federated learning task disclosed in this application;
[0036] Figure 3 A flowchart of another specific method for executing a federated learning task disclosed in this application;
[0037] Figure 4 This is a schematic diagram of the structure of a federated learning task execution device disclosed in this application;
[0038] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0040] Federated learning is a distributed machine learning technology that has emerged in recent years. All participants can jointly train models without centralizing data. This decentralized training model can not only avoid the privacy leakage risks caused by centralized data storage on central servers, but also make better use of decentralized data resources. With the development of big data and machine learning in recent years, federated learning frameworks are also emerging. These frameworks provide a secure computing framework to support the federated AI ecosystem, allowing enterprises and institutions to collaborate on data while protecting data security and data privacy, and only need to focus on the implementation of the scenario itself, without paying too much attention to the implementation of infrastructure. However, these federated learning frameworks are different in many aspects such as storage, communication transmission, resource usage, and security mechanisms. The data input and output format requirements of different frameworks are inconsistent. Business personnel need to analyze and study different frameworks in the steps of preparing data and converting results. This will cause business personnel to spend too much energy on learning and researching the framework instead of focusing on the implementation of the business.
[0041] To this end, this application provides a federated learning task execution solution that can eliminate the differences between different federated learning frameworks to improve the ease of executing federated learning tasks.
[0042] See also Figure 1 As shown, the embodiment of the present application discloses a method for executing a federated learning task, which is applied to a federated learning management platform, including:
[0043] Step S11: Receive a task description of a to-be-executed federated learning task arranged based on a protocol template.
[0044] In this embodiment, the federated learning management platform is an application developed based on the B / S architecture. The federated learning management platform is an application developed based on the B / S architecture (Browser / Server, i.e., browser / server mode). Federated learning tasks can be orchestrated on any node, and then the orchestration results can be distributed through secure connections with other nodes.
[0045] In this embodiment, before receiving the task description of the to-be-executed federated learning task arranged based on the protocol template, it also includes: when there is a to-be-executed federated learning task, calling the DNS management function of K8S to configure the surrounding environment of the federated learning management platform corresponding to the to-be-executed federated learning task; wherein the surrounding environment includes the network environment and the storage environment. Because the federated learning management platform is an application developed based on the B / S architecture, the API (Application Programming Interface) of K8S (Kubernetes, a new distributed architecture solution based on container technology) can be called to manage K8S resources and the management of federated learning services, and the network environment is configured based on the existing DNS (Domain Name System) management function of K8S. After the configuration is completed, it supports network communication and security authentication schemes based on protocols such as https (Hypertext Transfer Protocol Secure).
[0046] This embodiment gives different algorithm framework analyses. Each framework must have steps such as resource allocation, environment deployment, environment startup, and resource release. These steps can be abstractly encapsulated according to the interface design principles and encapsulated according to the requirements of the K8S object model. In this way, the extensible and scalable characteristics of K8S can be used to dynamically manage the environment without excessive manual intervention.
[0047] Any federated learning algorithm in this embodiment has the process of inputting data, computing and processing, and outputting results. These processes can also be encapsulated based on the object model of K8S. In this way, when the business uses it, it only needs to consider how to prepare data and algorithm descriptions based on the encapsulated protocol, without paying attention to the protocol and data format requirements of each framework.
[0048] In this embodiment, the receiving of the task description of the to-be-performed federated learning task arranged based on the protocol template includes: receiving the task description of the to-be-performed federated learning task arranged based on the protocol template and the preset data exchange format; wherein the preset data exchange format is any one of the YAML format, the XML format and the JOSN format. The business personnel arrange the task description based on the protocol template and the preset data exchange format, wherein the preset data exchange format can be any one of the YAML format, the XML format and the JOSN format. Since the resource arrangement of K8S is described in the YAML format, it has the characteristics of clear structure, clear semantics and customizability. Therefore, it is preferred to describe the computing task in the YAML format and arrange the task description of the to-be-performed federated learning task in the YAML format. Further, the federated learning management platform will also provide a framework list so that the business personnel can arrange the task description using the framework list. The task description includes the various target federated learning frameworks required for the to-be-performed federated learning task and the algorithm components in each target federated learning framework. In this way, when the subsequent federated learning management platform parses the task description, it can determine the various target federated learning frameworks and the algorithm components in each target federated learning framework. The task description may be as follows:
[0049] Job:id:l
[0050] Name: Data Verification (namely Data Verification)
[0051] tasks: (i.e. task list)
[0052] -id:tl (that is, the task ID is tl)
[0053] name: Two-way safe intersection (i.e. the task name is Two-way safe intersection)
[0054] framework: tensorflow (i.e. the framework for executing the task is tensorflow)
[0055] image: xxx / psi:1.0 (that is, the image to execute the task is xxx / psi:1.0)
[0056] params: {"method":"EDCH"} (i.e. the task parameter is "method":"EDCH")
[0057] in: (i.e. input data)
[0058] type: csv (that is, the data type is csv)
[0059] from: datal (the data source is datal)
[0060] out: tl_psi (i.e. the task output is tl_psi)
[0061] -id: t2 (that is, the task ID is t2)
[0062] name: Query verification status (that is, the task name is Query verification status)
[0063] framework: federateml (i.e. the framework for executing the task is federateml)
[0064] image: xxx / sir:1.0 (that is, the image to execute the task is xxx / sir:1.0)
[0065] params: {} (i.e. the task parameters are empty)
[0066] in: (i.e. input data)
[0067] type: csv (that is, the data type is csv)
[0068] from: tl (that is, the data source is tl)
[0069] out: t2_sir (that is, the task output is t2_sir)
[0070] Step S12: Parse the task description to determine each target federated learning framework.
[0071] Each task may be completed by combining different algorithms. During implementation, a component list is configured (file description, database storage, etc. can be selected). The list has at least the component name, component version, minimum required resources (memory, CPU), input, output, algorithm image and other attributes. The algorithm image provides a unified external entrance for executing algorithm parameter submission and executing the algorithm inside the image. The entrance is described by the agreed method: https: / / svcname:9010 and requested through the post method. The interface protocol agreement is as follows:
[0072] {
[0073] “job_id”:”submitted task id”,
[0074] “dsl-config”:”Task flow description”,
[0075] "runtime_config":"Task runtime description"
[0076] }
[0077] Among them, runtime_config describes the runtime information (that is, the configuration information of each node):
[0078] {
[0079] "nodes":{
[0080] "alice":{
[0081] "address":"192.168.3.10:9010",
[0082] "listen_addr":"192.168.3.10:9020"
[0083] },
[0084] "bob":{
[0085] "address":"192.168.3.11:9011",
[0086] "listen_addr":"192.168.3.11:9021"
[0087] }
[0088] }
[0089] }
[0090] The protocol describes the nodes involved in the calculation and their respective port information, where address is used to provide the platform with an interface for initiating calculations (that is, https: / / svcname:9010 as agreed above), and listen_addr is the address for exchanging security information when the algorithm is executed internally. If there are multiple addresses, the same rules are used for expansion.
[0091] dsl_config describes the dependencies and input-output relationships between tasks:
[0092] {
[0093] "tasks":[
[0094] {
[0095] "id":"t1",
[0096] "component":"intersect",
[0097] "man":"xxx fed",
[0098] "image":"xx.docker.com / intersect:v1.0.0",
[0099] "resources":{
[0100] "cpu":16,
[0101] "mem":"16Gi"
[0102] },
[0103] "params":{
[0104] "alorightm":"EDCH",
[0105] "cols":"x1,x2"
[0106] },
[0107] "input":{
[0108] "alice":{
[0109] "label":"y",
[0110] "dependencyTask":"t0"
[0111] },
[0112] "bob":{
[0113] "label":"y",
[0114] "dependencyTask":"t0"
[0115] }
[0116] },
[0117] "output":{
[0118] "alice":"dataset.csv",
[0119] "bob":"dataset.csv"
[0120] }
[0121] } ]
[0123] }
[0124] In this protocol, task t1 will read the output file to from task t0 of nodes alice and bob respectively, and generate a dataset.csv file after the task is completed. As for the input / output storage path of t0 and t1, it can be agreed upon according to the specific situation during implementation. For example, the files can be uniformly output to the / opt / attachments / directory, and the directory structure can be organized by job_id.
[0125] In this embodiment, the task description is parsed to determine each target federated learning framework, including: parsing the task description to determine each target federated learning framework and the running order of each target federated learning framework. After receiving the task description, the federated learning management platform first parses the target federated learning framework that needs to be started, for example, analyzing that target federated learning framework A and target federated learning framework B need to be run.
[0126] Step S13: launching and running each of the target federated learning frameworks to execute the to-be-executed federated learning tasks.
[0127] In this embodiment, the pulling up and running of each of the target federated learning frameworks to execute the federated learning tasks to be executed includes: pulling up and running the algorithm components in each of the target federated learning frameworks based on the running order to execute the federated learning tasks to be executed. For example, the federated learning task to be executed needs to run the target federated learning framework A and the target federated learning framework B. The specific steps for executing the federated learning task to be executed are: pulling up the target federated learning framework A, running the target federated learning framework A, that is, importing data into the target federated learning framework A, the target federated learning framework A performs calculations, and exports the calculation results of the target federated learning framework A; pulling up the target federated learning framework B, running the target federated learning framework B, that is, importing the calculation results of the target federated learning framework A into the target federated learning framework B, the target federated learning framework B performs calculations, and exports the final business results to complete the federated learning task to be executed.
[0128] The images encapsulated based on different federated learning frameworks need to configure the listening and communication links according to different frameworks. For example, the federated learning framework developed based on the distributed scheduling framework ray may need to at least expose the port of the ray cluster through the TCP method. The operations of starting ray and opening the communication port are initiated and submitted by the platform through the above-mentioned agreed post https: / / svcname:9010 entry to complete the connection.
[0129] The beneficial effects of this application are: this application is applied to a federated learning management platform, including: receiving a task description of a to-be-executed federated learning task arranged based on a protocol template; parsing the task description to determine each target federated learning framework; pulling up and running each of the target federated learning frameworks to execute the to-be-executed federated learning task. It can be seen that the task description received by this application is arranged based on a protocol template, that is, when arranging the task description, business personnel do not need to pay attention to the differences between the various target federated learning frameworks, that is, this application can eliminate the differences between different federated learning frameworks to improve the ease of executing federated learning tasks. Furthermore, the task description received by the federated learning management platform is also more standardized, and it is easier to parse out the various target federated learning frameworks required for the to-be-executed federated learning task.
[0130] The embodiment of the present application discloses a specific maintenance case loading scheme optimization method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Figure 2 As shown, specifically including:
[0131] Step S21: Receive a task description of a to-be-executed federated learning task arranged based on a protocol template.
[0132] Step S22: Parse the task description to determine each target federated learning framework.
[0133] Among them, the more specific processes of steps S21 and S22 can be found in the aforementioned disclosed embodiments, and will not be described in detail here.
[0134] Step S23: pull up and run each of the target federated learning frameworks to execute the to-be-executed federated learning tasks.
[0135] In this embodiment, the pulling up and running each of the target federated learning frameworks includes: pulling up the current target federated learning framework in a dynamic deployment manner, creating an identity of the current target federated learning framework, and then running the current target federated learning framework. Pull up the current target federated learning framework in a dynamic deployment manner (deployment), create an identity of the current target federated learning framework, and the format of the identity is, for example, jobId_taskId, and then run the current target federated learning framework.
[0136] Step S24: monitor the running status of each of the target federated learning frameworks; add the identity identifier of the target federated learning framework whose running status is ended to the list to be deleted, so as to clean up the target federated learning framework corresponding to the identity identifier stored in the list to be deleted.
[0137] In a specific embodiment, when running each target federated learning framework, it is also necessary to monitor the running status of each target federated learning framework. Because there is a sequence between the target federated learning frameworks, there will be target federated learning frameworks with a running status of ended, and there will be target federated learning frameworks with a running status of running and a target federated learning framework with a non-running status. Among them, the target federated learning framework with a running status of ended needs to be cleaned up in time. For this reason, this embodiment adds the identity of the target federated learning framework with a running status of ended to the list to be deleted. For example, there are 3 target federated learning frameworks corresponding to the federated learning tasks to be executed. The identities of these 3 target federated learning frameworks are jobId_task1, jobId_task2, and jobId_task3. At the current moment, there is an identity jobId_task2 in the list to be deleted, so the target federated learning framework with an identity jobId_task2 is searched from each target federated learning framework and then cleared.
[0138] In another specific embodiment, after completing the pending federated learning task, the federated learning management platform can set the running status of each target federated learning framework corresponding to the pending federated learning task to ended, then add the identity identifier of each target federated learning framework to the list to be deleted, and finally clear out each target federated learning framework corresponding to each identity identifier in the list to be deleted. For example, there are three target federated learning frameworks corresponding to the pending federated learning task, and the identities of these three target federated learning frameworks are jobId_task1, jobId_task2, and jobId_task3, respectively. When the pending federated learning task is completed, the federated learning management platform adds jobId_task1, jobId_task2, and jobId_task3 to the list to be deleted, and then clears out each target federated learning framework corresponding to jobId_task1, jobId_task2, and jobId_task3, respectively.
[0139] It can be seen that this application can dynamically clean up the federated learning framework and maintain the environment of the federated learning management platform. There is no need for business personnel or operation and maintenance personnel to manage and recycle resources after the business operation is completed, thereby reducing the difficulty of operation and maintenance and the waste of resources.
[0140] See also Figure 3 As shown, the embodiment of the present application discloses a method for executing a federated learning task, which is applied to a federated learning management platform, including:
[0141] Step S31: The federated learning management platform receives a task description of a to-be-executed federated learning task arranged based on a protocol template; the federated learning management platform includes container images corresponding to each preset federated learning framework generated based on business logic.
[0142] Cloud-native applications based on K8S can easily implement dynamic deployment of frameworks, dynamic resource management, etc. Therefore, the operating principles of different frameworks can be analyzed to abstract steps such as environment creation, resource allocation, and environment startup, and these steps can be connected to K8S capabilities to form dynamic management capabilities of computing resources; secondly, for the algorithm implementations of different frameworks, although the internal implementations may be different, they all have the INPUT-PROCESS-OUTPUT steps. Similarly, a unified algorithm description can be abstracted based on the K8S object model to abstract these steps. In the description, a unified interface can be provided for the common parts of the algorithm (such as encryption algorithms, etc.) to facilitate business optimization algorithms, and then form dynamic management capabilities of the algorithm.
[0143] In this embodiment, the federated learning management platform includes container (docker) images corresponding to each preset federated learning framework generated based on business logic, so that each federated learning task to be executed can be processed concurrently without affecting each other. It should be noted that when the framework is updated, it only needs to be packaged into a container image to simply upgrade or downgrade without affecting the operation of the overall business.
[0144] It is understandable that when generating container images corresponding to each preset federated learning framework, there is no need to pay attention to the interface method of each preset federated learning framework.
[0145] Step S32: Parse the task description to determine each target federated learning framework.
[0146] Step S33: pull up and run each of the target federated learning frameworks to execute the federated learning tasks to be executed.
[0147] Among them, the more specific processes of steps S32 and S33 are shown in the aforementioned disclosed embodiments, and will not be described in detail here.
[0148] It can be seen that the federated learning management platform of the present application includes container images corresponding to each preset federated learning framework generated based on business logic, which can support concurrent processing of federated learning tasks to be executed. Furthermore, the received task description is arranged based on the protocol template, so when implementing the business, there is no need to analyze the specific use of different frameworks too much, and more energy can be focused on the business. In addition, the present application has strong scalability. When there is a new framework, or even a self-developed framework, it can be simply processed according to the encapsulation strategy of this application and included in unified management and control.
[0149] See also Figure 4 As shown, the embodiment of the present application discloses a federated learning task execution device, which is applied to a federated learning management platform, including:
[0150] A task description receiving module 11 is used to receive a task description of a to-be-executed federated learning task arranged based on a protocol template;
[0151] A target framework acquisition module 12 is used to parse the task description to determine each target federated learning framework;
[0152] The learning task execution module 13 is used to pull up and run each of the target federated learning frameworks to execute the federated learning tasks to be executed.
[0153] The beneficial effects of this application are: this application is applied to the federated learning management platform, including: receiving a task description of a to-be-executed federated learning task arranged based on a protocol template; parsing the task description to determine each target federated learning framework; pulling up and running each of the target federated learning frameworks to execute the to-be-executed federated learning task. It can be seen that the task description received by this application is arranged based on the protocol template, that is, when arranging the task description, the business personnel do not need to pay attention to the differences between the target federated learning frameworks, that is, this application can eliminate the differences between different federated learning frameworks to improve the ease of execution of the federated learning task 3. Furthermore, the task description received by the federated learning management platform is more standardized, and it is easier to parse out the target federated learning frameworks required for the to-be-executed federated learning task.
[0154] In some specific embodiments, the federated learning management platform includes container images corresponding to respective preset federated learning frameworks, which are generated based on business logic.
[0155] In some specific embodiments, the task description receiving module 11 includes:
[0156] A description receiving unit is used to receive a task description of a to-be-executed federated learning task arranged based on a protocol template and a preset data exchange format; wherein the preset data exchange format is any one of a YAML format, an XML format, and a JOSN format.
[0157] In some specific embodiments, the target frame acquisition module 12 includes:
[0158] A parsing unit, configured to parse the task description to determine each target federated learning framework and the running order of each target federated learning framework;
[0159] In some specific embodiments, the learning task execution module 13 includes:
[0160] A federated learning task execution unit is used to pull up and run the algorithm components in each of the target federated learning frameworks based on the running order to execute the federated learning tasks to be executed.
[0161] In some specific embodiments, the federated learning task execution device includes:
[0162] An operation status detection unit, used to monitor the operation status of each of the target federated learning frameworks;
[0163] The learning framework cleaning unit is used to add the identity of the target federated learning framework whose running status is ended to the list to be deleted, so as to clean up the target federated learning framework corresponding to the identity stored in the list to be deleted.
[0164] In some specific embodiments, the learning task execution module 13 includes:
[0165] The learning framework running unit is used to pull up the current target federated learning framework in a dynamic deployment manner, create an identity identifier of the current target federated learning framework, and then run the current target federated learning framework.
[0166] In some specific embodiments, the federated learning management platform is an application developed based on a B / S architecture;
[0167] In some specific embodiments, the federated learning task execution device includes:
[0168] An environment configuration unit is used to call the DNS management function of K8S to configure the surrounding environment of the federated learning management platform corresponding to the federated learning task to be executed when there is a federated learning task to be executed; wherein the surrounding environment includes a network environment and a storage environment.
[0169] Furthermore, an embodiment of the present application also provides an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be considered as any limitation on the scope of use of the present application.
[0170] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the following steps:
[0171] Receive a task description of a federated learning task to be executed based on the protocol template;
[0172] Parsing the task description to determine each target federated learning framework;
[0173] Launch and run each of the target federated learning frameworks to execute the to-be-executed federated learning tasks.
[0174] In some specific implementations, the federated learning management platform includes container images generated based on business logic and corresponding to each preset federated learning framework.
[0175] In some specific implementations, the processor can implement the following steps by executing the computer program stored in the memory:
[0176] Receive a task description of a federated learning task to be executed that is arranged based on a protocol template and a preset data exchange format; wherein the preset data exchange format is any one of a YAML format, an XML format, and a JOSN format.
[0177] In some specific implementations, the processor can implement the following steps by executing the computer program stored in the memory:
[0178] Parsing the task description to determine each target federated learning framework and the running order of each target federated learning framework;
[0179] Accordingly, launching and running each of the target federated learning frameworks to execute the to-be-executed federated learning tasks includes:
[0180] Based on the running order, the algorithm components in each of the target federated learning frameworks are pulled up and run to execute the to-be-executed federated learning tasks.
[0181] In some specific implementations, the processor can implement the following steps by executing the computer program stored in the memory:
[0182] Monitoring the running status of each of the target federated learning frameworks;
[0183] The identity of the target federated learning framework whose running status is ended is added to the to-be-deleted list, so as to clear the target federated learning framework corresponding to the identity stored in the to-be-deleted list.
[0184] In some specific implementations, the processor can implement the following steps by executing the computer program stored in the memory:
[0185] The current target federated learning framework is pulled up in a dynamic deployment manner, and an identity identifier of the current target federated learning framework is created, and then the current target federated learning framework is run.
[0186] In some specific implementations, the federated learning management platform is an application developed based on a B / S architecture; the processor may further include the following steps by executing the computer program stored in the memory:
[0187] Before receiving the task description of the to-be-executed federated learning task arranged based on the protocol template, the method further includes:
[0188] When there is a federated learning task to be executed, the DNS management function of K8S is called to configure the surrounding environment of the federated learning management platform corresponding to the federated learning task to be executed; wherein the surrounding environment includes a network environment and a storage environment.
[0189] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0190] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0191] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.
[0192] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device to realize the operation and processing of the processor 21 on the massive data 223 in the memory 22, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the method for executing the federated learning task performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data transmitted from an external device received by the electronic device, the data 223 can also include data collected by its own input and output interface 25.
[0193] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed method for executing a federated learning task is implemented. The specific steps of the method can be referred to the corresponding contents disclosed in the aforementioned embodiment, and will not be repeated here.
[0194] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0195] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly with hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable EPROM (Erasable Programmable Read Only Memory), electrically erasable programmable EEPROM (Electrically Erasable Programmable read only memory), register, hard disk, removable disk, CD-ROM (CoMP23038003act Disc Read-Only Memory), or any other form of storage medium known in the technical field.
[0196] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0197] The above is a detailed introduction to a method, device, equipment and medium for executing a federated learning task provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for executing a federated learning task, characterized in that: Applied to the federated learning management platform, including: Receive a task description of a federated learning task to be executed based on the protocol template; Parsing the task description to determine each target federated learning framework; Launch and run each of the target federated learning frameworks to execute the to-be-executed federated learning tasks.
2. The method for executing a federated learning task according to claim 1, characterized in that: The federated learning management platform includes container images generated based on business logic and corresponding to each preset federated learning framework.
3. The method for executing a federated learning task according to claim 1, characterized in that: The receiving a task description of a to-be-executed federated learning task arranged based on a protocol template includes: Receive a task description of a federated learning task to be executed that is arranged based on a protocol template and a preset data exchange format; wherein the preset data exchange format is any one of a YAML format, an XML format, and a JOSN format.
4. The method for executing a federated learning task according to claim 1, characterized in that: The task description is parsed to determine each target federated learning framework, including: Parsing the task description to determine each target federated learning framework and the running order of each target federated learning framework; Accordingly, launching and running each of the target federated learning frameworks to execute the to-be-executed federated learning tasks includes: Based on the running order, the algorithm components in each of the target federated learning frameworks are pulled up and run to execute the to-be-executed federated learning tasks.
5. The method for executing a federated learning task according to claim 1, characterized in that: The process of pulling up and running each of the target federated learning frameworks to execute the to-be-executed federated learning tasks also includes: Monitoring the running status of each of the target federated learning frameworks; The identity of the target federated learning framework whose running status is ended is added to the to-be-deleted list, so as to clear the target federated learning framework corresponding to the identity stored in the to-be-deleted list.
6. The method for executing a federated learning task according to claim 5, characterized in that: The launching and running of each of the target federated learning frameworks includes: The current target federated learning framework is pulled up in a dynamic deployment manner, and an identity identifier of the current target federated learning framework is created, and then the current target federated learning framework is run.
7. The method for executing a federated learning task according to any one of claims 1 to 6, characterized in that: The federated learning management platform is an application developed based on the B / S architecture; Before receiving the task description of the to-be-executed federated learning task arranged based on the protocol template, the method further includes: When there is a federated learning task to be executed, the DNS management function of K8S is called to configure the surrounding environment of the federated learning management platform corresponding to the federated learning task to be executed; wherein the surrounding environment includes a network environment and a storage environment.
8. A federated learning task execution device, characterized in that: Applied to the federated learning management platform, including: A task description receiving module, used to receive a task description of a to-be-executed federated learning task orchestrated based on a protocol template; A target framework acquisition module, used to parse the task description to determine each target federated learning framework; The learning task execution module is used to pull up and run each of the target federated learning frameworks to execute the federated learning tasks to be executed.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the method for executing a federated learning task as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the federated learning task execution method as described in any one of claims 1 to 7 are implemented.