Federated learning platform

Abstract

The application discloses a kind of federal learning platforms, by task preprocessing module, the original data set corresponding to machine learning task is preprocessed to generate the sub federal task corresponding to the machine learning task, and the model corresponding to the machine learning task is obtained;Task running module loads the sub federal task corresponding to machine learning task and corresponding model, so as to complete federal training initialization;Federal training module simulates server and user end according to sub federal task and corresponding model to complete federal training by federated learning;Therefore, traditional machine learning task can be converted into the general task processing paradigm of federal training scene, so that the effect of federated learning on any machine learning task can be quickly verified.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a federated learning platform. Background Technology

[0002] In related technologies, deep learning technology has been widely used in various industries, such as recommendation systems, autonomous driving, and healthcare. However, with the improvement of data privacy protection laws and the increasing awareness of data privacy and security, the traditional data-centralized deep learning model training method may no longer be feasible. In order to address the above privacy challenges, federated learning, as a new paradigm of machine learning that can train models in a distributed manner without data leaving the local machine, has emerged.

[0003] However, transforming traditional centralized training into federated training presents the following major challenges: 1) The federated learning process includes several important stages such as local training, parameter exchange, and model aggregation, and different federated algorithms often adopt different strategies at different stages. However, there is still a lack of a general federated platform that can support different federated algorithms; 2) Different domain applications have huge differences in data characteristics, model architecture, task objectives, and evaluation metrics. Therefore, federating different applications often requires writing a lot of complex and redundant code to adapt to the specificity of the task, which makes it difficult to conduct federated training on datasets from different domains. Summary of the Invention

[0004] This invention aims to at least partially address one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a federated learning platform that can transform traditional machine learning tasks into a general task processing paradigm for federated training scenarios, enabling rapid verification of the effectiveness of federated learning on any machine learning task.

[0005] To achieve the above objectives, this invention proposes a federated learning platform, comprising: a task preprocessing module, used to preprocess the original dataset corresponding to a machine learning task to generate sub-federated tasks corresponding to the machine learning task, and to obtain the model corresponding to the machine learning task; a task execution module, used to load the sub-federated tasks and the corresponding models corresponding to the machine learning task to complete federated training initialization; and a federated training module, used to simulate server-side and client-side federated learning based on the sub-federated tasks and the corresponding models to complete federated training.

[0006] According to the federated learning platform proposed in this invention, the original dataset corresponding to the machine learning task is preprocessed by the task preprocessing module to generate sub-federated tasks corresponding to the machine learning task and to obtain the model corresponding to the machine learning task; the task execution module loads the sub-federated tasks and the corresponding models corresponding to the machine learning task to complete the federated training initialization; the federated training module simulates server and client-side federated learning according to the sub-federated tasks and the corresponding models to complete the federated training; thereby, traditional machine learning tasks can be transformed into a general task processing paradigm for federated training scenarios, enabling the effectiveness of federated learning on any machine learning task to be quickly verified.

[0007] In addition, the federated learning platform proposed according to the present invention may also have the following additional technical features:

[0008] Optionally, the federated learning platform further includes a log recording module, which is used to record intermediate results and corresponding data information during the federated training process; and a result analysis module, which is used to analyze the training results after the federated training is completed.

[0009] Optionally, the task preprocessing module includes: a task generator, which is used to create a root directory for the machine learning task, obtain the original dataset for loading into memory after feature preprocessing, divide the original dataset according to a specified distribution, and record the information used in the division and the division results to obtain the sub-federated tasks corresponding to the machine learning task, and generate names for the sub-federated tasks; and a task pipeline, which is used to store the sub-federated tasks so that the specified sub-federated tasks can be dynamically loaded at runtime.

[0010] Optionally, the task execution module further includes a task calculator, which provides various interface functions for the server and the user.

[0011] Optionally, the federated training module is also used to specify the federated optimization algorithm, sub-federated tasks and hyperparameters used in the federated training, and dynamically load the necessary components required for the federated training to complete the initialization of the federated training environment.

[0012] Optionally, the necessary components for federated training include a dataset, a model, a task calculator, a server, and a client. The server calls the run function to begin federated training of the model in an iterative manner, and the process of each iteration is implemented by the iterate function on the server.

[0013] Optionally, the federated training module is used to simulate server and client-side federated learning based on sub-federated tasks and corresponding models to complete federated training. This includes: in each iteration, the server samples the client-side participating in the training to establish a communication connection; after the server uses the `server.pack` function to package the information to be sent to the specified client-side, it listens for the client-side's reply; upon receiving the reply, it uses `server.unpack` to parse the content sent by the user and integrates it; the client-side listens for sending requests from the server; once a request is received, it responds using the `reply` function and uses the `client.unpack` function to parse the package sent by the server; after performing local computation, it uses the `client.pack` function to package the information and send it to the server; after receiving the package information from the client-side, the server enters the aggregation phase; the `aggregate` function is used for model aggregation, and after obtaining a new model, the next iteration begins until training ends.

[0014] Optionally, the client uses the reply function to determine local computation, which includes parsing the server package, training the model locally, and packaging the training results, and is implemented by the client.pack, client.train, and client.unpack functions, respectively.

[0015] Optionally, the logging module predefines a logger class MyLogger in the federated algorithm file waiting to run, and overrides the instance method log of the MyLogger class. The MyLogger.log method is called once every specified number of iterations, and the MyLogger.write function is used to record the input variable names and values ​​and save them in the MyLogger cache.

[0016] Optionally, the result analysis module obtains the names of all training result records from the log recording module, filters irrelevant training record names according to the obtained analysis conditions, and reads all remaining training record names through the read_records function to restore the corresponding training results for analysis. Attached Figure Description

[0017] Figure 1 A block diagram of a federated learning platform according to an embodiment of the present invention;

[0018] Figure 2 This is a block diagram of a federated learning platform according to an embodiment of the present invention. Detailed Implementation

[0019] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0020] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0021] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0022] refer to Figure 1 As shown, the federated learning platform proposed in this embodiment of the invention includes a task preprocessing module 10, a task execution module 20, and a federated training module 30.

[0023] The task preprocessing module 10 is used to preprocess the original dataset corresponding to the machine learning task to generate the sub-federated tasks corresponding to the machine learning task and to obtain the model corresponding to the machine learning task; the task running module 20 is used to load the sub-federated tasks and the corresponding models corresponding to the machine learning task in order to complete the federated training initialization; the federated training module 30 is used to simulate the server and the client to perform federated learning according to the sub-federated tasks and the corresponding models to complete the federated training.

[0024] As one example, the task preprocessing module includes a task generator and a task pipeline.

[0025] The task generator is used to create a root directory for machine learning tasks, obtain the original dataset for loading into memory after feature preprocessing, divide the original dataset according to a specified distribution, and record the information used in the division and the division results to obtain the sub-federated tasks corresponding to the machine learning tasks, and generate names for the sub-federated tasks; the task pipeline is used to store the sub-federated tasks so that the specified sub-federated tasks can be dynamically loaded at runtime.

[0026] As a specific implementation, the task preprocessing module's specific processing flow is as follows: First, for any given machine learning task, the task generator TaskGen creates a root directory ` / benchmark / task_name` for that task. After creating the task directory, TaskGen downloads the original dataset and performs feature preprocessing, then loads the original dataset into memory. Next, TaskGen partitions the original dataset according to a specified distribution, records the information used in the partitioning and the partitioning results, obtains the sub-federated tasks for that machine learning task, and generates names for these sub-federated tasks. Then, the task pipeline TaskPipe collects the original dataset information and partitioning information from TaskGen. After this, the generated sub-federated tasks are stored in a specified disk location `fedtask / federated_subtask_name` so that the specified sub-federated tasks can be dynamically loaded at runtime. Finally, in the machine learning task's root directory, the network model used by that machine learning task is written so that the federated training system can dynamically load the model for the specified machine learning task.

[0027] It should be noted that for any given federated subtask, the TaskPipe specified by its parent machine learning task dynamically reads the federated subtask into the federated training system and loads the specified model to help the federated training system complete its initialization.

[0028] As an example, the task execution module also includes a task calculator, which provides various interface functions for the server and the client.

[0029] For example, on the server side, TaskCalculator should provide necessary interfaces such as model evaluation interface functions, data loading interface functions, and data device migration interface functions to support real-time evaluation during the federated training process. On the user side, in addition to providing all the interface functions required by the server, TaskCalculator should also provide model training interface functions, forward propagation interface functions, etc., to support local computation on the user side and be able to adapt to different federated optimization algorithms.

[0030] Additionally, it should be noted that after the federated training system is initialized, the federated training of the model begins; the task-specific computation is performed by the TaskCalculator, which decouples the federated optimization algorithm and the machine learning task.

[0031] As an example, the federated training module is also used to specify the federated optimization algorithm, sub-federated tasks and hyperparameters used in federated training, and dynamically load the necessary components required for federated training to complete the initialization of the federated training environment.

[0032] It should be noted that the necessary components for federated training include a dataset, a model, a task calculator, a server, and a client. The server calls the run function to start federated training of the model in an iterative manner, and the process of each iteration is implemented by the iterate function on the server.

[0033] As an example, the federated training module is used to simulate server and client-side federated learning based on sub-federated tasks and corresponding models to complete federated training. This includes: in each iteration, the server samples the client-side participating in the training to establish a communication connection; the server uses the `server.pack` function to package information to be sent to a specified client-side and listens for a response from that client-side; upon receiving a response, the server uses `server.unpack` to parse and integrate the content sent by the user; the client listens for sending requests from the server; upon receiving a request, the client responds using the `reply` function and uses the `client.unpack` function to parse the package sent by the server-side; after performing local computation, the client uses the `client.pack` function to package the information and send it to the server-side; after receiving the package information from the client-side, the server enters the aggregation phase; the `aggregate` function is used to aggregate the model, and after obtaining a new model, the next iteration begins until training ends.

[0034] It should be noted that the client uses the reply function to determine local computation, which includes parsing the server package, training the model locally, and packaging the training results, which are implemented by the client.pack, client.train, and client.unpack functions, respectively.

[0035] In other words, for each iteration, the server first enters the sampling phase, using the function `sample` to sample the users participating in the training for this iteration, obtaining the communication IDs of the selected users. After sampling the users, the server enters the communication phase. The `communicate` function is used to communicate with the selected user IDs. The communication process between the server and each selected user is encapsulated by the function `communicate_with`, which performs the following functions: 1) For the server, after using the `server.pack` function to pack the information to be sent to the specified user ID, it listens for the user's reply; after receiving the reply, it uses `server.unpack` to parse the content sent back by the user and integrate it; 2) For the user, it listens for messages from the server... The system sends a request; upon receiving the request, it responds using the `reply` function and parses the package sent by the server using the `client.unpack` function; after appropriate local computation, it packages the information using the `client.pack` function and finally sends it to the server; the client uses the `reply` function to determine the local computation process, which includes parsing the server package, training the model locally, and packaging the training results, implemented by the `client.pack`, `client.train`, and `client.unpack` functions respectively; after receiving the package information from the user, the server enters the aggregation phase; the `aggregate` function is used to aggregate the model, and after obtaining the new model, the next iteration begins until training is complete.

[0036] It should be noted that the aforementioned federated learning platform can transform traditional machine learning tasks into a general task processing paradigm for federated training scenarios, enabling the effectiveness of federated learning on any machine learning task to be quickly verified; it simulates the main processes of federated distributed training on a single machine, thereby supporting the rapid implementation of different federated algorithms, and retaining the ability to extend to actual physical devices for distributed training.

[0037] In addition, such as Figure 2 As shown, the federated learning platform also includes a log recording module 40 and a result analysis module 50.

[0038] The log recording module 40 is used to record intermediate results and corresponding data information during the federated training process; the result analysis module 50 is used to analyze the training results after the federated training is completed.

[0039] As an example, the logging module 40 predefines a logger class MyLogger in the federated algorithm file waiting to run, and overrides the instance method log of the MyLogger class. The MyLogger.log method is called once every specified number of iterations, and the MyLogger.write function is used to record the input variable names and values ​​and save them in the MyLogger cache.

[0040] As an example, the result analysis module 50 obtains the names of all training result records from the log recording module 40, filters irrelevant training record names according to the obtained analysis conditions, and reads all remaining training record names through the read_records function to restore the corresponding training results so as to analyze the training results.

[0041] In other words, a logger class `MyLogger` is predefined in the federated algorithm file waiting to be run, and the instance method `log` of the `MyLogger` class is overridden to determine the intermediate results and other information that need to be recorded for the experiment. The `MyLogger.log` method is called once every specified number of iterations in `server.run`, so it can be used to output intermediate results in real time for corresponding experimental observations if necessary. The `MyLogger.write` function is used to record the input variable names and values ​​and save them in the `MyLogger` cache. This function can be used to insert code snippets and record arbitrary intermediate results to provide a basis for subsequent analysis. After federated training is completed, the server calls the `MyLogger.save` function to save all experimental results in the cache in the directory `fedtask / federated subtask name / record` of the run federated subtask. The `utils.results_analysis` module is used to analyze experimental results. First, the federated subtask for analyzing the results is specified, and the names of all experimental result records are retrieved from the directory `fedtask / federated subtask name / record`. Then, irrelevant experimental records are filtered out by the filename filter in this module based on user-specified record filtering conditions, such as hyperparameters and algorithm names. Next, the `read_records` function in this module reads all remaining experimental records and reconstructs the experimental results. Finally, the experimental results are customized for analysis based on user-written code.

[0042] It should be noted that the aforementioned federated learning platform can support the simulation of various complex real-world federated application scenarios and provide appropriate experimental recording and analysis tools; by using this platform, researchers can quickly verify and compare the performance of federated algorithms on any machine learning task.

[0043] In summary, the federated learning platform proposed in this invention is highly scalable and easy to use. It enables a standard paradigm for federating traditional machine learning tasks, reducing the complexity of federated training on datasets from different domains. It encapsulates various complex operators at the model level, supporting efficient aggregation, norm calculation, scaling, and other operations on model parameters. It simulates the main processes and default implementations of federated training, and supports the rapid implementation and comparison of various federated algorithms. It can simulate various complex federated application scenarios, such as federated learning scenarios with data heterogeneity, system heterogeneity, or different scales. It provides corresponding experimental log recording and result analysis modules, offering users comprehensive research tools. It retains the ability to migrate to real-world multi-device distributed training scenarios and can be quickly adapted to actual physical devices for federated training.

[0044] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0045] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0046] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0047] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0048] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. The illustrative expressions of the above terms in this specification should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0049] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A federated learning platform, characterized in that, include: The task preprocessing module is used to preprocess the original dataset corresponding to the machine learning task to generate the sub-federated task corresponding to the machine learning task and to obtain the model corresponding to the machine learning task. The task execution module is used to load the sub-federated tasks and corresponding models corresponding to the machine learning task in order to complete the federated training initialization. A federated training module is used to simulate server and user terminals to perform federated learning based on sub-federated tasks and corresponding models to complete federated training. The task preprocessing module includes: A task generator is used to create a root directory for the machine learning task, obtain the original dataset for loading into memory after feature preprocessing, divide the original dataset according to a specified distribution, and record the information used in the division and the division results to obtain the sub-federated tasks corresponding to the machine learning task, and generate names for the sub-federated tasks. A task pipeline is used to store the sub-federation tasks so that specified sub-federation tasks can be dynamically loaded at runtime. The federated training module is also used to specify the federated optimization algorithm, sub-federated tasks and hyperparameters used in the federated training, and to dynamically load the necessary components required for the federated training to complete the initialization of the federated training environment. The necessary components for federated training include a dataset, a model, a task calculator, a server, and a client. The server calls the run function to start federated training of the model in an iterative manner. The process of each iteration is implemented by the server's iterate function. The federated training module is used to simulate server-side and user-side federated learning based on sub-federated tasks and corresponding model simulations to complete federated training, including: During each iteration, the server samples the user terminals participating in the training to establish a communication connection with the user terminals participating in the training. After the server uses the `server.pack` function to package the information to be sent to the specified client, it listens for and waits for the client's reply; upon receiving the reply, it uses `server.unpack` to parse the content sent back by the user and integrate it. The client listens for requests from the server; upon receiving a request, it responds using the `reply` function, parses the package sent by the server using the `client.unpack` function, and after performing local calculations, uses the `client.pack` function to pack the information and send it to the server. After the server receives the package information from the user, it enters the aggregation phase; the aggregate function is used to aggregate the model, and after obtaining the new model, the next round of iteration begins until training is completed.

2. The federated learning platform as described in claim 1, characterized in that, Also includes: A logging module is used to record intermediate results and corresponding data information during the federated training process of the federated training module. The results analysis module is used to analyze the training results after the federated training is completed.

3. The federated learning platform as described in claim 2, characterized in that, The task execution module also includes a task calculator, which provides various interface functions for the server and the user.

4. The federated learning platform as described in claim 3, characterized in that, The client uses the `reply` function to determine local computation, which includes parsing the server package, training the model locally, and packaging the training results, and is implemented by the `client.pack`, `client.train`, and `client.unpack` functions, respectively.

5. The federated learning platform as described in claim 4, characterized in that, The logging module predefines a logger class MyLogger in the federated algorithm file waiting to run, and overrides the instance method log of the MyLogger class. The MyLogger.log method is called once every specified number of iterations, and the MyLogger.write function is used to record the input variable names and values ​​and save them in the MyLogger cache.

6. The federated learning platform as described in claim 5, characterized in that, The result analysis module obtains the names of all training result records from the log recording module, filters irrelevant training record names based on the obtained analysis conditions, and reads all remaining training record names through the read_records function to restore the corresponding training results for analysis.

Images