Lightweight distributed federated learning system and method

Abstract

The invention discloses a lightweight distributed federated learning system and method, and the system comprises a main control end node, a plurality of cooperation end nodes, and a feature processingmodule which is used for enabling a main control end to schedule all cooperation end nodes for the combined feature processing through a feature preprocessing interface; the model training module that is used for the main control end to schedule each cooperative end node to perform model training of federated learning through a model training interface; and the model evaluation module that is used for aggregating the prediction result of each cooperative end node and evaluating the model performance by the main control end through a model evaluation interface. The beneficial effects of the invention are that the invention achieves the quick integration of various types of open source machine learning libraries through the feature processing module, the model training module and the modelevaluation module; no matter whether a federated learning model uses gradient training or not, the framework can be used, the development cycle and development cost are low for different services, rapid landing can be realized, and the data security of each participant can be guaranteed.

Description

technical field [0001] The invention relates to the field of machine learning, in particular to a lightweight distributed federated learning system and method. Background technique [0002] With the development of the big data era, more and more attention is paid to data security, and regulations are constantly improving. Because federated learning technology can guarantee data privacy and security, it is also receiving more and more attention. Federated learning refers to multiple clients performing joint modeling (machine learning or deep learning models), and during the entire learning process, the client's data does not expose local data to other parties, which can ensure data privacy and security. [0003] In the existing federated learning technology, model training is mostly based on gradient value training, that is to say, it will rely on models that can perform gradient training. For models that do not meet gradient training, the federated learning process needs to...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a lightweight distributed federated learning framework and implementation method to solve the problem of re-implementing the bottom layer of the existing federated learning, the inability to reuse a large number of existing open source libraries, the low efficiency of the model customization federated learning that does not meet the gradient training, and the different The problem of low efficiency of business customized federated learning

Images