Parameter Sharing in Federated Learning

Abstract

One embodiment provides a method for federated learning across multiple data parties, the method including assigning each data party to a corresponding namespace in an object store, assigning a shared namespace in the object store, and triggering a round of federated learning by issuing a customized learning request to at least one data party. Each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace, and upload the local model resulting from the local training to the corresponding namespace in the object store to which the data party is assigned. The method further includes retrieving the local models uploaded to the object store from the object store during a round of federated learning, and aggregating the local models to obtain a shared model.

Description

[Technical field]

[0001] The technical field of embodiments of the present invention relates generally to federated learning in distributed data environments. [Background technology]

[0002] Federated learning involves collaboration among multiple parties to learn / train a predictive machine learning model based on training data that includes multiple data sources owned by the parties (i.e., data parties) without sharing / revealing the training data. The model resulting from the collaboration is more accurate than any model learned in isolation based on only one data source. The trust between the parties may vary depending on the relationship between the parties. Thus, the trust between the parties may be limited. The parties may include, but are not limited to, individuals providing personal data via mobile applications running on their mobile devices (e.g., smartphones, Internet of Things (IoT) devices, etc.), different institutions under the same umbrella company, competitors within the same domain / sector (e.g., cable companies, banks), etc.

[0003] Many challenges emerge with traditional federated learning systems that perform collaborative training of machine learning models. For example, the 2019 Conference on Systems and Machine Learning (SysML '19) included a session targeting scalable aggregators for federated learning that require custom development and maintenance to maintain scalability. As another example, another traditional approach, Open Machine Learning (OpenML), that uses blockchain to store model updates is too slow. As another example, traditional systems that implement collaboration among multiple IoT devices by aggregating data from them do not allow for model versioning (i.e., version control of machine learning models). Furthermore, such traditional systems do not provide a simple way for queries and do not allow for bidirectional data flow of information. Other traditional systems that use message queues such as Kafka® and RabbitMQ® do not allow for model versioning. Thus, the diversity of data parties / devices participating in federated learning requires a simple, robust, and cost-effective solution. Summary of the Invention

[0004] According to one aspect, a method for federated learning across multiple data parties is provided, the method including the steps of assigning each data party to a corresponding namespace in an object store; assigning a shared namespace in the object store; triggering a round of federated learning by issuing a customized learning request to at least one data party, where each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store and upload the local model resulting from the local training to the corresponding namespace in the object store to which the data party is assigned; retrieving from the object store at least one local model uploaded to the object store by the at least one data party during the round of federated learning; and aggregating the at least one local model retrieved from the object store to obtain a shared model.

[0005] According to another aspect, a system for federated learning across multiple data parties is provided, the system including at least one processor and a non-transitory processor-readable memory device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: assigning each data party to a corresponding namespace in the object store; assigning a shared namespace in the object store; and triggering a round of federated learning by issuing a customized learning request to at least one data party, where each customized learning request issued to the data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store and upload the local model resulting from the local training to the corresponding namespace in the object store to which the data party is assigned; reading from the object store at least one local model uploaded to the object store by the at least one data party during the round of federated learning; and aggregating the at least one local model read from the object store to obtain a shared model.

[0006] According to another aspect, a computer program product for federated learning across multiple data parties is provided, where the computer program product includes a computer-readable storage medium having program instructions embodied therein, the program instructions, when executed by a processor, cause the processor to assign each data party to a corresponding namespace in the object store; assign a shared namespace in the object store; trigger a round of federated learning by issuing a customized learning request to at least one data party, where each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store and upload the local model resulting from the local training to the corresponding namespace in the object store to which the data party is assigned; read from the object store at least one local model uploaded to the object store by the at least one data party; and aggregate the at least one local model read from the object store to obtain a shared model.

[0007] FIELD OF THE DISCLOSURE Embodiments of the present invention relate generally to federated learning in a distributed data environment, and more particularly to a method and system for parameter sharing in federated learning.

[0008] One embodiment of the present invention provides a method for federated learning across multiple data parties. The method includes the steps of assigning each data party to a corresponding namespace in an object store, assigning a shared namespace in the object store, and triggering a round of federated learning by issuing a customized learning request to at least one data party. Each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store, and upload the local model resulting from the local training to the corresponding namespace in the object store to which the data party is assigned. The method further includes the steps of retrieving from the object store at least one local model uploaded to the object store by at least one data party during a round of federated learning, and aggregating the at least one local model retrieved from the object store to obtain a shared model. The shared model is uploaded to the shared namespace in the object store. Each data party is notified of the shared models uploaded to the shared namespace in the object store. Another embodiment includes a system for federated learning across multiple data parties and a computer program product for federated learning across multiple data parties. These features contribute to the advantage of providing a scalable and fault-tolerant solution to federated learning that is simple, robust, and cost-effective. These features also contribute to providing a federated learning system that includes data parties and an aggregator, where the system is capable of operating under any type of network connectivity and where the system provides a standardized application programming interface (API) for the data parties and the aggregator.

[0009] One or more of the following features may be included: In some embodiments, for each data party, multiple versions of the uploaded local model are stored in a corresponding namespace in an object store assigned to the data party for multiple rounds of federated learning. In some embodiments, multiple versions of the uploaded shared model are stored in a shared namespace in an object store for multiple rounds of federated learning. These optional features contribute the advantage of retaining historical data from any data party and providing model versioning for backtracking to previous model states.

[0010] These and other aspects, features and advantages of embodiments of the present invention will be understood by reference to the drawings and detailed description herein, and will be realized by the various elements and combinations particularly pointed out in the appended claims. It is to be understood that both the general description above, and the brief description of the drawings and the detailed description of the embodiments of the present invention below, are exemplary and explanatory descriptions of preferred embodiments of the present invention, and are not limitations of the embodiments of the present invention as claimed.

[0011] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings: [Brief description of the drawings]

[0012] [Figure 1] 1 illustrates a cloud computing environment in accordance with an embodiment of the present invention. [Diagram 2] 1 illustrates an abstract model layer according to an embodiment of the present disclosure. [Diagram 3] 1 illustrates an exemplary system that implements a conventional approach to federated learning. [Figure 4] 1 illustrates an exemplary computing architecture for implementing distributed federated learning. [Diagram 5] 1 illustrates an exemplary federated learning system according to an embodiment of the present invention. [Figure 6] 1 is a flowchart of an exemplary process for distributed federated learning across multiple data parties, according to an embodiment of the invention. [Figure 7] 1 is a flowchart of an exemplary process for controlling federated learning across multiple data parties in a distributed manner, according to an embodiment of the present invention. [Figure 8] FIG. 1 is a high-level block diagram illustrating an information processing system useful for implementing embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The detailed description of the invention explains preferred embodiments of the invention, together with advantages and features, by way of example with reference to the drawings.

[0014] SUMMARY OF THE PRESENT EMBODIMENTS The present invention relates generally to federated learning in a distributed data environment, and more particularly to a method and system for parameter sharing in federated learning. One embodiment of the present invention provides a method for federated learning across multiple data parties. The method includes the steps of assigning each data party to a corresponding namespace in an object store, assigning a shared namespace in the object store, and triggering a round of federated learning by issuing a customized learning request to at least one data party. Each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store, and upload the local model resulting from the local training to the corresponding namespace in the object store to which the data party is assigned. The method includes the steps of retrieving from the object store at least one local model uploaded to the object store by at least one data party during a round of federated learning, and aggregating the at least one local model retrieved from the object store to obtain a shared model.

[0015] Another embodiment of the present invention provides a system for federated learning across multiple data parties. The system includes at least one processor and a non-transitory processor-readable memory device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include assigning each data party to a corresponding namespace in the object store, assigning a shared namespace in the object store, and triggering a round of federated learning by issuing a customized learning request to at least one data party. Each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store, and upload the local model resulting from the local training to the corresponding namespace in the object store to which the data party is assigned. The operations include retrieving from the object store at least one local model uploaded to the object store by at least one data party during a round of federated learning, and aggregating the at least one local model retrieved from the object store to obtain a shared model.

[0016] One embodiment of the present invention provides a computer program product for federated learning across multiple data parties. The computer program product includes a computer-readable storage medium having program instructions embodied therein, which, when executed by a processor, cause the processor to assign each data party to a corresponding namespace in an object store, assign a shared namespace in the object store, and trigger a round of federated learning by issuing a customized learning request to at least one data party. Each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store, and to upload the local model resulting from the local training to the corresponding namespace in the object store to which the data party is assigned. The program instructions, when executed by the processor, further cause the processor to read, from the object store, at least one local model uploaded to the object store by at least one data party during a round of federated learning, and aggregate the at least one local model read from the object store to obtain a shared model.

[0017] Embodiments of the present invention provide a simple, robust, cost-effective, scalable and fault-tolerant solution for federated learning. One embodiment provides a federated learning system including data parties and aggregators, where the system is capable of operating under any type of network connectivity, can retain historical data from any data party, provides model versioning for backtracking to previous model states, and provides standardized application programming interfaces (APIs) for the data parties and aggregators.

[0018] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings detailed herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or hereafter developed.

[0019] Cloud computing is a model of service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. In one embodiment, this cloud model includes at least five characteristics, at least three service models, and at least four deployment models.

[0020] The characteristics are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, as they need it automatically, without the need for human interaction with the service provider. Broadband Network Access: Capabilities are available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, personal digital assistants). Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided to them, but there is a sense of location independence in the sense that they can specify location at a higher level of abstraction (e.g. country, state, or data center). Rapid Elasticity: Capacity can be rapidly and elastically, in some cases automatically, provisioned to quickly scale out and rapidly released to quickly scale in. To the consumer, the provisioned available capacity often appears externally unlimited, and any amount can be purchased at any time. Measured Services: Cloud systems automatically control and optimize resource usage by utilizing metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, number of active users). Resource usage is monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0021] The service model is as follows: Software as a Service (SaaS): The capability offered to the consumer is to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities with the potential exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The capability offered to a consumer is to deploy consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and possibly the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The capability provided to the consumer is to provide processing, storage, networking, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over selected networking components (e.g., host firewalls).

[0022] The deployment model is as follows: Private Cloud: The cloud infrastructure is used exclusively for one organization. In one embodiment, this may be managed by the organization or a third party and may exist on-premise or off-premise. Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with common concerns (e.g., mission, security requirements, policy and compliance considerations). In one embodiment, this may be managed by the organization or a third party and may exist on-premise or off-premise. Public Cloud: The cloud infrastructure is available to the general public or large industry entities and is owned by organizations that sell cloud services. Hybrid Cloud: A cloud infrastructure is a blend of two or more clouds (private, community or public) that remain unique entities but are joined by standardized or proprietary technologies that allow data and application portability (e.g. cloud bursting for load balancing between clouds).

[0023] Cloud computing environments are service-oriented with an emphasis on statelessness, low coupling, modularity and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0024] FIG. 1 illustrates a cloud computing environment 50 according to an embodiment of the present invention. As shown, in one embodiment, the cloud computing environment 50 includes one or more cloud computing nodes 10 that communicate with local computing devices used by cloud consumers, such as, for example, a PDA or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or a combination thereof. In one embodiment, the nodes 10 communicate with each other. In one embodiment, they are physically or virtually grouped (not shown) in one or more networks, such as private, community, public, or hybrid clouds, as described above, or a combination thereof. This enables the cloud computing environment 50 to provide infrastructure, platform, or software, or a combination thereof, as a service, for which the cloud consumer does not need to maintain resources on a local computing device. The types of computing devices 54A-54N shown in FIG. 7 are for illustrative purposes only, and it will be understood that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network, network addressable connection (e.g., using a web browser), or both.

[0025] 2 illustrates a set of functional abstraction layers provided by cloud computing environment 50, in accordance with an embodiment of the present invention. It will be understood that the components, layers, and functions illustrated in FIG. 2 are for illustrative purposes only, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0026] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, servers based on RISC (Reduced Instruction Set Computing) architecture 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0027] In one embodiment, the virtualization layer 70 provides an abstraction layer from which examples of virtualized entities are provided, such as virtualized servers 71, virtualized storage 72, virtualized networks including virtual private networks 73, virtualized applications and operating systems 74, and virtual clients 75.

[0028] In one embodiment, the management layer 80 provides the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides tracking of costs as resources are utilized within the cloud computing environment and billing or invoicing for consumption of these resources. In one embodiment, these resources include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and fulfillment 85 provides pre-allocation and procurement of cloud computing resources in anticipation of future demand in accordance with SLAs.

[0029] In one embodiment, the workload layer 90 provides examples of functionality for which a cloud computing environment is utilized. In one embodiment, example workloads and functionality provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and federated learning 96 (e.g., federated learning system 430, described in more detail later herein).

[0030] P i D generally represents a data party (i.e., a participant) that owns one or more data sources, where 1≦i≦N, and i and N are integers. i Generally, the data party P i Represents a data source owned by R. iGenerally, the data source D i In response to a query received by a data source D i Represents a response from.

[0031] FIG. 3 illustrates an exemplary system 300 that implements a conventional approach to federated learning. The system 300 includes: (1) multiple data parties P1, P2, ..., and P N D1, D2, …, and D N and (2) an aggregator 310 configured to aggregate the collected data. In a given epoch (i.e., a federated learning round or a learning / training run), the aggregator 310 collects data from data parties P i Data source D owned by i Send a query Q to a data source D i to the answer R to the query Q i For example, as shown in FIG. 3, at a given epoch, each data party P i In response to a query Q sent to data source D1, the aggregator 310 receives responses R1 from data source D1, R2 from data source D2, ..., and N Reply from R N where R1 = Q(D1), R2 = Q(D2), …, R N =Q(D N ). The system 300 generates responses R1, R2, ..., and R N Train a model M based on the aggregation / composition of .

[0032] The system 300 has high requirements for the infrastructure to ensure disaster recovery and high availability at the aggregator 310. For example, the system 300 includes a backup aggregator 320 for the aggregator 310.

[0033] System 300 lacks ease of setup and requires complex and costly configuration System 300 lacks multi-platform / operating system support (i.e., lacks the ability to extend to other platforms / operating systems).

[0034] Other issues that may arise with the system 300 include loss of data (e.g., historical data) from data parties that are not selected in a given epoch, difficulties in logging, slower development cycles, and memory congestion at the data parties if the aggregator 310 fails to collect data continuously.

[0035] 4 illustrates an exemplary computing architecture 400 for implementing distributed federated learning according to one embodiment of the present invention. In one embodiment, computing architecture 400 is a centralized computing architecture. In another embodiment, computing architecture 400 is a distributed computing architecture.

[0036] In one embodiment, computing architecture 400 includes computational resources, such as, but not limited to, one or more processor units 410 and one or more storage units 420. One or more applications may execute / run on computing architecture 400 utilizing the computational resources of computing architecture 400. In one embodiment, applications on computing architecture 400 include, but are not limited to, federated learning system 430. As described in more detail later herein, federated learning system 430 provides distributed, fault-tolerant, and scalable federated learning.

[0037] In one embodiment, the federated learning system 430 is configured to exchange data with one or more electronic devices 450 or one or more remote server devices 460, or both, via a connection (e.g., a wi-fi connection or a cellular data connection, a wired connection, or a combination of the two).

[0038] In one embodiment, electronic device 450 includes one or more computational resources, such as one or more processor units 451 and one or more storage units 452. One or more applications may execute / run on electronic device 450 utilizing the one or more computational resources of electronic device 450, such as, but not limited to, one or more software applications 454 that are loaded or downloaded to electronic device 450. Examples of software applications 454 include, but are not limited to, artificial intelligence (AI) applications, and the like.

[0039] Examples of electronic devices 450 include, but are not limited to, desktop computers, mobile electronic devices (e.g., tablets, smartphones, laptops, etc.), wearable devices (e.g., smart watches, etc.), Internet of Things (IoT) devices, smart appliances such as smart televisions, etc.

[0040] In one embodiment, electronic device 450 includes one or more input / output (I / O) units 453, such as a keyboard, a keypad, a touch interface, a display screen, etc., incorporated into or coupled to electronic device 450. A user may utilize I / O module 453 of electronic device 450 to configure one or more user settings, configure one or more parameters (e.g., validation properties, thresholds, etc.), provide input, etc.

[0041] In one embodiment, the electronic device 450 or the remote server device 460 or both may be a source of historical data or documents or both.

[0042] In one embodiment, the federated learning system 430 may be accessed or utilized by one or more online services (e.g., AI services) hosted on a remote server device 460 and / or one or more software applications 454 (e.g., AI applications) running on the electronic device 450. For example, in one embodiment, a virtual assistant, search engine, or other type of software application 454 running on the electronic device 450 may invoke the system 430 to perform a task.

[0043] 5 illustrates an exemplary federated learning system 430 according to an embodiment of the present invention. In one embodiment, the system 430 includes an aggregator 500 and an object store 510. The aggregator 500 distributes one or more model parameters to one or more Ps of multiple data parties via the object store 510 during each round of federated learning. i By sharing with multiple data parties in a distributed manner, i (For example, P1, P2, …, and P N ) is configured to control one or more rounds of associative learning (i.e., one or more epochs or one or more learning / training runs).

[0044] In one embodiment, the aggregator 500 may collect the data from each data party P before one or more rounds of federated learning are initiated / triggered. i About Data Party P i 5, the aggregator 500 is configured to define corresponding namespaces in the object store 510 for each of the namespaces S1, S2, ..., and SN are data parties P1, P2, …, and P N Assign to.

[0045] In one embodiment, the aggregator 500 allocates a shared namespace in the object store 510 and stores the model information M before one or more rounds of federated learning are started / triggered. S in a shared namespace in the object store 510. Initially, at the start of the first / initial epoch, the model information M S contains either an initial model (base model) or an initial model parameter set that contains one or more model parameters that define the initial model.

[0046] Q i Generally, the data party P i present customized learning requirements to

[0047] In one embodiment, the aggregator 500 is a data party P i The corresponding namespaces in the object store 510 and the shared namespaces in the object store 510 are assigned / defined for each data party P i where the shared namespace is the model information M S For example, in one embodiment, at a given epoch, the aggregator 500 may collect data from one or more of the data parties P i For data party P i Customized learning requirements for Q i The learning request Q i Links to the object store 510, data parties P i the corresponding namespace in the object store 510 assigned / defined to iThe file name to use to store the response to the request, the credentials for the object store 510, and the shared namespace within the object store 510.

[0048] In one embodiment, the aggregator 500 may collect, at a given epoch, a set of data from multiple data parties P i Each of the learning requirements Q i In another embodiment, the aggregator 500 is configured to issue, at a given epoch, one or more P i Select one or more data parties P selected at a given epoch. i Learning requirements Q i The system 430 supports training of multiple local models in parallel in a given epoch.

[0049] R i 'Generally, the data party P i We denote the locally trained model (local model) by

[0050] In one embodiment, at a given epoch, the training request Q at the given epoch from the aggregator 500 i In response to receiving the data, the data party P i (1) A shared namespace in the object store 510 (learning request Q i (included in) from model information M S (2) fetch the model information M S、 and Data Party P i (3) train a model locally based on the training data held by R, and generate a local model R obtained from the local training. i ' to the corresponding namespace S in the object store 510 i (Learning requirement Q i It is configured to upload the file to the

[0051] For example, as shown in FIG. 5, at a given epoch, a data party P1 has model information M S and locally train a model R1′ based on a data source D1 owned by the data party P1, and upload the model R1′ resulting from the local training to a corresponding namespace S1 in the object store 510. As another example, the data party P2 may upload model information M S and locally train a model R2′ based on a data source D2 owned by the data party P2, and upload the model R2′ resulting from the local training to a corresponding namespace S2 in the object store 510. N is model information M S and Data Party P N Data source D owned by N Based on Model R N ' locally and the model obtained from the local training R N ' to the corresponding namespace S in the object store 510 N Upload to.

[0052] As mentioned above, if the given epoch is the first / initial epoch, then the model information M S contains either an initial (i.e., base) model or an initial model parameter set that contains one or more model parameters that define the initial model. i Each data party P i The data party P i Train an initial model locally based on training data owned by and then use that local model R i ' to get '.

[0053] As will be described in more detail below, if a given epoch is a subsequent epoch (i.e., not the first / initial epoch), then the model information M SA shared model includes either a shared model or a model parameter set including one or more model parameters that define a shared model. A shared model is a set of local models R uploaded to the object store 510 in a previous epoch. i '. In the subsequent epoch, the learning requirement Q i Each data party P i The data party P i Train a shared model locally based on training data owned by R i Get the new / updated version of '.

[0054] In one embodiment, at a given epoch, a data party P i corresponds to a namespace S in the object store 510. i The local model R i In response to uploading ', the object store 510 uploads its local model R i ' corresponds to the namespace S i In another embodiment, in a given epoch, the data party P i corresponds to a namespace S in the object store 510. i The local model R i In response to uploading the 'Data Party P i is its local model R i ' corresponds to the namespace S i For example, in a given epoch, the data party P i The aggregator 500 receives a corresponding namespace S in the object store 510. i Local model R iIn yet another embodiment, in a given epoch, the aggregator 500 notifies the object store 510 of one or more local models R that have been uploaded to the object store 510. i For example, the aggregator 500 may be configured to periodically (i.e., in a time-based mode) check the object store 510 for new / updated versions of one or more namespaces S in the object store 510 according to a time schedule. i One or more local models from R i The device is configured to retrieve new / updated versions of one or more of the following:

[0055] In one embodiment, in a given epoch, after checking the object store 510 for one or more uploads, or after checking the object store 510 or one or more data parties P i After receiving one or more notifications of one or more uploads from the object store 510, the aggregator 500 may retrieve one or more local models R from the object store 510. i Reads out one or more local models R i , and stores the resulting shared model in a shared namespace in the object store 510. If the given epoch is the first / initial epoch, the aggregator 500 aggregates one or more local models R i If the given epoch is a subsequent epoch, the aggregator 500 aggregates one or more local models R uploaded to the object store 510 in the subsequent epoch to obtain a shared model. i ' to get the new / updated version of the shared model.

[0056] In one embodiment, the aggregator 500 writes the shared model and each new / updated version of the shared model to a shared namespace in the object store 510 (i.e., the aggregator persists multiple versions of the shared model to the shared namespace after several epochs). For example, each version of the shared model is annotated or labeled with a version number indicating the particular epoch during which the aggregator 500 obtained the version of the shared model via aggregation. As the system 430 keeps track of each model after each epoch, the system 430 enables automatic logging and automatic model versioning, enabling further applications. These features contribute to the benefits of retaining historical data from all data parties and providing model versioning to go back to previous model states.

[0057] In another embodiment, in a given epoch, the aggregator 500 may store old / previous versions of the shared model in the shared namespace of the object store 510 as the shared model M for the given epoch. S Replace it with a new / updated version.

[0058] In one embodiment, the system 430 distributes new / updated versions of the shared model uploaded to the shared namespace to each data party P via a publish-subscribe type mechanism. i Notify.

[0059] In one embodiment, the aggregator 500 is configured to perform accelerated learning. For example, in one embodiment, each data party P i For i In the local model R i Maintain a corresponding history containing multiple versions of

[0060] In one embodiment, after a given epoch is finished, the aggregator 500 uploads the shared model obtained during the given epoch to the shared namespace and generates a new training request Q i one or more data parties P i Start / trigger another iteration / round of federated learning (i.e., another epoch or another learning / training run) by issuing a

[0061] In one embodiment, failure recovery in the system 430 is easy because the aggregator 500 is largely stateless. i However, the data party P can fetch the shared model directly from the shared namespace. i does not need to rely on the aggregator 500 for the sharing model. Unlike aggregators in conventional systems, the aggregator 500 is not a bottleneck or single point of failure.

[0062] In one embodiment, system 430 improves scalability by providing increased network connectivity (i.e., it can handle more connections and can operate under any type of network connectivity) and improved storage (i.e., better storage management and avoidance of extra hops). System 430 provides a scalable, fault-tolerant solution to federated learning that is simple, robust, and cost-effective.

[0063] In one embodiment, the system 430 allows for faster development cycles because the APIs for object stores are well-tested and known. The system 430 provides a standardized API to data parties and aggregators.

[0064] FIG. 6 is a flow diagram of an exemplary process 600 for distributed federated learning across multiple data parties, according to an embodiment of the present invention. At process block 601, process 600 begins. At process block 602, an aggregator (e.g., aggregator 500 of FIG. 5) defines a corresponding namespace for each data party (stakeholder) in an object store (e.g., object store 510 of FIG. 5), defines a shared namespace in the object store, and persists an initial model in the shared namespace. At process block 603, for one or more parties, the aggregator issues a customized learning request to the party that includes a link to the object store, the corresponding namespace in the object store, the current epoch (the file name used to store), credentials to the object store, and the shared namespace. At process block 604, each party that issued the learning request downloads the model from the shared namespace, trains the model locally, and uploads the resulting local model to the corresponding namespace in the object store. At processing block 605, the aggregator checks the object store after a predetermined amount of time for a local model uploaded to the object store for the current epoch or receives notification from the object store or one or more parties of a local model uploaded to the object store for the current epoch. At processing block 606, the aggregator reads the local model uploaded to the object store for the current epoch and performs aggregation. At processing block 607, the aggregator performs accelerated learning. At processing block 608, the aggregator stores the aggregated model for the current epoch in a shared namespace. At processing block 609, the aggregator determines whether there is another round of federated learning. If the aggregator determines there is another round of federated learning, the process returns to processing block 603. If the aggregator determines there is not another round of federated learning, the process continues to block 610, where the process 600 ends.

[0065] In one embodiment, processing blocks 601 - 610 are performed by one or more components of system 430 , such as aggregator 500 and object store 510 .

[0066] FIG. 7 is a flow chart of an exemplary process 700 for controlling federated learning across multiple data parties in a distributed manner, according to an embodiment of the present invention. At process block 701, each data party is assigned to a corresponding namespace in the object store. At process block 702, a shared namespace is assigned in the object store. At process block 703, a round of federated learning is triggered by issuing a customized learning request to at least one data party, where each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store, and upload the local model resulting from the local training to the corresponding namespace in the object store to which the data party is assigned. At process block 704, at least one local model uploaded to the object store by at least one data party during the round of federated learning is retrieved from the object store. At process block 705, the at least one local model retrieved from the object store is aggregated to obtain a shared model.

[0067] In one embodiment, processing blocks 701 - 705 are performed by one or more components of system 430 , such as aggregator 500 .

[0068] 8 is a high-level block diagram illustrating an information processing system 800 useful for implementing embodiments of the present invention. The computer system includes one or more processors, such as processor 802. Processor 802 is connected to a communications infrastructure 804 (e.g., a communications bus, crossbar, or network).

[0069] The computer system may also include a display interface 806 that transfers graphics, text, and other data from the voice communications infrastructure 804 (or from a frame buffer, not shown) for display on a display unit 808. In one embodiment, the computer system also includes a main memory 810, preferably random access memory (RAM), and a secondary memory 812. In one embodiment, the secondary memory 812 includes, for example, a hard disk drive 814 and / or a removable storage drive 816, for example, a floppy disk drive, magnetic tape drive, or optical disk drive. The removable storage drive 816 reads from and / or writes to a removable storage unit 818 in a manner well known to those skilled in the art. The removable storage unit 818 may represent, for example, a floppy disk, compact disk, magnetic tape, or optical disk that is read from and written to by the removable storage drive 816. As will be appreciated, the removable storage unit 818 includes a computer readable medium having stored thereon computer software and / or data.

[0070] In alternative embodiments, the secondary memory 812 includes other similar means for allowing computer programs or other instructions to be loaded into the computer system. Such means include, for example, a removable storage unit 820 and an interface 822. Examples of such means include program packages and package interfaces (such as found in video game devices), removable memory chips (e.g., EPROM or PROM) and associated sockets, and other removable storage units 820 and interfaces 822 that allow software and data to be transferred from the removable storage unit 820 to the computer system.

[0071] In one embodiment, the computer system also includes a communications interface 824. The communications interface 824 allows software and data to be transferred between the computer system and external devices. In one embodiment, examples of the communications interface 824 include a modem, a network interface (e.g., an Ethernet card), a communications port, or a PCMCIA slot and card. In one embodiment, the software and data transferred through the communications interface 824 are in the form of signals, which may be, for example, electronic, electromagnetic, optical, or other signals capable of being received by the communications interface 824. These signals are provided to the communications interface 824 via a communications path (or channel) 826. In one embodiment, the communications path 826 carries signals and is implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channels, or a combination thereof.

[0072] The embodiments of the invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to perform aspects of the embodiments of the invention.

[0073] A computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Illustrative enumeration of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, punch cards or mechanically encoded devices such as ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. Computer-readable storage media, as used herein, is not to be construed as a transitory signal per se, such as an electric wave, a freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0074] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.

[0075] The computer readable program instructions for carrying out the operations of embodiments of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented languages ​​such as Smalltalk®, C++ or the like, procedural languages ​​such as the C programming language or similar programming languages. The computer readable program instructions may be executed as a stand-alone software package, entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, the electrical circuitry may execute computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electrical circuitry, which may include, for example, programmable logic circuitry, a field programmable gate array (FPGA), or a programmable logic array (PLA), to perform aspects of embodiments of the present invention.

[0076] Aspects of embodiments of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer readable program instructions.

[0077] These computer readable program instructions, when provided to a computer processor or other programmable data processing apparatus, cause the instructions to execute on the computer processor or other programmable data processing apparatus to produce a machine-generated means for implementing the functions / acts identified in the block or blocks of the flowchart diagrams and / or blocks. These computer readable program instructions may also be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus or other device or combination thereof to function in a particular manner, such that a computer readable storage medium having instructions stored thereon includes an article of manufacture including instructions that implement aspects of the functions / acts identified in the block or blocks of the flowchart diagrams and / or blocks.

[0078] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to generate a computer implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement aspects of the functions / actions identified in a block or blocks of the flowchart and / or blocks.

[0079] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment or part of instructions, including one or more executable instructions for implementing a particular logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order depicted in the drawings. For example, two blocks shown in succession may in fact be realized as a single step, may be executed simultaneously, may be executed substantially simultaneously in a partially or fully overlapping manner, or the blocks may be executed in reverse order depending on the functionality involved. It should be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations may be implemented by a special purpose hardware-based system that performs a particular function or action or implements a combination of special purpose hardware and computer instructions.

[0080] From the above description, it can be seen that the embodiments of the present invention provide a system, a computer program product, and a method for implementing the embodiments of the present invention. The embodiments of the present invention further provide a non-transitory computer usable storage medium for implementing the embodiments of the present invention. The non-transitory computer usable storage medium has a computer readable program, which when executed on a computer, causes the computer to perform the steps of the embodiments of the present invention described herein. References to elements in the singular form in the claims are not intended to mean "only one" unless so expressly stated, but rather "one or more." All structural and functional equivalents to the elements of the above exemplary embodiments now known or that become known in the future to those skilled in the art are intended to be encompassed within the scope of the claims. No claim element herein should be construed under the provisions of 35 U.S.C. 112, paragraph 6, unless the element is expressly recited using the phrase "means for" or "step for."

[0081] The terms used herein are for the purpose of describing particular embodiments of the invention only and are not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, it is to be understood that the terms "comprise" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements or components, or combinations thereof, and do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0082] In the following claims, equivalents of corresponding structures, materials, acts, and all means or step-plus-function elements are intended to include any structure, material, or act for performing a function in combination with other claimed elements as expressly claimed.

[0083] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to allow other skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method for federated learning across multiple data parties, comprising: at least one processor comprising: assigning each data party to a corresponding namespace in an object store; allocating a shared namespace within the object store; triggering a round of federated learning by issuing a customized learning request to at least one data party, where each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store, and upload a local model resulting from the local training to a corresponding namespace in the object store to which the data party is assigned; retrieving from the object store at least one local model uploaded to the object store by the at least one data party during the federated learning round; aggregating the at least one local model retrieved from the object store to obtain a shared model; A method for performing.

2. The method of claim 1 , wherein the shared model is uploaded to the shared namespace in the object store.

3. The method of claim 2 , wherein each data party is notified of the shared model that is uploaded to the shared namespace in the object store.

4. The method further comprises the step of: triggering subsequent rounds of federated learning by issuing subsequent customized learning requests to at least one data party, where each subsequent customized learning request issued to a data party triggers the data party to read the shared model from the shared namespace in the object store, locally train the shared model based on training data owned by the data party, and upload an updated version of the local model resulting from the local training to a corresponding namespace in the object store to which the data party is assigned. The method of claim 2 or 3, further comprising:

5. The step of triggering a round of federated learning includes: selecting a set of data parties; Triggering the set of data parties to locally train a set of models in parallel by issuing a customized training request to each data party of the set of data parties. The method according to any one of claims 1 to 4, comprising:

6. The step of triggering a round of federated learning includes: Triggering multiple data parties to locally train multiple models in parallel by issuing customized training requests to each data party. The method according to any one of claims 1 to 5, comprising:

7. The method further comprises the step of: storing, for each data party, multiple versions of the uploaded local model in a corresponding namespace in the object store assigned to the data party during multiple rounds of federated learning. The method of any one of claims 1 to 6, further comprising:

8. The method further comprises the step of: storing multiple versions of the uploaded shared model in a shared namespace in the object store during multiple rounds of federated learning. The method of any one of claims 1 to 7, further comprising:

9. The method further comprises the step of: receiving from the object store at least one notification of the at least one local model uploaded to the object store by the at least one data party during the federated learning round; The method of any one of claims 1 to 8, further comprising:

10. The method further comprises the step of: receiving from the at least one data party at least one notification of the at least one local model uploaded to the object store by the at least one data party during the federated learning round; The method of any one of claims 1 to 9, further comprising:

11. The method further comprises the step of: periodically checking the object store for one or more local models uploaded to the object store by one or more data parties according to a schedule. The method of any one of claims 1 to 10, further comprising:

12. A system for federated learning across multiple data parties, At least one processor; a non-transitory processor-readable memory device for storing instructions; the instructions, when executed by the at least one processor, cause the at least one processor to perform an operation, the operation comprising: assigning each data party to a corresponding namespace in the object store; Allocating a shared namespace within the object store; triggering a round of federated learning by issuing a customized learning request to at least one data party, where each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store, and upload a local model resulting from the local training to a corresponding namespace in the object store to which the data party is assigned; Retrieving from the object store at least one local model uploaded to the object store by the at least one data party during the federated learning round; aggregating the at least one local model retrieved from the object store to obtain a shared model; Including, the system.

13. The system of claim 12 , wherein the shared model is uploaded to the shared namespace in the object store.

14. The operation includes: triggering subsequent rounds of federated learning by issuing subsequent customized learning requests to at least one data party, where each subsequent customized learning request issued to a data party triggers the data party to read a shared model from the shared namespace in the object store, locally train the shared model based on training data owned by the data party, and upload an updated version of the local model resulting from the local training to a corresponding namespace in the object store to which the data party is assigned. The system according to claim 12 or 13, comprising:

15. Triggering a round of federated learning selecting a set of data parties; triggering the set of data parties to locally train a set of models in parallel by issuing a customized training request to each data party of the set of data parties. The system according to any one of claims 12 to 14, comprising:

16. Triggering a round of federated learning Triggering multiple data parties to locally train multiple models in parallel by issuing customized training requests to each data party. The system according to any one of claims 12 to 15, comprising:

17. The operation includes: storing, for each data party, multiple versions of the uploaded local model in a corresponding namespace in the object store assigned to the data party during multiple rounds of federated learning; The system according to any one of claims 12 to 16, comprising:

18. The operation includes: storing multiple versions of the uploaded shared model in a shared namespace within the object store during multiple rounds of federated learning; The system according to any one of claims 12 to 17, comprising:

19. 1. A computer-readable storage medium storing a computer program for federated learning across multiple data parties, the computer program comprising: assigning each data party to a corresponding namespace in the object store; Allocating a shared namespace within the object store; triggering a round of federated learning by issuing a customized learning request to at least one data party, where each customized learning request issued to a data party triggers the data party to locally train a model based on training data owned by the data party and one or more model parameters stored in the shared namespace in the object store, and upload a local model resulting from the local training to a corresponding namespace in the object store to which the data party is assigned; Retrieving from the object store at least one local model uploaded to the object store by the at least one data party during the federated learning round; aggregating the at least one local model retrieved from the object store to obtain a shared model; A computer-readable storage medium for causing the computer to execute the method of the present invention.

20. 20. The computer-readable storage medium of claim 19, wherein the shared model is uploaded to the shared namespace in the object store.

21. A computer program comprising program code means adapted to perform the method according to any one of claims 1 to 11, when the program is run on a computer.

Images