System and method for training a machine learning model in a distributed system

By compressing update sizes in federated learning systems using dense arrays of parameter deltas ordered by a reference model, the method addresses communication bottlenecks in distributed training, ensuring efficient data transmission and model accuracy in resource-limited environments.

US20250103902A1Pending Publication Date: 2025-03-27KK TOSHIBA
11 Cites 4 Cited by

Patent Information

Application Number
US18/471749
Authority / Receiving Office
US · United States
Patent Type
Applications(United States)
Current Assignee / Owner
Filing Date
2023-09-21
Publication Date
2025-03-27

AI Technical Summary

Technical Problem

Federated learning systems face challenges in efficiently training machine learning models due to the large amount of data required for updates across distributed nodes, particularly in scenarios where IoT devices have limited network quality, leading to communication bottlenecks and resource constraints.

Method used

The method involves compressing update sizes based on communication link quality by using dense arrays of parameter deltas ordered by a reference model, allowing for adaptive reduction of parameter updates and efficient data transmission, without the need for pre-training or specific training data to select parameters for pruning.

Benefits of technology

This approach reduces the overall data transmitted during training, enabling more frequent and efficient updates between nodes while maintaining model accuracy, even in resource-constrained environments.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

A computer-implemented method for training a machine learning model in a distributed system, The distributed system comprises a plurality of nodes that exchange updates to communally train the machine learning model. Each node of the plurality of nodes maintains a local version of the machine learning model. The local version of the machine learning model of each of the plurality of nodes has been initialised with the same one or more respective parameter values. The method comprises a node: receiving an update to a local model from at least one other node in the distributed system, the local model comprising the local version of the machine learning model and the update comprising a dense array of one or more first parameter deltas, the one or more first parameter deltas being ordered in the dense array in an order determined by a reference model, each first parameter delta representing a difference between a parameter of the local model and a corresponding parameter of an updated version of the machine learning model that is maintained by the at least other node; updating the local model based on the received update and the reference model to determine an updated local model; determining one or more second parameter deltas, each second parameter delta representing a difference between a parameter of the updated local model and a corresponding parameter of a previous version of the local model; and sending an update to the at least one other node in the distributed system, wherein the update comprises a dense array of the one or more second parameter deltas, the one or more second parameter deltas being ordered in the dense array in an order determined by the reference model.
Need to check novelty before this filing date? Find Prior Art