Methods and equipment for federated learning

Abstract

Methods and apparatuses for performing federated learning are provided. A global model is distributed from a server to multiple client devices. At each of the client devices: model inversion is performed on the global model to generate synthetic data; the global model is trained on an augmented dataset of the collected and synthetic data to generate a corresponding client model; and the corresponding client model is sent to the server. At the server: client models are received from the multiple client devices, where each client model is received from a corresponding client device among the multiple client devices; model inversion is performed on each client model to generate a synthetic dataset; the client models are averaged to generate an average model; and the average model is trained using the synthetic dataset to generate an updated model.

Description

Technical Field

[0001] This disclosure generally relates to machine learning methods, and more specifically to methods and apparatus for improving fairness in federated learning. Background Technology

[0002] Federated learning is a training paradigm used in domains where data is prohibited from leaving the local client device due to data sensitivity or privacy concerns. However, in distributed scenarios, local client devices may exhibit high statistical heterogeneity. Many federated learning methods that pursue a single model ignore deviations generated towards the dominant data distribution, raising fairness issues.

[0003] A model is considered fairer than other models if its performance (e.g., accuracy) across M different devices or categories is more uniform than that of other models across the same devices or categories. Specifically, when measured across M categories or devices, the fairer model has a smaller variance in performance outcomes than the other models.

[0004] Federated learning enables the training of machine learning models on heterogeneous distributed networks in a privacy-preserving manner by maintaining user data on each local client device and sharing model updates only with the global server. The server generates an average model by taking a weighted average of the local client models. In generating the average, the objective is to minimize the following objective function mentioned in equation (1):

[0005]

[0006] Among them, F k (w):=E x ~D k [f k [(w;x_k)] is the local objective function, N is the number of devices or clients, and p k ≥0 represents the weight of each device.

[0007] Statistical heterogeneity in the data distribution of client devices is a challenge in federated learning. When federated learning focuses on learning a single global model, it is often hampered by model divergence because local client models can vary significantly. Summary of the Invention

[0008] According to one embodiment, a method for performing federated learning on a client device is provided. The client device receives a global model from a server. The client device performs model inversion on the global model to generate synthetic data for one or more categories of data in the global model. The client device augments collected data in its dataset with the synthetic data to generate an augmented dataset with a more uniform distribution of data across categories than the original dataset. The client device trains the global model on the augmented dataset to generate a client-side model. The client device then sends the client-side model from the client device to the server.

[0009] According to one embodiment, a method for performing federated learning on a server is provided. The server distributes a global model to multiple client devices. The server receives client models from the multiple client devices. Each client model is received from a corresponding client device among the multiple client devices. The server performs model inversion on each client model to generate synthetic data for each client model. The server uses the synthetic data from the client models to generate a synthetic dataset. The server averages the client models to generate an average model. The server trains the average model using the synthetic dataset to generate an updated global model.

[0010] According to one embodiment, a client device is provided for performing federated learning. The client device includes a processor and a non-transitory computer-readable storage medium storing instructions. When executed, the instructions cause the processor to receive a global model from a server and perform model inversion on the global model to generate synthetic data for one or more categories of data in the global model. The instructions also cause the processor to augment collected data in the client device's dataset with the synthetic data to generate an augmented dataset with a more uniform distribution of data across categories than the original dataset. The instructions further cause the processor to train the global model on the augmented dataset to generate a client model and send the client model to the server.

[0011] According to one embodiment, a server is provided for performing federated learning. The server includes a processor and a non-transitory computer-readable storage medium storing instructions. When executed, the instructions cause the processor to distribute a global model to a plurality of client devices and receive client models from the plurality of client devices. Each client model is received from a corresponding client device among the plurality of client devices. The instructions also cause the processor to perform model inversion on each client model to generate synthetic data for each client model, and to generate a synthetic dataset using the synthetic data from the client models. The instructions further cause the processor to average the client models to generate an average model, and to train the average model using the synthetic dataset to generate an updated global model. Attached Figure Description

[0012] The above and other aspects, features, and advantages of certain embodiments of this disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 This is a diagram illustrating a federated learning system according to one embodiment;

[0014] Figure 2 This is a flowchart illustrating a method for federated model inversion (FMIL) at a local node using zero-sample data generation (ZSDG) according to one embodiment;

[0015] Figure 3 This is a flowchart illustrating a method for Local Model Inversion (LMIF) on a federated server using ZSDG, according to one embodiment; and

[0016] Figure 4 This is a block diagram of an electronic device in a network environment according to one embodiment. Detailed Implementation

[0017] In the following description, embodiments of the present disclosure will be specifically described with reference to the accompanying drawings. It should be noted that the same elements will be designated by the same reference numerals, although they are shown in different drawings. In the following description, only specific details, such as particular configurations and components, are provided to aid in a comprehensive understanding of the embodiments of the present disclosure. Therefore, it will be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Additionally, for clarity and brevity, descriptions of well-known functions and structures are omitted. The terminology described below is intended to define functions within the scope of this disclosure and may vary depending on the user, the user's intent, or habits. Therefore, the definitions of the terms should be determined based on the content throughout this specification.

[0018] This disclosure can have various modifications and embodiments, of which are described in detail below with reference to the accompanying drawings. However, it should be understood that this disclosure is not limited to the embodiments, but includes all modifications, equivalents, and substitutions within the scope of this disclosure.

[0019] While terms including ordinal numbers such as first, second, etc., can be used to describe various elements, structural elements are not limited by such terms. The term is used only to distinguish one element from another. For example, a first structural element may be referred to as a second structural element without departing from the scope of this disclosure. Similarly, a second structural element may also be referred to as a first structural element. As used herein, the term "and / or" includes any and all combinations of one or more associated items.

[0020] The terminology used herein is for the purpose of describing various embodiments of this disclosure only and is not intended to limit this disclosure. The singular forms are intended to include the plural forms unless the context clearly indicates otherwise. In this disclosure, the terms “comprising” or “having” indicate the presence of a feature, number, step, operation, structural element, part, or combination thereof, and do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, structural elements, parts, or combinations thereof.

[0021] Unless otherwise defined, all terms used herein shall have the same meaning as understood by one of skill in the art to which this disclosure pertains. For example, terms as defined in commonly used dictionaries shall be interpreted as having the same meaning as in the context of the relevant field of this art, and shall not be interpreted as having an ideal or overly formal meaning, unless expressly defined herein.

[0022] The electronic device according to one embodiment can be one of various types of electronic devices. For example, the electronic device may include a portable communication device (e.g., a smartphone), a computer, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to one embodiment of this disclosure, the electronic device is not limited to those described above.

[0023] The terminology used in this disclosure is not intended to limit the disclosure, but is intended to include various changes, equivalents, or substitutions for the corresponding embodiments. Regarding the description of the drawings, similar reference numerals may be used to refer to similar or related elements. The singular form of a noun corresponding to an item may include one or more of that thing, unless the relevant context clearly indicates otherwise. Each phrase used herein, such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C,” may include all possible combinations of the items listed together in one of the corresponding phrases. Terms used herein, such as “first,” “second,” “first,” and “second,” may be used to distinguish a corresponding component from another component, but are not intended to limit the components in other respects (e.g., importance or order). The intention is that if an element (e.g., a first element) is referred to as “coupled to another element (e.g., a second element)”, “coupled to another element (e.g., a second element)”, “connected to another element (e.g., a second element)”, or “connected to another element (e.g., a second element)” with or without the terms “operationally” or “communicatively”, it indicates that the element can be coupled to other elements directly (e.g., wired), wirelessly, or via a third element.

[0024] As used herein, the term "module" can include units implemented in hardware, software, or firmware, and can be used interchangeably with other terms such as "logic," "logic block," "part," and "circuit system." A module can be a single integrated component, or a minimum unit, or a portion thereof, suitable for performing one or more functions. For example, according to one embodiment, a module can be implemented as an application-specific integrated circuit (ASIC).

[0025] Federated learning is a training paradigm that adapts models to scenarios with distributed data. Figure 1 This is a diagram illustrating a federated learning system according to an embodiment. The federated server 102 distributes a global model to client devices 104-1 to 104-n. Client devices 104-1 to 104-n return client models with local updates to the federated server while maintaining their local data.

[0026] While the model can maintain acceptable average performance across client devices, its performance on individual client devices may vary. For example, when the collected data is skewed towards multiple classes or categories, the model is more likely to achieve better performance on data with fewer classes compared to data with fewer classes. Therefore, federated learning algorithms may result in models with high average performance but also high variance in performance across different client devices.

[0027] As used herein, the term "collected data" can generally refer to data that is being processed. Collected data does not need to be data captured by sensors of the device performing the processing. Collected data may have already undergone some processing.

[0028] Skewed distributions of collected data can cause unfairness, or high performance variance. Here, an averaging method is provided that mitigates the skewness of the distribution within the training / collected data to train a more equitable model. This method addresses the statistical heterogeneity of federated learning with ZSDG (i.e., data augmentation without explicit data sharing) and achieves a more equitable model across different client devices.

[0029] Embodiments of this disclosure provide a federated learning approach that employs ZSDG to mitigate statistical heterogeneity regarding the distribution of insufficient data. This encourages more uniform performance accuracy across client devices in a federated network. This approach improves both accuracy and fairness.

[0030] A first method is provided to improve the fairness of federated learning via FMIL, in which client devices (e.g., local nodes) perform model inversion on a global model transmitted from a federated server. Thus, data is inferred on the client device even without sharing training data from other client devices.

[0031] A second method is provided to improve the fairness of federated learning via LMIF, wherein a federated server performs model inversion on the received client models to generate synthetic data, thereby generating a balanced dataset on the federated server to be used in additional training steps. Thus, data is derived on the server even without sharing the training data from the client devices.

[0032] According to the second method, assuming the federated server already provides a fair model using LMIF, client devices can also be trained to match the layer statistics of the federated server. This prevents client devices from deviating excessively from their own data.

[0033] Both the first and second methods perform data-free model inversion by generating pseudo-data or synthetic data from the trained model without accessing the actual training data. Each data sample has a class label.

[0034] In the first embodiment of model inversion, model M is set as a neural network comprising L layers. For simplicity, it is assumed that model M has L batch normalization (BN) layers, and the activations before the i-th BN layer are represented as z. i .

[0035] During forward propagation, z i The mean μ parameterized from the i-th BN layer i and variance σ i Normalization. Note that, given a pre-trained model M, the BN statistics for all BN layers are stored and accessible. Therefore, to generate a model that best matches the BN statistics stored in the BN layers with a given target class, normalization is required. pseudo-input The optimization problem can be solved as proposed in the following equation (2).

[0036]

[0037] To solve equation (2), the model parameters are fixed and the input from random samples is updated via gradient descent. Because input reconstruction does not require training data, it is called ZSDG. Since the pre-trained model M is fixed, the generated pseudo-input... The visual quality is highly dependent on the performance of M.

[0038] In another embodiment of model inversion, the generator is trained to generate synthetic data using adversarial knowledge distillation without data. The generator is trained with adversarial loss to force it to produce synthetic data, similar to performing knowledge distillation on collected data (e.g., real or raw data) from a reference network (teacher) to another network (student). The generator is trained to force its synthetic data to produce statistics that match the Batch Normalization (BN) statistics stored in the reference model pre-trained on the collected data. The generator is also trained such that its synthetic images produce small categorical entropy and high batch entropy from the reference pre-trained network. A conditional generator can be used to generate data corresponding to specific labels.

[0039] According to an embodiment designed to improve fairness in federated learning via FMIL, a client device (e.g., a local node) derives data by inverting a model delivered by a federated server. This generated data is then used for data augmentation to improve the training of the local client model.

[0040] Here, (x; y) is set as the real local training input / label pair, and The generated synthetic data (e.g., pseudo-data) is used. To mitigate statistical heterogeneity, each client device augments its training set with synthetic data for categories with insufficient collected data.

[0041] Specifically, the server distributes model M to each client device, and the client device iterates through all possible models. To execute ZSDG to generate This enables local training to be based on a more balanced dataset. For example, if (x i y i ) is the data used for the i-th client device, and If the synthetic data is generated by the i-th client device, then the i-th client device will use the enhanced data. After local training, the client device returns the updated model M to the server for aggregation.

[0042] Each client device has its own training / collected data (x) with different statistical properties. i y i A collection of data, and synthetic data generated on each client. The images generated during model inversion processing are not the same.

[0043] Figure 2 This is a flowchart illustrating a method for using ZSDG in FMIL according to an embodiment. For simplicity, the label y corresponding to data element x is removed.

[0044] In 202, during communication round t, the federated server will send the federated global model M. f，t-1 A subset S distributed to client devices t In 204, the subset S of client devices t Each client device i generates a model for the federated global model M by performing model inversion on the federated global model (e.g., by using image distillation or ZSDG). f Synthetic data for all categories, or for a subset of categories for which insufficient data has been collected. In 206, each client device i synthesizes data. Enhance its collected data x i To generate a cross-federal global model M f The category is compared to the collected data x i Augmented dataset with a more uniform distribution

[0045] Typically, the frequency distribution of the classes in a dataset is used as a measure when determining whether a dataset has a more uniform distribution across classes. A dataset is more uniform if the class distribution approximates a discrete uniform distribution. For example, if the size of the dataset (e.g., the number of labeled dataset points) is S, and there are N classes, then the dataset is uniform if there are S / N labeled points for each of the N classes.

[0046] In 208, each client device i trains a federated global model M on its augmented dataset. f，t-1 And generate an updated client model. In 210, each client device i will update the client model. Send to the federated server. In 212, the federated server uses a weighted average to calculate the subset S from the client devices. t The received client models are averaged, and a new federated average model is generated that can be redistributed to client devices. This method returns to 202.

[0047] When testing the FMIL approach, the server can select 100 client devices with a score C = 0:1 during each communication round, for a total of T = 100 rounds. Each selected client device can train its own model for E = 5 local epochs with a minimum batch size B = 10. Each client device has images from at most 2 categories, with each category comprising 250 images. Starting at epoch 80, ZSDG can be initiated for local training. The number of augmented categories is 64. After data-augmented federated learning, the final aggregated model is tested, and test accuracy and variance are obtained across all client devices. The final aggregated model of the FMIL approach exhibits higher test accuracy and lower variance (greater fairness) compared to the standard federated average model. In particular, performance and fairness are improved simultaneously.

[0048] This approach to FMIL can be appropriately used when the client device has sufficient computational and storage complexity, for example, when the client device does not have access to data from other hospitals but is interested in learning from such data.

[0049] According to an embodiment provided to improve the fairness of federated learning via LMIF, the model transmitted from the local node is inverted on the federated server to derive synthetic data. This generated synthetic data is used by the federated server to improve the composition of local client models or for model averaging.

[0050] Constraints on local client devices include limited computing resources and storage capacity, which may limit the application of data augmentation relative to FMIL as described above. Additionally, the i-th client device may be more concerned with its own collected data (xi). i y i The model performance is optimized for specific datasets, rather than a general model that works for all data. Therefore, in this embodiment, ZSDG and federated learning are proposed on the federated server side.

[0051] Specifically, the federated server distributes the federated global (average) model M to each client device. Each client device i uses its collected local data (x... i y i Train the federated global model M. Each client device i will update model M. i Return to the federated server. The server updates model M for each update. i ZSDG is performed to generate synthetic data for each client device i, and the synthetic data is mixed in a balanced manner. The server uses an averaging algorithm to aggregate newly received models into an updated average federated model. The server fine-tunes the weights of the average model to generate a fair federated model by training on the mixed synthetic data.

[0052] According to another embodiment, the server runs a averaging algorithm on the received client model to generate a first average federated model. The server generates synthetic data (e.g., pseudo-data) from the first average federated model. The server evaluates the first average federated model based on the generated synthetic data and identifies underperforming categories. For the underperforming categories, the server generates more data from its first average federated model and mixes the additionally generated synthetic data with the previous data to generate a dataset with more samples from the underperforming categories. The server fine-tunes its first average model using the newly generated data.

[0053] Compared to client-side data augmentation (FMIL), server-side augmentation (LMIF) avoids the computational and storage limitations of local devices. Furthermore, each client device updates the received model from the same initialization point, encouraging equitable model performance as a whole.

[0054] Figure 3 This is a flowchart illustrating a method for using ZSDG-based LMIF according to an embodiment.

[0055] In 302, during communication round t, the federated server will send the federated global model M. f，t-1 A subset S distributed to client devices t In 304, a subset S of the client devices t Each client device i in its real data {x i} t Training the federated global model M f，t-1 And generate an updated client model. In 306, each client device i updates its client model. Send to the federal server.

[0056] In 308, the federated server processes each received client model. Perform model inversion and generate synthetic data. In order to generate a global model M with cross-federation f Balanced synthetic federated datasets with uniform distribution of categories In another embodiment, if the server has prior knowledge of the distribution of client data, the server can generate a skewed dataset to compensate for insufficient categories.

[0057] As mentioned above, if the class distribution is close to a discrete uniform distribution, the dataset has a more uniform distribution across classes. For example, if the size of the dataset (e.g., the number of labeled dataset points) is S, and there are N classes, then the dataset is uniform if there are S / N labeled points for each of the N classes.

[0058] In 310, the federated servers use a weighted average. The received client models are averaged. In another embodiment, the federated server addresses insufficient category forms. From its first average federal model More data is generated to make the dataset have more samples of the under-resourced categories. In another embodiment, the category server uses data generated from previous federated training epochs. Enhance its dataset.

[0059] In 312, the server is balancing the synthetic federated dataset. Training or fine-tuning the average federated model To generate an updated federated model M f，t The updated federated model can then be redistributed to client devices by returning to a 302 redirect.

[0060] During the same federal study period, or during alternating federal study periods, Figure 3 Implementation examples (LMIF) and Figure 2 The implementation examples (FMIL) can be executed simultaneously.

[0061] When a fair model is generated on the server side, the training algorithm on the local client device (e.g., a local node) can be modified, and training can be regularized to ensure that the statistics of the model weights on the client device are close to those of the model weights on the federated server, thereby encouraging a fair local model. In one embodiment, stored BN statistics of the federated model are used. In another embodiment, for each batch of input data, the statistics of the k-th client model and the l-th layer of the federated model f are matched.

[0062] Update the weights of the k-th user during training epoch (t+1) to minimize the objective function, where F k (.) is the primary training objective. μ k，l μ f，l These are the averages of the k-th user and the l-th layer of the federated model (or stored in their respective batch normalization layers), respectively. Similarly, σ k，l , σ f，l These are the standard deviations of the k-th user and the l-th layer of the federated model, respectively, or the standard deviations of their corresponding BN layers. The objective function is described in the following equation (3).

[0063]

[0064] Therefore, the client device matches the statistics of the federated model, and implicitly assumes that the federated model is fair. This training process can be deployed on the client device along with the LMIF on the federated server, which guarantees that the federated model distributed to the client device is fair.

[0065] In another embodiment, instead of minimizing the distance between the mean and variance as described above, the distance between the distributions is minimized by minimizing the KL divergence between two Gaussian distributions with corresponding means and variances, as proposed in the following equation (4).

[0066]

[0067] Figure 4 This is a block diagram of an electronic device in a network environment according to one embodiment. Relative to... Figure 2 Implementation examples, Figure 4 The electronic device 401 can be specifically implemented as a client device, and Figure 4 The 408 error on the server can be specifically implemented as a federated server. Compared to... Figure 3 Implementation examples, Figure 4 The electronic device 401 can be specifically implemented as a federated server, and the external electronic devices 402 and 404 can be specifically implemented as client devices.

[0068] refer to Figure 4 In network environment 400, electronic device 401 can communicate with external electronic device 402 via a first network 498 (e.g., a short-range wireless communication network), or with external electronic device 404 or server 408 via a second network 499 (e.g., a long-range wireless communication network). Electronic device 401 can communicate with external electronic device 404 via server 408. Electronic device 401 may include processor 420, memory 430, input device 450, sound output device 455, display device 460, audio module 470, sensor module 476, interface 477, haptic module 479, camera module 480, power management module 488, battery 489, communication module 490, subscriber identification module (SIM) 496, or antenna assembly 497. In one embodiment, at least one of the components (e.g., display device 460 or camera module 480) may be omitted from electronic device 401, or one or more other components may be added to electronic device 401. Some of the components may be implemented as a single integrated circuit (IC). For example, sensor module 476 (e.g., fingerprint sensor, iris sensor, or illuminance sensor) can be implemented as embedded in display device 460 (e.g., display).

[0069] Processor 420 may run software (e.g., program 440) to control at least one other component (e.g., hardware or software component) of electronic device 401 coupled to processor 420, and may perform various data processing or calculations. As at least part of data processing or calculation, processor 420 may load commands or data received from another component (e.g., sensor module 476 or communication module 490) into volatile memory 432, process commands or data stored in volatile memory 432, and store generated data in non-volatile memory 434. Processor 420 may include a main processor 421 (e.g., central processing unit (CPU) or application processor (AP)) and an auxiliary processor 423 (e.g., graphics processing unit (GPU), image signal processor (ISP), sensor hub processor, or communication processor (CP)) operably independent of or combined with the main processor 421. Additionally or alternatively, auxiliary processor 423 may be adapted to consume less power than the main processor 421 or to perform specific functions. The auxiliary processor 423 can be implemented separately from the main processor 421 or as part of the main processor 421.

[0070] When the main processor 421 is disabled (e.g., in sleep mode), the auxiliary processor 423 can control, in conjunction with the main processor 421, at least some of the functions or states related to at least one component of the electronic device 401 (e.g., display device 460, sensor module 476, or communication module 490). The auxiliary processor 423 (e.g., an image signal processor or a communication processor) can be implemented as part of another component functionally related to the auxiliary processor 423 (e.g., camera module 480 or communication module 490).

[0071] The memory 430 may store various data used by at least one component of the electronic device 401 (e.g., processor 420 or sensor module 476). The various data may include, for example, software (e.g., program 440) for commands associated with it and input or output data. The memory 430 may include volatile memory 432 or non-volatile memory 434.

[0072] Program 440 may be stored as software in memory 430 and may include, for example, an operating system (OS) 442, middleware 444, or application 446.

[0073] Input device 450 can receive commands or data from outside electronic device 401 (e.g., from a user) to be used by another component of electronic device 401 (e.g., processor 420). Input device 450 may include, for example, a microphone, mouse, or keyboard.

[0074] The sound output device 455 can output sound signals to the outside of the electronic device 401. The sound output device 455 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or recording, and the receiver can be used for incoming calls. The receiver can be implemented separately from the speaker or as part of the speaker.

[0075] Display device 460 can visually provide information to the outside of electronic device 401 (e.g., to a user). Display device 460 may include, for example, a display, a holographic device, or a projector, and a control circuitry system that controls a corresponding one of the display, holographic device, and projector. Display device 460 may include a touch circuitry system adapted to detect touch, or a sensor circuitry system (e.g., a pressure sensor) adapted to measure the intensity of the force caused by touch.

[0076] The audio module 470 can convert sound into electrical signals and vice versa. The audio module 470 can obtain sound via the input device 450, or output sound via the sound output device 455 or via headphones of the external electronic device 402 coupled directly (e.g., wired) or wirelessly to the electronic device 401.

[0077] Sensor module 476 can detect the operating state of electronic device 401 (e.g., power or temperature) or the environmental state outside electronic device 401 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. Sensor module 476 may include, for example, a gesture sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0078] Interface 477 may support one or more specified protocols for direct (e.g., wired) or wireless coupling of electronic device 401 to external electronic device 402. Interface 477 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0079] Connection terminal 478 may include a connector through which electronic device 401 can be physically connected to external electronic device 402. Connection terminal 478 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0080] The haptic module 479 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that can be recognized by a user through touch or muscle movement sensation. The haptic module 479 may include, for example, a motor, a piezoelectric element, or an electrical exciter.

[0081] Camera module 480 can capture still or moving images. Camera module 480 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0082] The power management module 488 can manage the power supplied to the electronic device 401. The power management module 488 can be implemented as at least part of a power management integrated circuit (PMIC).

[0083] Battery 489 can supply power to at least one component of electronic device 401. Battery 489 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0084] Communication module 490 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 401 and external electronic devices (e.g., external electronic device 402, external electronic device 404, or server 408), and the execution of communication via the established communication channel. Communication module 490 may include one or more communication processors that are independent of processor 420 (e.g., AP) and support direct (e.g., wired) or wireless communication. Communication module 490 may include wireless communication module 492 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module 494) or wired communication module 494 (e.g., local area network (LAN) communication module or power line communication (PLC) module). One of these communication modules can communicate with an external electronic device via a first network 498 (e.g., a short-range communication network, such as Bluetooth™, Wi-Fi Passthrough, or the Infrared Data Association (IrDA) standard) or a second network 499 (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN)). These various types of communication modules can be implemented as a single component (e.g., a single IC) or as multiple separate components (e.g., multiple ICs). The wireless communication module 492 can use user information (e.g., International Mobile Subscriber Identity (IMSI)) stored in the user identification module 496 to identify and verify the electronic device 401 in the communication network (e.g., the first network 498 or the second network 499).

[0085] Antenna module 497 can transmit or receive signals or power to or from the external electronic device 401 (e.g., an external electronic device). Antenna module 497 may include one or more antennas, and at least one antenna can be selected by communication module 490 (e.g., wireless communication module 492) from which a communication scheme suitable for use in a communication network (e.g., a first network 498 or a second network 499) can be selected. Signals or power can then be transmitted or received between communication module 490 and the external electronic device via the selected at least one antenna.

[0086] At least some of the aforementioned components can be coupled to each other and transmit signals (e.g., commands or data) therebetween via inter-peripheral communication schemes (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0087] Commands or data can be sent or received between electronic device 401 and external electronic device 404 via server 408 coupled to a second network 499. Each of the external electronic devices 402 and 404 can be a device of the same or different type as electronic device 401. All or some of the operations to be performed on electronic device 401 can be performed on one or more external electronic devices 402, 404, or server 408. For example, if electronic device 401 is required to automatically, or in response to a request from a user or another device, perform a function or service, instead of or in addition to performing that function or service, electronic device 401 can request one or more external electronic devices to perform at least a portion of the function or service. The one or more external electronic devices receiving the request can perform at least a portion of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to electronic device 401. Electronic device 401 can provide the result as at least part of a response to the request, with or without further processing of the result. For this purpose, cloud computing, distributed computing, or client-server computing technologies can be used, for example.

[0088] One embodiment may be implemented as software (e.g., program 440) including one or more instructions stored in a storage medium (e.g., internal memory 436 or external memory 438) readable by a machine (e.g., electronic device 401). For example, a processor of electronic device 401 may, under the control of the processor, invoke and execute at least one of the one or more instructions stored in the storage medium, with or without one or more other components. Thus, the machine can operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is a tangible means and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between semi-permanent storage of data in the storage medium and temporary storage of data in the storage medium.

[0089] According to one embodiment, the methods disclosed herein may include and be provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed online (e.g., downloaded or uploaded) via an app store (e.g., Play Store™), or distributed directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (e.g., the memory of a manufacturer's server, the memory of an app store's server, or the memory of a relay server).

[0090] According to one embodiment, each of the above components (e.g., a module or a program) may include a single entity or multiple entities. One or more of the above components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component can still perform one or more functions of each of the multiple components in the same or similar manner as one or more functions were performed by a corresponding component of the multiple components prior to integration. Operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be run in a different order or omitted, or one or more other operations may be added.

[0091] While certain embodiments of this disclosure have been described in the detailed description thereof, this disclosure may be modified in various forms without departing from its scope. Therefore, the scope of this disclosure should not be determined solely based on the described embodiments, but rather on the appended claims and their equivalents.

Claims

1. A method for performing federated learning on a client device, the method comprising: The client device receives the global model from the server; Perform model inversion on the global model to generate synthetic data for one or more categories of data in the global model, wherein performing model inversion includes performing at least one of image distillation and zero-sample data generation (ZSDG) on the global model; The collected data in the client device's dataset is augmented with synthetic data to generate an augmented dataset that includes both the collected data and synthetic data, wherein the augmented dataset has a more uniform distribution of data across categories than the original dataset. Train a global model on an augmented dataset to generate client-side models; and Send the client model from the client device to the server.

2. The method of claim 1, wherein, The execution model inversion includes: Generate synthetic data for all categories of data used in the global model; or When there is insufficient collected data, synthetic data of one or more categories is generated for use in the global model.

3. The method of claim 1, further comprising: The client device receives an updated global model from the server, wherein the updated global model is generated by averaging multiple client models, including the client model sent, on the server.

4. The method of claim 1, wherein, When the category distribution of the augmented data approaches a discrete uniform distribution, a more uniform distribution of the data is achieved.

5. A method for performing federated learning on a server, the method comprising: Distribute the global model from the server to multiple client devices; Receive client models from multiple client devices, wherein each client model is received from a corresponding client device among the multiple client devices; Model inversion is performed for each client model to generate synthetic data for each client model, wherein the synthetic data has a data distribution across the categories of the global model to compensate for insufficient collected data in the categories of the client models, and wherein performing model inversion includes performing at least one of image distillation and zero-sample data generation (ZSDG) on the global model. Generate synthetic datasets using synthetic data from client-side models; Average the client-side model to generate an average model; and The average model is trained using a synthetic dataset to generate an updated global model.

6. The method of claim 5, wherein, Each client model is generated by training a global model on the collected data of the respective client device.

7. The method of claim 6, wherein, Regularization is applied to the training of the global model on each client device so that the statistics of each client model are within a specific range of the statistics of the global model.

8. The method of claim 5, wherein, Synthetic data can have a uniform distribution across categories in the global model, or a skewed distribution across categories in the global model.

9. The method of claim 5, further comprising augmenting the synthetic dataset with data generated from a previous training period.

10. A client device for performing federated learning, the client device comprising: processor; and A non-transitory computer-readable storage medium storing instructions that, when executed, cause the processor to: Receive the global model from the server; Perform model inversion on the global model to generate synthetic data for one or more categories of data in the global model, wherein performing model inversion includes performing at least one of image distillation and zero-sample data generation (ZSDG) on the global model; The collected data in the client device's dataset is augmented with synthetic data to generate an augmented dataset that includes both the collected data and synthetic data, wherein the augmented dataset has a more uniform distribution of data across categories than the original dataset. Train a global model on an augmented dataset to generate client-side models; and Send the client model to the server.

11. The client device of claim 10, wherein, When performing model inversion, the instructions cause the processor to: Generate synthetic data for all categories of data used in the global model; or When there is insufficient collected data, synthetic data of one or more categories is generated for use in the global model.

12. The client device of claim 10, wherein, The instructions further cause the processor to: The updated global model is received from the server, whereby the updated global model is generated by averaging multiple client models, including the client models sent, on the server.

13. The client device of claim 10, wherein, When the category distribution of the augmented data approaches a discrete uniform distribution, a more uniform distribution of the data is achieved.

14. A server for performing federated learning, the server comprising: processor; and A non-transitory computer-readable storage medium storing instructions that, when executed, cause the processor to: Distribute the global model to multiple client devices; Receive client models from multiple client devices, wherein each client model is received from a corresponding client device among the multiple client devices; Model inversion is performed for each client model to generate synthetic data for each client model, wherein the synthetic data has a data distribution across the categories of the global model to compensate for insufficient collected data in the categories of the client models, and wherein performing model inversion includes performing at least one of image distillation and zero-sample data generation (ZSDG) on the global model. Generate synthetic datasets using synthetic data from client-side models; Average the client-side model to generate an average model; and The average model is trained using a synthetic dataset to generate an updated global model.

15. The server of claim 14, wherein, Each client model is generated by training a global model on the collected data of the respective client device.

16. The server of claim 15, wherein, Regularization is applied to the training of the global model on each client device so that the statistics of each client model are within a specific range of the statistics of the global model.

17. The server of claim 14, wherein, Synthetic data can have a uniform distribution across categories in the global model, or a skewed distribution across categories in the global model.

18. The server of claim 14, wherein, The instructions further enable the processor to augment the synthetic dataset with data generated from previous training periods.

Images