Federated learning methods and apparatus for client heterogeneity and data heterogeneity scenarios

Abstract

The application discloses a federated learning method and device for client isomerism and data isomerism scenes. Wherein, the method comprises: a central server sends a current global model to each selected target client; the local training capacity of each target client is calculated; according to the training capacity of each target client, the corresponding local training round of each target client is automatically determined; according to the local training round, each target client updates the local model based on the introduced optimization loss function to obtain the updated model parameters; the central server performs the aggregation update of the global model and starts the next round of training. When the client performs local training, the training round is automatically selected through the training capacity of different clients, and the data offset problem caused by Non-IID data is weakened through optimization of the objective function, which reduces the system heterogeneity and data heterogeneity in federated learning and improves the training efficiency of the model.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a federated learning method and apparatus for client heterogeneity and data heterogeneity scenarios. Background Technology

[0002] With the rapid development of information technology, the widespread use of hardware and data has provided a vast application space and conditions for the development of artificial intelligence. In an environment that utilizes data for machine learning modeling, enterprises and individuals possess unprecedented computing power and data application advantages. Every enterprise possesses sensitive data: trade secrets, intellectual property, key business information, business partner information, and customer information. However, in most industries, data exists in silos. Due to issues such as industry competition, privacy and security, and complex administrative procedures, even data integration between different departments within the same company faces numerous obstacles. In reality, integrating data scattered across various locations and institutions is almost impossible, or at least extremely costly.

[0003] On the other hand, with the further development of big data, prioritizing data privacy and security has become a global trend, and privacy computing has emerged as a major new industry worldwide. How to design a machine learning framework that allows artificial intelligence systems to more efficiently and accurately share their respective data while meeting data privacy, security, and regulatory requirements is a crucial issue in the development of artificial intelligence. Federated learning, as an important privacy protection solution, has also experienced rapid development in recent years. Federated learning involves participating parties jointly building models using data from other parties during the machine learning process. Parties do not need to share local data resources; that is, data can be jointly trained and a shared machine learning model can be established without the data leaving their local locations.

[0004] Federated learning has garnered widespread attention since its inception and has been applied in various scenarios. It solves the problem of data aggregation, enabling the design and training of machine learning models and algorithms across institutions and departments. In particular, federated learning has demonstrated good performance and robustness for machine learning model applications on mobile devices. Furthermore, for users (clients) who lack sufficient private data to develop accurate local models, federated learning can significantly improve the performance of machine learning models and algorithms.

[0005] However, because federated learning focuses on obtaining a high-quality global model through distributed learning of local data from all participating clients (devices), it fails to capture the personal information of each device, leading to performance degradation in inference or classification. Furthermore, traditional federated learning requires all participating devices to agree on a common model for collaborative training, which is impractical in complex real-world IoT applications. The problems faced by federated learning in practical applications generally fall into the following categories:

[0006] (1) System heterogeneity: Due to changes in hardware (CPU, memory), network connectivity (3G, 4G, 5G, wifi) and power supply (battery power), the storage, computing and communication capabilities of each device in the federated network may be different, and synchronous training of different clients may have a negative impact on the training results.

[0007] (2) Data heterogeneity: The data heterogeneity problem caused by the non-independent and identically distributed (Non-IID) local data of each client (device); client drift caused by Non-IID data has a significant negative impact on FedAvg, such as Figure 1 As shown. Under IID settings, the globally optimal value w * Approaching a local optimum w1 * and w2 * Therefore, the average model is also close to the global optimum. However, in the Non-IID setting, due to the global optimum w * Far from local optimum w1 * Therefore, the average model is also far from the global optimum.

[0008] (3) The heterogeneity of models required by each client according to its application scenario.

[0009] Therefore, system heterogeneity and data heterogeneity in federated learning have become serious bottlenecks in federated learning training. Summary of the Invention

[0010] To address the problems in the prior art, this invention provides a federated learning method and apparatus for client-side and data-side heterogeneous scenarios, thereby mitigating system and data heterogeneity in federated learning and obtaining high-quality personalized models for each device.

[0011] In a first aspect, embodiments of the present invention provide a federated learning method for client-side heterogeneous and data-side heterogeneous scenarios, comprising the following in each training round:

[0012] S1. The central server sends the current global model to each selected target client;

[0013] S2. Calculate the local training capability of each target client;

[0014] S3. Based on the training capability of each target client, automatically determine the local training rounds corresponding to each target client;

[0015] S4. According to the local training rounds, each target client updates its local model based on the loss function optimized by ElasticNet to obtain the updated model parameters.

[0016] S5. Each target client sends the updated model parameters to the central server, so that the central server can perform global model aggregation and update and start the next round of training until the model convergence condition is met.

[0017] Optionally, S2 includes:

[0018] Calculate the ratio of the total loss of each training sample to the standard deviation of the current probability vector corresponding to each target client;

[0019] The training capability of each target client is determined based on the difference between the current training ratio and the previous training ratio.

[0020] Optionally, S3 includes:

[0021] Based on the selection function, the local training rounds for each target client are automatically determined according to the training capabilities of each target client and the average training capabilities of all target clients.

[0022] Optionally, the selection function is e ~ α*N(e begin ),

[0023] The rules for adjusting local training rounds are as follows:

[0024] when hour,

[0025] The MaxMinNormalization function will Mapped to the interval [1,5], init represents the basic unit of step size for each training round.

[0026] Optionally, the loss function introduced with ElasticNet optimization is:

[0027]

[0028] Among them, l(w,D) k ) is an optimized function for the original FedAvg client. For newly added near-end items, This represents the local model of the k-th target client in the t-th round, where w represents the global model in this round, and α represents the model in this round. The weight.

[0029] Optionally, in S1, the central server selects the target client for this round of federated training from all clients with a certain probability.

[0030] Optionally, in step S3, the central server performs a global model aggregation update, including:

[0031] The central processing unit performs a weighted average of the updated model parameters received from each target client to obtain the updated global model parameters.

[0032] Secondly, embodiments of the present invention also provide a federated learning device for client-side heterogeneous and data-heterogeneous scenarios, comprising:

[0033] The global model sending module is used to send the current global model to each selected target client via a central server;

[0034] The local training capability determination module is used to calculate the local training capability of each target client.

[0035] The local training round determination module is used to automatically determine the local training round corresponding to each target client based on the training capability of each target client.

[0036] The local model optimization and update module is used to update the local model for each target client based on the loss function optimized by ElasticNet according to the local training rounds, so as to obtain the updated model parameters.

[0037] The global aggregation update module is used to send the updated model parameters to the central server through each target client, so that the central server can perform global model aggregation update and start the next round of training until the model convergence condition is met.

[0038] This invention provides a personalized federated learning method (pFedEN) for heterogeneous client and data scenarios. By dynamically selecting the training rounds based on the local training capabilities of each target client, the communication volume is reduced to a certain extent, thereby accelerating the entire training process. In addition, when the client performs local training, a regularization term ElasticNet is added to the client's optimization objective function to mitigate the data bias problem caused by Non-IID data and eliminate instability during the training process. Attached Figure Description

[0039] Figure 1This is a schematic diagram of data drift under Non-IID in the prior art;

[0040] Figure 2 A schematic diagram illustrating the training effect of using the FedAvg algorithm on non-independent and identically distributed data;

[0041] Figure 3 This is a main module diagram of a federated learning method for client heterogeneous and data heterogeneous scenarios in an embodiment of the present invention;

[0042] Figure 4 This is a flowchart of a federated learning method for client heterogeneity and data heterogeneity scenarios in an embodiment of the present invention;

[0043] Figure 5a The figure shows the experimental results of a comparative experiment on four algorithms, FedAvg, FedNova, FedProx, and pFedEN, using the CIFAR-10 dataset with Partition=1 in this embodiment of the invention.

[0044] Figure 5b The figure shows the experimental results of a comparative experiment on four algorithms, FedAvg, FedNova, FedProx, and pFedEN, using the CIFAR-10 dataset with Partition=2 in this embodiment of the invention.

[0045] Figure 5c This is a schematic diagram of the experimental results of a comparative experiment on four algorithms, FedAvg, FedNova, FedProx, and pFedEN, using the CIFAR-10 dataset with Partition=3 in this embodiment of the invention.

[0046] Figure 6 This is a schematic diagram showing the experimental results of comparing four algorithms—FedAvg, FedNova, FedProx, and pFedEN—on the MNIST dataset in this embodiment of the invention with Partition=1, Partition=2, and Partition=3 respectively. Detailed Implementation

[0047] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0048] Traditional federated learning assumes that the system data is uniformly and independently distributed. However, when federated learning is applied to real-world scenarios, most node data is not independent and identically distributed, making data heterogeneity one of the most serious bottlenecks in federated learning.

[0049] The goal of traditional federated learning algorithms is to process data collected from multiple devices and a central server to tune the global model. Specifically, assuming there are N clients, the optimization function is:

[0050]

[0051] Where N is the number of clients, p k ≥0 and ∑ k p k =1, generally speaking, the local loss function of the client can be defined as:

[0052]

[0053] Among them, D k Data sets representing different distributions all have n on device k. k One sample, therefore, can be set n = ∑ k n k This represents the total number of all data.

[0054] The limitations of traditional algorithms are obvious. In heterogeneous environments where user data distributions are not entirely uniform, a global model obtained by minimizing the average loss may perform poorly when applied to each user's local dataset. To address the challenges of data heterogeneity, models need to be personalized. pFedEN provides each client with its own model, and during training, the client combines the local and global models for overall updates. This ensures that even with insufficient overall data, clients with significant data skew can adapt well to new labeled data.

[0055] In the classic federated learning algorithm FedAvg, when device k is trained locally, the objective function to be minimized is the sum of losses across all samples. This is to ensure the global model performs better on the local dataset. However, if the data across devices is not independent and identically distributed (i.i.d.), the model optimized by each client will differ significantly from the initial global model assigned by the server. This will cause the local model to deviate from the initial global model, slowing down the convergence of the global model. For the results of training on non-independent and identically distributed data using the FedAvg algorithm, see [link to FedAvg algorithm]. Figure 2 .

[0056] pass Figure 2As can be seen, with identically distributed (i.e., independent) data, FedAvg converges faster than Stochastic Gradient Descent (SGD) in machine learning. However, this comes at a cost: when the computational cost per client node (measured in epochs) is the same, FedAvg's convergence speed is slower than traditional SGD. Furthermore, it can be observed that when the dataset is the same, the greater the heterogeneity of the data across nodes in the same training epochs, the more significant the difference. Figure 2 As shown, the number after niid represents the data type of each client. The smaller the number, the greater the heterogeneity of the data, and the worse the performance of FedAvg. Therefore, the technical solution in this embodiment will improve the FedAvg algorithm by addressing data heterogeneity.

[0057] This invention studies the interaction between the aforementioned data heterogeneity and system heterogeneity, and argues that the laggard problem caused by system heterogeneity and the biased parameter information sent by laggards to the central server will further increase data heterogeneity, thereby aggravating the data drift phenomenon and ultimately affecting the convergence process.

[0058] To address this issue, embodiments of the present invention provide a federated learning method (pFedEN) for scenarios with heterogeneous clients and heterogeneous data. For example... Figure 3 As shown, pFedEN mainly consists of three parts: a local training capability determination module, a local training epoch determination module, and a local model optimization and update module. The local training capability determination module defines the training capability of the current client node and automatically determines the training epochs for this training, providing clear guidance for the method's operation. The local training epoch determination module automatically adjusts the training epochs based on the local training capabilities of different clients, reducing communication overhead while ensuring model accuracy. The local model optimization and update module optimizes the local update scheme to mitigate the data bias problem caused by Non-IID data and eliminate instability during training.

[0059] The embodiments of the present invention fully consider the training capabilities of different clients, balance the training characteristics of each client node through the concerted efforts of the three main modules, and take into account the client offset problem caused by data heterogeneity, thereby improving training efficiency and local training accuracy.

[0060] See details Figure 4 The method includes:

[0061] S1. The central server sends the current global model to each selected target client.

[0062] S2. Calculate the local training capability of each target client.

[0063] Existing technologies typically employ a method of performing the same training rounds on all client nodes. However, since each client node has different amounts of data and computing power, simply relying on the output of the neural network's probability vector to guide the training scheme is incomplete, neglecting the contribution of each client node to the training rounds.

[0064] Therefore, in this embodiment, the total sample loss Global for each training session is used. i and the standard deviation Vpred of the current probability vector of each client node i As a standard for characterizing the current training capability of client nodes, ESI is used to constrain the relationship between the two, defined as follows:

[0065]

[0066] In practical applications, the previous ESI value is defined as:

[0067]

[0068] The ability to dynamically adjust the number of epochs (training rounds) is represented by the difference between the current ESI and past ESI.

[0069]

[0070] In this embodiment, ESI can improve the training efficiency of the model in the following two ways:

[0071] 1) It can reflect the client node's ability to select the training epoch during initial training. During initial training, each client receives the initial model from the central server. After the initial epoch iterations on its local machine, the resulting ESI metric indicates the degree of imbalance in the client's computing power.

[0072] 2) It can reflect the ability of client nodes to dynamically adjust epochs during training. Throughout the federated learning training process, the model becomes more and more accurate, so the overall Globali will become smaller and smaller, while Vpredi will become larger and larger (the personalized model prediction is more accurate), and ESI will continue to decrease. Therefore, ESI can be used as a parameter standard for dynamically adjusting epochs.

[0073] S3. Based on the training capability of each target client, automatically determine the local training rounds corresponding to each target client.

[0074] In this embodiment, considering The significance of the training epoch for the model can be indicated by designing a selection function that dynamically changes the epoch.

[0075] Specifically, the selection function is e ~ α*N(e begin The input is a default epoch value. The dynamic epoch adjustment mechanism is as follows:

[0076] condition:

[0077] renew:

[0078] When the condition formula is met, the epoch of the client node is dynamically adjusted. During the adjustment process, a selection function is used to ensure that the epoch in each round is adjusted according to the client node's state. For example, when... When this happens, it indicates that the current client's training capability is worse than other nodes. The node will increase the local training epochs to reduce training bias, and at the same time, the node will adjust the training frequency according to the MaxMinNormalization function. Mapped to the interval [1,5], init represents the basic unit of step size for each epoch adjustment (default is 5).

[0079] For federated learning training, this strategy reduces communication volume to some extent by increasing the number of local iterations, thereby speeding up the entire training process in epochs.

[0080] S4. Based on the local training rounds, each target client updates its local model using a loss function optimized with ElasticNet, to obtain the updated model parameters.

[0081] Specifically, the loss function introduced with ElasticNet optimization is:

[0082]

[0083] Among them, l(w,D) k ) is an optimized function for the original FedAvg client. For newly added near-end items, This represents the local model of the k-th target client in the t-th round, where w represents the global model in this round, and α represents the model in this round. The weight.

[0084] This invention adds a regularization term, ElasticNet, to the client's optimization objective function. ElasticNet combines Lasso and Ridge into a single model with two penalty factors: one proportional to the L1 norm and the other proportional to the L2 norm. The model obtained using this method is as sparse as pure Lasso regression, but at the same time has the same regularization capability as Ridge regression. This method can make the optimization method more stable, and ultimately make pFedEN converge faster even under data heterogeneity.

[0085] For example, the local data in this client can be specific data information involved in autonomous driving of the vehicle. By using the method in the embodiments of the present invention to perform federated learning on this data, an accurate autonomous driving model can be trained.

[0086] S5. Each target client sends the updated model parameters to the central server, so that the central server can perform global model aggregation and update and start the next round of training until the model convergence condition is met.

[0087] In this embodiment, pFedEN operates on the central server in the same way as FedAvg, both employing a weighted average global model update strategy. However, to address the data heterogeneity issue, the k-th node no longer performs E-round SGD, but instead solves the following optimization problem involving ElasticNet:

[0088]

[0089] The pFedEN implementation in this embodiment incorporates ElasticNet updates by combining the advantages of L1 and L2 norms. Regularization is added to allow each client to perform personalized learning during local training and updates. While L1 norm is introduced to avoid overfitting, it can lead to a sparse model. Simultaneously introducing L2 norm eliminates the sparsity problem caused by L1 norm, ensuring model stability. Therefore, pFedEN, after using the new optimization function, can perform both regularization and variable selection, resulting in a better local model for the client.

[0090] More specifically, the federated learning method for client-side heterogeneous and data-heterogeneous scenarios provided in this embodiment includes the following steps:

[0091] Input: w 0 Randomly initialize parameters, where N is the total number of devices;

[0092] Output: The final global model parameters w t+1 .

[0093] (1) FOR the global round number t=0,1,…,T-1;

[0094] (2) The central server Server with probability p k Randomly select K devices and specify a fixed learning rate;

[0095] (3) The central server (Server) sends the current global model w * Give the selected device;

[0096] (4) Each client uses local data for training and performs ESI before sending the data to the cloud. p The operation with ESI yields Used to calculate epoch values;

[0097] (5) For each device k = 1, 2, ..., K, parallel calculations are performed based on the calculated epoch values:

[0098] (6) Each client updates its local model based on the new loss optimization function.

[0099]

[0100] (7) Repeat steps 4)-6), end the parallel computing, and each device will output the computing results. Transmitted to the central server;

[0101] (8) Model update: The central processing unit updates the data received from K clients. We obtain w by updating the model parameters using a weighted average. t+1 ;

[0102] (9) The central server will update the model parameters w t+1 Send to the client;

[0103] (10) Repeat 2)-9)t times;

[0104] (11) End.

[0105] Furthermore, the present invention provides experimental verification of the proposed pFedEN, the details of which are as follows:

[0106] 1. Experimental setup

[0107] Test platform setup: pFedEN was deployed on 5 TITAN XP GPUs, one of which simulated a central cloud server, and the other GPUs simulated edge computing devices.

[0108] Model and Datasets: To investigate the effectiveness of pFedEN on Non-IID data settings, this embodiment conducted extensive experiments on three image datasets (MNIST, CIFAR-10, SVHN), and the statistical data of the datasets are summarized in Table 1 below.

[0109] Table 1 Data Statistics

[0110] Dataset Training Examples Test Instance Feature number Number of categories MNIST 60000 10000 784 10 CIFAR-10 60000 10000 784 10 SVHN 73257 26032 1024 10

[0111] For the image dataset, this embodiment uses three classic CNN models to train the image classification task: RESNET-20, RESNET-50, and VGG16. Among them, RESNET-20 and RESNET-50 are computationally intensive networks, while VGG16 is a communication-intensive network.

[0112] Benchmark Metric: Top-1 accuracy on the test dataset was used as the metric for evaluating the studied algorithms. All studied algorithms were run for the same number of rounds for fair comparison. Unless otherwise specified, the number of rounds was set to 50 by default. Comparative experiments were conducted against typical federated learning algorithms, including FedAvg (baseline), FedProx, and FedNova. Training was performed on different datasets, different models, and datasets with varying degrees of heterogeneity, fully validating the versatility and superiority of the pFedEN algorithm. Here, Partition = k indicates that each client has only k distinct data samples; a smaller k indicates a greater degree of data heterogeneity between clients.

[0113] Experimental results:

[0114] 1) Comparative experiments were conducted on the CIFAR-10 dataset using four algorithms: FedAvg, FedNova, FedProx, and pFedEN, with partitions of 1, 2, and 3. Each training round consisted of 10 epochs. The experimental results are as follows: Figure 5a -c is shown.

[0115] 2) The MNIST dataset was tested with Partition=1, Partition=2, and Partition=3, with 10, 20, 40, and 80 training rounds respectively. Four algorithms—FedAvg, FedNova, FedProx, and pFedEN—were applied for comparative experiments. The results are as follows: Figure 6 As shown, the four bar charts for each training round correspond to the experimental results of the four algorithms FedAvg, pFedEN, FedNova, and FedProx, from left to right.

[0116] 3) Simultaneously record the overall average algorithm accuracy of the three datasets under different configurations with the introduction of a strategy to dynamically adjust the training rounds, as shown in Table 2 below.

[0117] Table 2. Average accuracy of different algorithms with varying degrees of data heterogeneity.

[0118] Dataset Data heterogeneity FedAvg FedProx FedNova PFedEN MNIST Partiton = 1 29.8％±7.9％ 39.8％±22.2％ 39.2％±22.1％ 40.9％±23.1％ Partiton = 2 95.8％±0.3％ 96.4％±0.3％ 97.4％±0.5％ 97.0％±0.4％ Partiton = 3 97.4％±04 97.9％±0.2％ 96.1％±0.6％ 98.0％±0.5％ IID 98.9％±0.1％ 98.9％±0.1％ 98.9％±0.1％ 98.9％±0.1％ CIFAR-10 Partiton = 1 10.0％±0.0％ 12.3％±0.2％ 10.0％±0.0％ 12.0％±0.0％ Partiton = 2 48.8％±3.3％ 49.8％±2.1％ 46.5％±3.5％ 51.2％±1.2％ Partiton = 3 58.3％±12％ 57.6％±1.3％ 57.1％±14％ 62.5％±2.4％ IID 70.4％±0.5％ 70.2％±0.2％ 69.5％±1.0％ 70.5％±0.3％ SVHN Partiton = 1 11.1％±0.0％ 18.7％±0.0％ 10.6％±0.8％ 18.6％±0.0％ Partiton = 2 78.4％±0.9％ 80.2％±0.7％ 754％±4.8％ 81.1％±0.4％ Partiton = 3 82.0％±0.7％ 82.1％±1.0％ 80.5％±1.2％ 82.2％±0.3％ IID 88.5％±0.8％ 88.5％±1.0％ 88.4％±0.5％ 88.6％±0.3％

[0119] 2. Analysis of Experimental Results

[0120] 1) When the data heterogeneity is 1, 2, and 3, the CIFAR-10 dataset is trained using FedAvg, FedNova, FedProx, and pFedEN respectively. It can be seen that pFedEN can achieve higher training accuracy than advanced or classic federated learning algorithms.

[0121] 2) When training MNIST with VGG16, pFedEN improved the training accuracy compared with the classic FedAvg, FedNova and FedProx algorithms under different data heterogeneity levels of 1, 2 and 3 and different training rounds of 10, 20, 40 and 80, demonstrating the strong applicability of pFedEN.

[0122] 3) In combination with three datasets, four different algorithms, and environments with independent and identically distributed data (Partiton-1, 2, 3), pFedEN showed higher training accuracy in most cases compared to FedAvg, FedNova, and FedProx.

[0123] Example 1

[0124] In increasingly competitive industries such as food and beverage, daily necessities, and cosmetics, maintaining a competitive edge, effectively reaching target audiences, and achieving high conversion rates while protecting data security are the most pressing concerns for advertisers. Advertisers with some technical capabilities often face data security challenges when developing audience targeting strategies. To implement more personalized strategies, advertisers typically need to upload their own data to media platform tools for insight and analysis, but industry-specific considerations or data security concerns often prevent them from doing so.

[0125] To maximize traffic value and conversion efficiency, media platforms have launched their own federated learning frameworks in recent years, collaborating with advertisers to explore data value through AI joint modeling. Federated learning primarily addresses the issue of cross-domain data, ensuring data remains within its own domain. This means concentrating computing resources towards data resources, guaranteeing data security, privacy, and compliance. In a typical AI joint modeling scenario, data participants usually include media platforms, advertisers, and third-party service providers. The media platform provides the basic framework and data source feature library for federated learning, such as various consumer behaviors and interests on media platforms. The client provides a client-side sample feature library, such as the large amount of basic retail customer information and rich product and content information accumulated by retail companies. Third-party service providers may provide sample feature libraries based on other data sources. By introducing pFedEN, personalized models can be built for each target group to enable precise ad delivery and improve advertising revenue.

[0126] Example 2

[0127] The development of vehicle-to-everything (V2X) technology is a crucial application within the 5G Internet of Things (IoT) field, especially with the deployment of autonomous driving, which has accelerated the process of connecting in-vehicle systems to the cloud. Statistics show that a single autonomous vehicle generates over 4TB of data daily. However, autonomous driving involves the willingness of users to have their driving data collected and uploaded to servers, and often this is not the case. Without data uploads, autonomous driving cannot develop optimal models. Federated learning helps break down data silos in vehicle-road cooperation, establishing efficient local V2X networks and reducing the risk of privacy leaks related to in-vehicle data. Federated learning allows all user data to remain local, providing users with only one model for feedback. This enables collaborative modeling, resulting in a better autonomous driving model while protecting user data privacy. With increasingly stringent data regulations in the future, federated learning offers a solution for training autonomous driving models.

[0128] The inherent heterogeneity of devices, statistics, and models in complex connected vehicle environments poses a significant challenge to traditional federated learning. Applying the pFedEN algorithm can alleviate the inherent device heterogeneity, statistical heterogeneity, and model heterogeneity problems in connected vehicle applications as a whole and improve model accuracy, achieving a "win-win" situation for data protection and model training.

[0129] Furthermore, embodiments of the present invention also provide a federated learning device for client-side heterogeneous and data-side heterogeneous scenarios, comprising:

[0130] The global model sending module is used to send the current global model to each selected target client via a central server;

[0131] The local training capability determination module is used to calculate the local training capability of each target client.

[0132] The local training round determination module is used to automatically determine the local training round corresponding to each target client based on the training capability of each target client.

[0133] The local model optimization and update module is used to update the local model for each target client based on the loss function optimized by ElasticNet according to the local training rounds, so as to obtain the updated model parameters.

[0134] The global aggregation update module is used to send the updated model parameters to the central server through each target client, so that the central server can perform global model aggregation update and start the next round of training until the model convergence condition is met.

[0135] Specifically, the local training capability determination module is used for:

[0136] Calculate the ratio of the total loss of each training sample to the standard deviation of the current probability vector corresponding to each target client;

[0137] The training capability of each target client is determined based on the difference between the current training ratio and the previous training ratio.

[0138] The local training round determination module is specifically used to: automatically determine the local training round for each target client based on the selection function, the training capability of each target client, and the average training capability of all target clients.

[0139] Wherein, the selection function is e~α*N(e begin ),

[0140] The rules for adjusting local training rounds are as follows:

[0141] when hour,

[0142] The MaxMinNormalization function will Mapped to the interval [1,5], init represents the basic unit of step size for each training round.

[0143] The loss function introduced by ElasticNet optimization is as follows:

[0144]

[0145] Among them, l(w,D) k ) is an optimized function for the original FedAvg client. For newly added near-end items, This represents the local model of the k-th target client in the t-th round, where w represents the global model in this round, and α represents the model in this round. The weight.

[0146] Specifically, the central server selects the target client for this round of federated training from all clients with a certain probability.

[0147] The central server performs global model aggregation updates, including: the central processor performs a weighted average of the updated model parameters received from each target client to obtain the updated global model parameters.

[0148] The federated learning device for client heterogeneous and data heterogeneous scenarios provided in this embodiment of the invention can execute the federated learning method for client heterogeneous and data heterogeneous scenarios provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0149] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A federated learning method for client-side heterogeneous and data-heterogeneous scenarios, characterized in that, Each training round includes: S1. The central server sends the current global model to each selected target client; S2. Calculate the local training capability of each target client; S3. Based on the training capability of each target client, automatically determine the local training rounds corresponding to each target client; S4. According to the local training rounds, each target client updates its local model based on the loss function optimized by ElasticNet to obtain the updated model parameters. S5. Each target client sends the updated model parameters to the central server, so that the central server can perform global model aggregation and update and start the next round of training until the model convergence condition is met. The S3 includes: automatically determining the local training rounds corresponding to each target client based on the selection function, according to the training capability of each target client and the average training capability of all target clients; The selection function is: , The rules for adjusting local training rounds are as follows: when hour, , in, This represents the initial training rounds. The effective coefficient is set. This represents the difference between the current ESI and past ESI values. The average ESI for all target clients. The function will Mapped to the interval [1, 5] This represents the basic unit of step size for each training round.

2. The method according to claim 1, characterized in that, S2 includes: Calculate the ratio of the total loss of each training sample to the standard deviation of the current probability vector corresponding to each target client, ESI; The training capability of each target client is determined based on the difference between the current training ratio and the previous training ratio.

3. The method according to claim 1, characterized in that, The loss function introduced with ElasticNet optimization is: ； in, This is an optimized function for the original FedAvg client. For newly added near-end items, It is the local model of the k-th target client in the t-th round. Represents the global model for this round. Represents this round The weight.

4. A federated learning device for client-side heterogeneous and data-heterogeneous scenarios, characterized in that, include: The global model sending module is used to send the current global model to each selected target client via a central server; The local training capability determination module is used to calculate the local training capability of each target client. The local training round determination module is used to automatically determine the local training round corresponding to each target client based on the training capability of each target client. The local model optimization and update module is used to update the local model for each target client based on the loss function optimized by ElasticNet according to the local training rounds, so as to obtain the updated model parameters. The global aggregation update module is used to send the updated model parameters to the central server through each target client, so that the central server can perform global model aggregation update and start the next round of training until the model convergence condition is met. The local training round determination module is based on a selection function and automatically determines the local training round corresponding to each target client according to the training capability of each target client and the average training capability of all target clients. The selection function is: , The rules for adjusting local training rounds are as follows: when hour, ,in, This represents the initial training rounds. The effective coefficient is set. This represents the difference between the current ESI and past ESI values. The average ESI for all target clients. The function will Mapped to the interval [1, 5] This represents the basic unit of step size for each training round.

Images