Data heterogeneity-oriented federated learning method based on particle swarm optimization

Abstract

This invention discloses a federated learning method for heterogeneous data based on particle swarm optimization. Before training begins, the server sends a scoring dataset D to each client. s D s It is server data D m The invention utilizes independent and identically distributed subsets of the federated learning model. During communication, the client uploads its local model's historical best score (pbest) to the server. The server compares these scores to determine the best-performing client on the scoring dataset, requests the client's model parameters, and aggregates the client's local model parameters to update the global model. This invention combines particle swarm optimization with the federated learning training process, transforming the communication data format from model parameters to model scores, effectively reducing communication costs. It also proposes a model retraining strategy to guide the optimization of the federated model in a controllable manner.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of big data, and particularly relates to a data-heterogeneous-oriented federated learning method based on particle swarm optimization. BACKGROUND

[0002] With the rapid development of big data technology, machine learning is increasingly applied in scientific research and business decision-making. Data with high expressiveness can be used to build more complex and accurate machine learning models, thereby bringing people more accurate services and better decision support. However, there are still problems in data application. On the one hand, with the popularity of intelligent devices and the rise of the Internet, the amount of data generated every day is growing explosively, including various data such as photos and voice. On the other hand, traditional machine learning relies on a master server to store data and train models, but most data is scattered on various mobile terminals, resulting in great challenges in data aggregation and model training. In addition, countries around the world have introduced laws and regulations on data privacy protection. The establishment of these regulations will help protect data security, but also restrict the development of machine learning, making it difficult to mine and utilize massive data.

[0003] A federated learning system usually consists of one server and multiple clients. FedAvg algorithm is the most typical federated learning algorithm. The idea of this algorithm is to combine each client running stochastic gradient descent with the server running model average calculation. As shown in FIG. 1, in each round of training, the client downloads the global model and trains the model using its local data, and then uploads the local model parameters to the server. The server is responsible for coordinating the training of each client, and updating the global model by aggregating the local model parameters. Figure 1

[0004] Federated learning is a new distributed machine learning framework that can train a good global model without uploading data to the server. In the training process of federated learning, each client downloads the global model from the server, trains the model using local data, ensures that data does not leave the local, and then uploads the trained local model to the server for model aggregation update. Through iterative training, a good global model is obtained. Federated learning is similar to traditional distributed machine learning, where learning tasks are assigned to multiple clients to complete, but the difference is that the training data of federated learning is not sent to each client by the server, and the training data of each client is private local data. Because the data is always saved locally and will not be uploaded to the server or shared with other clients, the privacy and security of the data can be guaranteed.

[0005] ​However, federated learning still faces some challenges. First, the training process requires frequent parameter transfers between the server and clients. Currently, neural network models are most commonly used, and their large number of parameters leads to high communication costs during training. Second, in reality, the data held by different clients is often non-independent and identically distributed (Non-IID), meaning there is a data heterogeneity problem. This can cause a decrease in the accuracy of the model trained by federated learning, or even prevent training from converging. Summary of the Invention

[0006] Federated learning is an emerging privacy-preserving distributed machine learning framework. Its core feature is the ability to perform distributed machine learning without accessing the client's raw data. Clients train models using local data and then upload the model parameters to the server for aggregation, ensuring that client data is always protected. However, this process suffers from high communication costs due to frequent parameter transfers and the problem of heterogeneous data held by different clients, both of which severely restrict the application of federated learning. To address these issues, this invention proposes a particle swarm optimization-based federated learning method for heterogeneous data—FedPSG. This method transforms the data transmitted from the client to the server from model parameters to model scores. In each training round, only one client needs to upload model parameters to the server, thus reducing communication costs. Simultaneously, a model retraining strategy is proposed, using server data to perform a second iteration of training on the global model. This further improves model performance by mitigating the impact of data heterogeneity on federated learning. Experiments on the MNIST and CIFAR-10 datasets simulate different heterogeneous data environments. The results show that FedPSG effectively improves the accuracy of the model under different heterogeneous data environments, and verify that the model retraining strategy effectively solves the client data heterogeneity problem.

[0007] To achieve the above objectives, this invention proposes a federated learning method based on particle swarm optimization for heterogeneous data, comprising the following steps:

[0008] Before training begins, the server sends the scoring dataset D to each client. s D s It is server data D m Independent and identically distributed subsets;

[0009] During FedPSG communication, the client uploads the historical best score pbest of the local model to the server. After obtaining the historical best scores transmitted by each client, the server compares them to determine the client that performs best on the scoring dataset.

[0010] Request the model parameters of this client;

[0011] aggregate the local model parameter updates of the clients to update the global model.

[0012] Further, to exert the guiding effect of the global best model and the local best model on the client model training, the update of the client neural network model in FedPSG includes one round of particle update and E rounds of local iteration; the particle update process is shown in the following formula:

[0013]

[0014] w t = V t + w t-1

[0015] wherein w p represents the local best model of the client, w g represents the global best model, and w t represents the client model at the tth round of communication, represents the parameters of the lth layer of the neural network model, and the parameters of all layers constitute V t , and a represents an inertia weight, r1 and r2 are random values between 0 and 1, and c1 and c2 respectively represent the acceleration factors of w p and w g The greater c2 is, the more the model training will be biased towards the global model, and the greater c2 is, the more the model training will be biased towards the local model.

[0016] Further, to reduce the impact of the Non-IID problem on the federated learning training, a model retraining strategy is used at the server end: in each round of communication, after receiving the client model, the server re-trains the model using the server data D m

[0017]

[0018] wherein w t is the client model received by the server in the tth round of communication, η is a learning rate, and D m embodies the data distribution of the total training data of all clients.

[0019] Further, D m is an independent and identically distributed subset of the total training data D, and D m is a subset of D s , and the ratio of the data volume of D m to the data volume of D s is γ, that is,

[0020] Further, D s is an independent and identically distributed subset of D m , representing the optimization direction of the FedPSG client model, and D s is a subset of Dm The ratio of the data amount is delta, that is

[0021] The beneficial effects of the present application are as follows:

[0022] (1) Combine the particle swarm optimization algorithm with the federated learning training process, and change the communication data form of federated learning from model parameters to model scores. Compared with the traditional federated learning algorithm, FedPSG can effectively reduce the communication cost.

[0023] (2) A model retraining strategy is proposed. The server data is constructed, and the model is retrained and optimized on this basis, which not only alleviates the influence of data heterogeneity on federated learning, but also improves the accuracy of the model. In addition, this strategy can also guide the optimization of the federated model in a controllable way.

[0024] (3) Simulate different data heterogeneity conditions on MNIST and CIFAR-10 data sets, and compare and analyze the accuracy of FedPSG, FedAvg and FedShare algorithms. The experimental results prove that the model obtained by FedPSG has higher accuracy, and the model retraining strategy can effectively solve the data heterogeneity problem; Design an experiment to study the influence of server data amount on the accuracy of FedPSG model, the results prove that the larger the server data amount, the higher the accuracy of FedPSG model, and the stronger the robustness of model accuracy when the data heterogeneity condition changes. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 Traditional federated learning training process diagram;

[0026] Figure 2 The training process of the present application;

[0027] Figure 3 Accuracy of four algorithms on MNIST, IID data set;

[0028] Figure 4 Accuracy of four algorithms on CIFAR, IID data set. DETAILED DESCRIPTION

[0029] The present application will be further described below in conjunction with the drawings, but in no way limits the present application, any transformation or replacement based on the teaching of the present application belongs to the protection scope of the present application.

[0030] The present application realizes that even in the case of client data heterogeneity, the federated learning can still maintain low communication cost and train a relatively high-quality global model. Unlike the federated learning algorithm proposed in the past, FedPSG takes the client model score as the communication data, and only one client needs to send the model parameters to the server in one round of communication. In addition, the method sets a model retraining strategy at the server end, and constructs the server data D m that can reflect the overall training data distribution m , and then uses D i to perform secondary iteration training on the model. The present application simulates the case of client data heterogeneity locally, and experiments prove that FedPSG can effectively improve the accuracy of the model in different data heterogeneous environments. At the same time, the model retraining strategy is verified to effectively solve the data heterogeneity problem.

[0031] Kennedy et al. proposed the particle swarm algorithm in 1995. The algorithm is inspired by bird and fish groups in nature, can optimize multiple variables at the same time, has the advantages of easy implementation, strong scalability, strong robustness, fast convergence and simple mathematical operation. These advantages make the particle swarm algorithm require less memory when solving complex problems on a computer, but solve problems faster.

[0032] The particle swarm algorithm system is composed of a group of particles, and each particle represents a solution to the problem. Each particle has parameters: position x and velocity v. In order to find the global best value, particles communicate with each other and share their local best solutions (p i ). Each particle sets the global best solution (p g ) as the maximum value of each particle p i : p g =max(p1,p2,…,p K ), K is the size of the particle swarm. In one iteration, the update of the velocity of particle i is affected by p i and p g at the same time, and the update process is as follows:

[0033] v i =αv i +c1r1(p i -x i )+c2r2(p g -x i ) (1)

[0034] where, alpha is a constant representing the inertia weight, p i is the best solution of particle i, p g is the global best solution, c1 is the acceleration constant of p i , and c2 is the acceleration constant of p gThe acceleration constant of the particle i is: a i = -k * r i, where k is a constant, and r i is a random number between 0 and 1. The velocity v i of the particle i is obtained by: v i = v i + a i * dt i After that, the particle i updates its own position:

[0035] x i = v i * dt i + x i i (1)

[0036] In the traditional federated learning framework, the precondition of model aggregation is that the server receives a certain proportion of the model parameters uploaded by the clients. However, this process is extremely susceptible to the amount of model parameters and the network conditions of the clients. For example, if a VGG16 model is trained by federated learning, the amount of parameters that need to be transmitted by the clients in each round of training will be as high as 130 million. Even for the lightweight MobileNet series, the MobileNetV1 model still has more than 4 million parameters. In the context of needing to transmit such a large amount of parameters, once the network conditions of the clients are unstable during the federated learning communication process, the training speed of the entire federated learning will be affected. Therefore, it is necessary to reduce the amount of data transmitted during the federated learning communication process.

[0037] To reduce the communication cost and improve the robustness of the federated learning training process, the present application proposes FedPSG, which changes the form of data transmitted between the clients and the server. The specific implementation is to change the form of transmitted data to the score (accuracy rate set by the present application) of the client model instead of the model parameters with large data volume. Before the training starts, the server sends the scoring data set D s to each client, where D s is an independent and identically distributed subset of the server data D m . As shown in Fig. Figure 2 Unlike the FedAvg algorithm that transmits model parameters, in the communication process of FedPSG, the client uploads the historical best score of the local model to the server. After obtaining the scores transmitted by each client, the server compares and determines the client with the best performance on the scoring data set, and then requests to obtain the model parameters of the client. The detailed process is shown in Algorithm 1. Therefore, in one round of communication, the server under the FedPSG framework only needs to receive the model parameters of the client numbered as gid, without requiring all clients to send model parameters. This can greatly reduce the communication cost and improve the communication efficiency, and further improve the stability of the federated learning training process.

[0038] Algorithm 1 FedPSG

[0039] Input: number of clients K, number of communications T, local iteration number E, learning rate η, inertia weight α, acceleration factors c1 and c2

[0040] Output: global model w T

[0041]

[0042] The server initializes gbest, pbest, and gid, where pbest is the maximum value of the accuracy of the client model on the scoring dataset, pbest is the maximum value of the accuracy of multiple client models on the scoring dataset in each round of communication, and gid is the client number.

[0043] To play the guiding role of the global best model and the local best model in the training of the client model, the update of the client neural network model in FedPSG includes one round of particle update and E rounds of local iteration. The particle update process is shown in equations (3) and (4):

[0044]

[0045] w t = V t + w t-1 (4)

[0046] where w p represents the local best model of the client, w g represents the global best model, and w t represents the client model at the tth round of communication. represents the parameters of the lth layer of the neural network model, and the parameters of all layers constitute V t , a represents the inertia weight, r1 and r2 are random values between 0 and 1, and c1 and c2 represent the acceleration factors of w p and w g , respectively. The larger c2 is, the more the model training will be biased towards the global model, and the larger c2 is, the more the model training will be biased towards the local model.

[0047] The data of a single client cannot represent the data of all clients, especially in a Non-IID data environment. If a client model is simply set as a global model, it will lead to inconsistency between the data distribution used in the training of the global model in federated learning and the real data distribution, and further lead to a decrease in model accuracy. Consider the following federated learning scenario: the MNIST dataset is used as training data, but under the extreme condition of data non-independent and identical distribution, i.e., each client only has one type of data. In this case, the generalization effect of the local model trained by a single client is obviously poor. If a single client model is sent to each client as a global model, since the global model is actually trained on only one type of data, the global model can only improve the recognition ability of the client model for a certain type of label data, and the global model obtained through the above training process will also lack generalization ability.

[0048] To mitigate the impact of Non-IID issues on federated learning training, FedPSG designed a model retraining strategy on the server side: during each round of communication, after receiving the client's model, the server uses server data D... m Retrain the model:

[0049]

[0050] Among them, w t The client model received by the server during the t-th round of communication, where η is the learning rate and D... m It should reflect the overall data distribution of training data across all clients. In this paper, D m It is an independent and identically distributed subset of the total training data D, D m The ratio of the data volume to the D data volume is γ, that is In practice, D can be obtained through data voluntarily shared by users or by recruiting volunteers to participate in internal testing. m Typically, servers possess superior information resources and can construct D based on historical data and their own specific circumstances. m .

[0051] From another perspective, model retraining strategies offer a method to address the unfairness problem in federated learning. Server data D m It is not necessary for D to be necessarily related to the overall training data D; instead, D should be selected based on the actual purpose. m For example, D might contain some gender or wealth discrimination, but we can construct a D specifically for this purpose. m This guides the optimization of federated learning models, guiding them towards unbiasedness and fairness.

[0052] Before FedPSG training begins, the server needs to send the scoring dataset D to the client. s Because in D s Only the client with the highest accuracy can send the model to the server, so D s This actually represents the direction of FedPSG client model optimization. Because the server can construct D based on the target... m To ensure that the training objectives of the server and the client are consistent, this invention will use D s Set to D m A subset of independent and identically distributed data, D s Data volume and D m The ratio of the data volume is δ, that is

[0053] The environment used in the experiment is an Ubuntu system configured with an Intel(R) Xeon(R) Platinum 8350C CPU @ 2.60GHz processor and an NVIDIA GeForce RTX 3090 graphics card. The experiment is implemented by Python language, and the PyTorch neural network framework is used. The Python version used is 3.8, and the PyTorch version used is 1.11.

[0054] In the experiment of the application, MNIST dataset and CIFAR-10 dataset are used to evaluate FedPSG, and FedPSG is compared with FedAvg and FedShare algorithm under different data heterogeneity. At the same time, the influence experiment of server data on FedPSG is set, and the accuracy of FedPSG on MNIST dataset and CIFAR-10 dataset under different γ values is compared.

[0055] The MNIST dataset consists of 60,000 training samples and 10,000 test samples, each of which is a 28x28 pixel grayscale handwritten digit image. The CIFAR-10 dataset consists of 50,000 training samples and 10,000 test samples, each of which is a 32x32 pixel color image. Compared with the samples in the MNIST dataset, the samples in the CIFAR-10 dataset are real objects in the real world, with large noise and different features and sizes, so the recognition difficulty is greater. To simulate the data distribution environment of federated learning, this paper sets up 100 clients as training nodes, and divides the training data into IID, Non-IID(1) and Non-IID(2):

[0056] (1) IID: randomly and uniformly distribute the training data to each client. Taking the division of the MNIST dataset as an example, there are 6000 training samples for each class of data in the dataset, and the 10 classes of data in the dataset are randomly and uniformly distributed to the 100 clients, so that each client is allocated 600 data and can have data with 10 labels. Such a division method can ensure that the training dataset of each client has a certain diversity and representativeness, which is beneficial to improve the generalization ability of the model;

[0057] (2) Non-IID(1): each client only has data with one label. Taking the division of the MNIST dataset as an example, first, the dataset is divided into 10 groups according to the label, and each group of data is evenly divided into 100 data slices. Then, each client takes 10 data slices from a random group to form the client training data.

[0058] (3) Non-IID(2): Each client only has data of 2 types of labels. Taking the division of the MNIST dataset as an example, first, the dataset is divided into 10 groups according to the label, and each group of data is evenly split into 200 data slices. Then, each client takes 10 data slices from each of two random groups to form the client training data.

[0059] The experimental model uses a convolutional neural network model: two 5x5 convolutional layers, each followed by a 2x2 Max pooling layer, followed by three fully connected layers, and finally outputting a 10-dimensional vector.

[0060] For the MNIST dataset, the batch_size is set to 20, the local iteration number E is set to 10, and the learning rate η = 0.001. For the CIFAR-10 dataset, the batch_size is set to 20, the local iteration number E is set to 10, and the learning rate η = 0.01. The FedAvg algorithm selects the proportion of clients C = 1.0. FedShare is an improved algorithm of FedAvg, which mainly studies the Non-IID problem in federated learning, and takes 1000 samples from each type of data to form a global shared dataset G, and then allocates a proportion of β of the global shared dataset to each client, and β = 0.05. In FedPSG, D s The ratio of the amount of data to D m The ratio of the amount of data δ is 0.2.

[0061] To test the performance of FedPSG under the condition of client data heterogeneity, the IID, Non-IID(1) and Non-IID(2) environments are set, and the accuracy of FedPSG, FedAvg, FedShare and FedPS algorithms are compared on the MNIST and CIFAR-10 datasets. Among them, the FedPS algorithm is a simplified version of FedPSG without model retraining strategy, and the server data D m The ratio of the amount of data to the amount of client training data γ = 0.2. IID data is independent and identically distributed, and there is no client data heterogeneity. Non-IID(1) and Non-IID(2) data are non-independent and identically distributed, but Non-IID(1) data has a higher degree of heterogeneity. The experimental results are shown in Figure 3 and Table 1.

[0062] It can be observed that the accuracy of FedAvg algorithm on MNIST and CIFAR-10 datasets decreases significantly as the client data heterogeneity increases. When the data heterogeneity changes from IID to Non-IID(1), the accuracy of FedAvg algorithm on MNIST dataset decreases from 97.93% to 40.20%, and the accuracy on CIFAR-10 dataset decreases from 52.28% to 16.39%. It can be seen that after 100 rounds of training, the accuracy of FedAvg under Non-IID(1) data condition is less than half of that under IID data condition, which verifies that the client data heterogeneity has a great influence on the training of federated learning.

[0063] Table 1 Accuracy of FedPSG, FedShare, FedAvg and FedPS on CIFAR-10 and MNIST datasets after 100 rounds of training under three client data heterogeneity conditions (%)

[0064]

[0065] Compared with FedAvg, FedShare and FedPSG perform more stably under Non-IID data condition. Especially on MNIST dataset, when the data heterogeneity changes from IID to Non-IID(1) or Non-IID(2), the decrease of accuracy of FedShare and FedPSG is not more than 2%. However, the accuracy of FedPSG on CIFAR-10 dataset is significantly higher than that of FedShare, and as the client data heterogeneity increases, the decrease of accuracy of FedPSG is lower than that of FedShare. It can be seen that the FedPSG proposed in the present application is effective in dealing with data heterogeneity, and the effect is better than that of general improved algorithm.

[0066] As Figure 3As shown, the accuracy of FedPS on the MNIST and CIFAR-10 datasets under different data heterogeneous conditions is smaller than that of FedPSG, and even smaller than that of FedAvg. And with the increase of the degree of client data heterogeneity, the accuracy of FedPS on the MNIST and CIFAR-10 datasets decreases significantly. When the data heterogeneity changes from IID to Non-IID (1), as shown in Table 1, the accuracy of FedPS on the MNIST dataset decreases from 92.27% to 9.8%, and the accuracy on the CIFAR-10 dataset decreases from 37.2% to 10.55%. Since FedPS only saves the model retraining strategy compared with FedPSG, and FedPSG can maintain high accuracy under different data heterogeneous conditions, it can be verified that the model retraining strategy can effectively improve the accuracy of the model, and improve the robustness of the training quality of federated learning when the data heterogeneous condition changes.

[0067] Table 2 Accuracy of FedPSG on CIFAR-10 and MNIST datasets after 100 rounds of training under three client data heterogeneous conditions (unit: %)

[0068]

[0069] The model retraining strategy helps to improve the accuracy of FedPSG under different data heterogeneous conditions. Since the data used when executing the strategy is the server data D m , in order to further explore the influence of D m on FedPSG, five γ values are set in the experiment: 0.2, 0.16, 0.1, 0.06, 0.02, and the closer the γ value is to 1, the closer the D m data volume is to the total training data D. The accuracy of FedPSG on the MNIST and CIFAR-10 datasets under different data heterogeneous conditions is compared, and the results are shown in Table 2 and Table 3. Figure 4 and Table 2.

[0070] It can be observed that under different data heterogeneous conditions, with the increase of γ value, the accuracy of FedPSG on the MNIST and CIFAR-10 datasets is constantly improved, which indicates that the increase of D m data volume helps to improve the accuracy of the model.

[0071] After 100 rounds of training, when the data heterogeneity condition changes from IID to Non-IID(1), the accuracy of the model(γ=0.20) on the MNIST dataset decreases from 97.34% to 97.05%, a decrease of 0.29%, while the accuracy of the model(γ=0.02) decreases from 92.27% to 90.01%, a decrease of 2.26%, indicating that the accuracy of the model(γ=0.02) decreases more than the model(γ=0.20); the accuracy of the model(γ=0.20) on the CIFAR-10 dataset decreases from 53.49% to 50.09%, a decrease of 3.4%, while the accuracy of the model(γ=0.02) decreases from 41.84% to 28.39%, a decrease of 13.45%, indicating that the accuracy of the model(γ=0.02) decreases more than the model(γ=0.20). It can be seen that as γ decreases, the robustness of the FedPSG model accuracy when the data heterogeneity changes decreases. Therefore, in order to maintain the high quality of the FedPSG training results under different data heterogeneity conditions, the amount of server data D m should be increased as much as possible.

[0072] The beneficial effects of the present application are as follows:

[0073] (1) Combine the particle swarm optimization algorithm with the federated learning training process, and change the communication data form of federated learning from model parameters to model scores. Compared with the traditional federated learning algorithm, FedPSG can effectively reduce the communication cost.

[0074] (2) A model retraining strategy is proposed. The server data is constructed, and the model is optimized by secondary training based on the server data, which not only alleviates the influence of data heterogeneity on federated learning, but also improves the accuracy of the model. In addition, this strategy can also guide the optimization of the federated model in a controllable way.

[0075] (3) Simulate different data heterogeneity conditions on the MNIST and CIFAR-10 datasets, and compare the accuracy of FedPSG, FedAvg and FedShare algorithms. The experimental results prove that the model obtained by FedPSG has higher accuracy, and the model retraining strategy can effectively solve the data heterogeneity problem; design an experiment to study the influence of server data quantity on the accuracy of FedPSG model, and the results prove that the larger the server data quantity, the higher the accuracy of FedPSG model, and the stronger the robustness of model accuracy when the data heterogeneity condition changes.

[0076] As used herein, the word "preferably" is used interchangeably with the word "preferably," to mean that an example, implementation or embodiment is preferred, but not necessarily advantageous over other examples, implementations or embodiments. As used herein, the word "preferably" is used to mean that an aspect, design, or implementation so described is preferred, but not necessarily advantageous over other aspects, designs, or implementations. The use of the terms "preferably," "more preferably," "most preferably," and the like are intended to further clarify the order of preference of various aspects, designs, or implementations. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise, or clear from context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied under any of the foregoing instances. In addition, the articles "a," "an," and "the" as used in the context of this document are to be construed to be open-ended terms (i.e., to mean "one or more"). Follow-on Related Application

[0077] Moreover, although the present disclosure has been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based on the foregoing description and accompanying drawings. The present disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular, with reference to the various functions performed by the above described components (e.g., elements, etc.) to achieve the results, the terms used to describe certain claim components should not be construed to be limited to the specific aspects as set forth above but rather should be interpreted as broadly as is reasonable. For example, the term "comprises" is used in the specification and claims to mean that the disclosed implementations include, but are not limited to, the features, elements, and / or components described in the specification and / or claims. In addition, the term "comprises" is used in the specification and claims to mean that the disclosed implementations include, but are not limited to, the features, elements, and / or components described in the specification and / or claims. Further, the term "comprises" is used in the specification and claims to mean that the disclosed implementations include, but are not limited to, the features, elements, and / or components described in the specification and / or claims.

[0078] The various functional units in the embodiments of the present application can be integrated in one processing module, or each unit can exist physically, or a plurality of or more units can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of a software functional module. If the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. The above-mentioned devices or systems can execute the storage method in the corresponding method embodiments.

[0079] In summary, the above-mentioned embodiments are one embodiment of the present application, but the embodiments of the present application are not limited by the above-mentioned embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principle of the present application are equivalent replacement modes and are included in the protection scope of the present application.

Claims

1. A data heterogeneity-oriented federated learning method based on particle swarm optimization, characterized in that, Comprising the following steps: Before the training starts, the server sends the scoring dataset D s to each client s D m is an independent and identically distributed subset of the server data D In the communication process of federated learning, the client uploads the historical best score pbest of the local model to the server, and the server compares the historical best scores passed by each client to determine the client with the best performance on the scoring data set; Request to obtain the model parameters of the client; Aggregate the local model parameters of the client to update the global model; Wherein, in order to play the guiding role of the global best model and the local best model on the training of the client model, the update of the client neural network model in FedPSG includes one round of particle update and E rounds of local iteration; The particle update process is as follows: wherein, denotes the locally optimal model of the client, denotes the globally optimal model, denotes the client model at the t-th round of communication, denotes the parameters of the l-th layer of the neural network model, and the parameters of all layers constitute , a denotes the inertia weight, r1, r2 are random values between [0, 1], and c1, c2 respectively denote and the acceleration factor of and the greater c2, the more the model training will be biased towards the global model, and the greater c2, the more the model training will be biased towards the local model. To reduce the impact of Non-IID problem on federated learning training, a model retraining strategy is used on the server side: in each round of communication, after receiving the client model, the server uses the server data D m Retrain the model: wherein, is the server received client model in the t-th round of communication process, is the learning rate, D m embodies the data distribution of all client overall training data.

2. The data heterogeneity oriented federated learning method based on particle swarm optimization according to claim 1, characterized in that, D m is an independent and identically distributed subset of the overall training data D, D m The ratio of the amount of data to the amount of data of D is i.e. .

3. The data heterogeneity oriented federated learning method based on particle swarm optimization according to claim 2, characterized in that, D s For D m , an independent and identically distributed subset, represents the direction of FedPSG client model optimization, D s The ratio of the data volume to D m The ratio of the data volume to D .

Images