A lightweight federated learning method and system for spatiotemporal data heterogeneous scenarios

Abstract

The application discloses a lightweight federated learning method and system for a spatio-temporal data heterogeneous scene, which can allocate different communication probabilities to clients according to the change of client data distribution, so that the clients with a large degree of data distribution change and not perceived by the global federated model participate in the federated communication with a higher probability, instead of equally improving the communication amount of all clients. By tilting the communication resources, the global data distribution change can be perceived with less communication amount, and the communication efficiency of federated learning in the spatio-temporal heterogeneous scene is improved. By using the method and system provided by the application, each image acquisition and analysis terminal utilizes the change information of the private case data distribution and the gradient information of the private detection model of the image acquisition and analysis terminal, dynamically adjusts the probability of each image acquisition and analysis terminal participating in the federated communication in each round of federated process, and realizes the same federated learning performance with less federated communication cost, thereby effectively reducing the federated learning communication amount.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of federated learning, and particularly relates to a lightweight federated learning method and system for a spatio-temporal data heterogeneous scene. BACKGROUND

[0002] Deep learning technology has significantly improved the performance of computer vision, natural language processing and other tasks in recent years. By training a deep neural network (DNN) with millions of parameters on a large amount of data, the performance of the deep learning model gradually exceeds that of traditional machine learning, becoming the mainstream of the industrial landing of artificial intelligence technology. Traditional deep learning, due to its dependence on a large amount of training data, usually adopts a centralized mode of collecting data on edge devices and uploading to a cloud server for model training and inference services.

[0003] However, in recent years, with the introduction of laws on data security and personal privacy protection, the process of uploading private data collected from user devices such as mobile phones and tablets to a central server has faced many restrictions. This trend has made the previous mode of centralized model training and service on end devices a challenge. To solve this problem, federated learning (FL) is an attractive solution that enables collaborative learning across various devices without sharing private data. Specifically, FL is a decentralized collaborative training framework that helps to deliver encrypted models or gradients to collaboratively train client models. During the FL process, model training based on private data is offloaded to the client for local execution, rather than being uploaded to the federated server for centralized training, thereby protecting the security of private data.

[0004] However, applying federated learning in real-world scenarios requires dealing with spatio-temporal heterogeneous data environments. On the one hand, federated learning faces the challenge of spatial heterogeneity, i.e., the private data of clients may not satisfy the independent and identically distributed assumption. For example, the distribution of detected diseases differs among hospitals due to the characteristics of the cases they treat. Specifically, the distribution of patient diseases in orthopedic hospitals and infectious disease hospitals will differ significantly. Applying federated learning on such data will result in a global model update direction that does not align with the true data distribution, leading to slow model convergence or poor performance, i.e., data exists spatial heterogeneity. On the other hand, federated learning faces the challenge of temporal heterogeneity, i.e., the distribution of client private data changes over time, and from the perspective of the federated system, the distribution of global data is unstable. For example, due to time-related factors such as seasons and epidemics, the proportion of detected allergic diseases in hospitals increases regularly from April to May each year. That is, data exists temporal heterogeneity. Both of these phenomena are commonly seen in federated learning application scenarios. In the face of spatio-temporal heterogeneous data distribution, the federated server needs to communicate with the clients frequently to maintain the consistency of the target distribution of the federated global model with the true distribution.

[0005] With the increasing performance of deep models, the demand for multi-modal information processing and the demand for model representation and reasoning ability have made the structure of the model become larger and larger. For example, the parameter quantity of VGG deep network commonly used in medical image recognition tasks in intelligent medical scenarios has reached the level of billions. The parameter quantity of GPT-3 deep model commonly used in intelligent medical scenarios such as medical assistant tasks is even as high as 100 billion, which is outstanding in performance. Frequent communication of large models will cause heavy communication cost and power consumption of mobile intelligent devices, which seriously hinders the application of federated learning. Further, the above contradictions have given rise to the challenge of low-cost client collaborative training under the premise that the client's private data is not exposed in the spatio-temporal data heterogeneous scene.

[0006] In view of the lightweight challenge of federated models, existing works have been carried out from two angles of reducing the number of federated communications and reducing the amount of federated communications. Among them, in terms of reducing the number of federated communications, the Fed average algorithm controls the communication cost of the client by adjusting the proportion of the randomly selected client in each round of federated learning. However, this method lacks differentiated consideration of the value of the client. The FedPNS method controls the communication of the client from the gradient consistency of the client and the global federated model. This method uses the similarity between the gradient update direction of the client and the global federated update gradient to measure the importance of the client, and allocates the communication probability of the federation based on this importance measure. In addition to the gradient-based method, methods such as FedMCCS use a multi-criteria client selection mechanism based on client performance to tilt the communication demand to clients with relatively abundant resources, thereby balancing the resource load of the clients and reducing the communication efficiency of federated learning by reducing the communication demand of weak resource clients. On the other hand, in terms of reducing the amount of federated communication, existing works start from the compression of the communication model quantity and propose a variety of small model communication methods. The Hermes method proposes that in a federated system where client data distribution is inconsistent, a super network is generated for the global federated model to learn global knowledge, and a sub-network with fewer parameters is selected at each client using a neural network pruning method to fit the client's private knowledge. During the federated communication process, the client and the server only interact with the intersection part of the model, thereby reducing the communication parameter quantity of a single federated process.

[0007] However, most existing works assume that the distribution of the client is stable over time. Therefore, when the data distribution changes over time, the existing method can only non-differentially improve the communication probability of all clients to enable the global federated model to perceive the data distribution change as soon as possible. This undoubtedly further worsens the communication burden of the federated system. SUMMARY

[0008] The present application aims at the deficiencies of the prior art, and proposes a lightweight federated learning method and system for spatio-temporal data heterogeneous scenarios, which can assign different communication probabilities to clients according to the change of client data distribution, so that the clients with large data distribution change degree and not perceived by the federated global model participate in federated communication with a higher probability, instead of equally increasing the communication amount of all clients. By tilting the communication resources, the global data distribution change can be perceived with less communication amount, thereby improving the communication efficiency of federated learning in spatio-temporal heterogeneous scenarios.

[0009] In order to achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0010] In one aspect, the present application provides a lightweight federated learning method for spatio-temporal data heterogeneous scenarios, comprising the following steps:

[0011] S1, client local update: after receiving the latest federated global model, the federated global model is updated to a more localized model that conforms to the local private data distribution by using the private data of the client;

[0012] S2, client importance evaluation: the importance of the client participating in federated communication to the federated global model is measured by using the distribution change degree of the private data and the consistency of the gradient information of the client model and the gradient information of the global model;

[0013] S3, client communication probability control: the client communication probability is dynamically adjusted according to the importance of the client obtained in step S2, and it is determined whether to upload according to the probability, if yes, step S4 is entered, if not, the federated global model is distributed to all clients;

[0014] S4, federated server aggregation: the gradient information uploaded by the client participating in federated update in this round is used to update the federated global model, and the updated federated global model is distributed to all clients, and the next round of federated process is entered until the preset federated round is reached.

[0015] Further, the specific process of step S1 is: based on the federated global model ω t , the private data set of the client i in the t round is used to continue training by the gradient descent iterative algorithm, to obtain the local model The optimization objective function of the client is Where n is the sample size, and f is the loss function of the client i. In the classification task, it is usually cross-entropy loss.

[0016] Further, the specific process of step S2 is:

[0017] S21, the model obtained by the client i after the local update in the tth round is First, calculate the model The Top-1 accuracy on the latest data shard Second, calculate the model The accuracy on the last n rounds of data shards Finally, calculate the absolute value of the accuracy difference of the same model on data at two different time ends Reflect the absolute distribution change of the local data of the client, denoted as

[0018] S22, the client caches the local model uploaded to the federal server last time Compare the difference between it and the local model in this round Get the gradient information of the client not uploaded Compare the difference between the federal global model ω s received after the last upload of the client and the latest federal global model ω t Get the perceived gradient information of the federal global model By calculating the cosine similarity of the client gradient information and the federal global model gradient information Measure the relative difference between the gradient to be uploaded by the client and the perceived gradient of the federal global model, denoted as

[0019] S23, combine the absolute change AC and the relative change RC in the form of multiplication as the importance measure of the client i in the tth round:

[0020] Further, the specific process of step S3 is:

[0021] S31, input the client importance measure value into the communication probability mapping module, and update the communication probability of the client i as Where tau controls the communication resources allocated by the federal system for the time change of data distribution, reflecting the sensitivity to distribution change. A larger tau will make the federal system adapt to the distribution change of the client faster, but will also produce additional higher communication overhead; P floor Control the basic communication probability of the client; therefore, the importance of the client S is higher, and the uplink probability is larger, and vice versa, the importance of the client is lower, and the uplink probability is smaller.

[0022] S32, the communication probability control module generates a OneHot vector in each round of federal process according to the probability, that is, the OneHot vector, where one dimension value is 1 and other dimension values are 0, and requires the corresponding client to upload the gradient information of the local model.

[0023] Further, the specific process of step S4 is:

[0024] S41, the selected client i uploads the gradient accumulated gradient from the s moment to the t moment in the t round

[0025] S42, the client set uploaded in the round is C t , the historical federal global model is omega t , the accumulated gradient information of the uploaded client is updated to the federal global model with a learning rate eta

[0026] S43, the updated federal global model is issued to all clients.

[0027] On the other hand, the application also provides a lightweight federated learning system for a spatio-temporal data heterogeneous scene, comprising the following modules to realize the method of any one of the above:

[0028] The client local update module is used to update the federal global model to a more localized model that is more consistent with the local private data distribution after receiving the latest federal global model by using the private data of the client locally;

[0029] The client importance evaluation module is used to measure the importance of the client participating in the federal communication to the federal global model by using the consistency of the gradient information of the client model and the global model gradient information and the degree of change of the distribution of the private data;

[0030] The client communication probability control module dynamically adjusts the client communication probability according to the client importance obtained by the client importance evaluation module, and determines whether to upload according to the probability, if yes, enters the federal server aggregation module, if not, the federal global model is issued to all clients;

[0031] The federal server aggregation module is used to update the federal global model by using the gradient information uploaded by the client participating in the federal update in the round, and issue the updated federal global model to all clients, and enter the next round of federal process until the preset federal round is reached.

[0032] Compared with the prior art, the application has the following beneficial effects:

[0033] The lightweight federated learning method and system for a spatio-temporal data heterogeneous scene can allocate different communication probabilities to clients according to the change of the client data distribution, so that the client with a large data distribution change degree and not perceived by the federal global model can participate in the federal communication with a higher probability, instead of equally improving the communication amount of all clients. By tilting the communication resources, the global data distribution change can be perceived with less communication amount, thereby improving the communication efficiency of federated learning in a spatio-temporal heterogeneous scene.

[0034] By using the lightweight federated learning method and system for spatio-temporal data heterogeneous scene proposed in the application, each image collection and analysis terminal calculates its importance in each round of federated process by using the change information of its private case data distribution and the gradient information of the image collection and analysis terminal private detection model. And according to the image collection and analysis terminal importance measurement result, the probability of each round of federated process image collection and analysis terminal participating in federated communication is dynamically adjusted. By distributing the communication demand to more important image collection and analysis terminals, the communication amount of low importance image collection and analysis terminals is reduced, the same federated learning performance is achieved with less federated communication cost, and the federated learning communication amount is effectively reduced. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0036] Figure 1 The flowchart of the lightweight federated learning method for spatio-temporal data heterogeneous scene provided for the embodiments of the present application.

[0037] Figure 2 The lightweight federated learning system architecture diagram for spatio-temporal data heterogeneous scene provided for the embodiments of the present application. DETAILED DESCRIPTION

[0038] In order to better understand the technical solutions, the method of the present application will be described in detail below with reference to the drawings.

[0039] In the spatio-temporal data heterogeneous scene, the present application uses the distribution change degree of the client private data and the gradient information of the client model to dynamically adjust the uplink communication probability of the client in the federated process, and reduces the communication overhead of the low importance client. Thus, the communication efficiency bottleneck problem of the federated system in the spatio-temporal data heterogeneous scene is effectively solved, and a lightweight federated learning system is realized.

[0040] In the spatio-temporal data heterogeneous environment, the federated global model needs to constantly fit the changing data distribution. Due to data privacy protection, the fitting process of the distribution change needs to be completed through the upload of the federated client model information. However, due to the different degrees of change of the data distribution of each client over time, when fitting the global data distribution change, how to select the client participating in the communication will greatly affect the fitting effect and the required communication amount. In order to realize the fitting of the data distribution change in a lightweight manner, the lightweight federated learning method and system for spatio-temporal data heterogeneous scene proposed in the present application, such as Figure 1 andFigure 2 The system comprises four main modules, namely a client local update module, a client importance evaluation module, a federal communication probability control module, and a federal server aggregation module.

[0041] The client local update module first uses private data to update the federal global model to a private model locally at the client. Then, the invention introduces a client importance detection module for client distribution change degree, to quantify the contribution of each client communication to the global model distribution change perception in each round of federated learning.

[0042] Firstly, if the client local data distribution is stable, its upload communication cannot contribute to the fitting of the global distribution change. Therefore, the importance detection module uses the difference between the accuracy of the latest client local model on the current round of incremental samples and the accuracy on historical samples to reflect the distribution difference between the current round of local private incremental data and historical data. This measurement method is called client absolute distribution change degree measurement.

[0043] Secondly, since federated learning involves multiple clients, if a client's local distribution change information has been perceived by the federal global model through communication with other clients, the importance of the client's communication to the distribution change perception is lower than that of the client whose distribution change information has not been perceived by the federal global model. Therefore, the cosine similarity between the client's unuploaded gradient information and the gradient information of the global model corresponding to the time window is further introduced to reflect the relative change degree of the distribution change carried by the client's incremental data and the global model's perceived distribution. If the client's model change is in the same direction as the federal global model's change, i.e. the similarity is high, the part of the client's incremental information that has not been perceived by the global model is less, and the corresponding importance is lower. Finally, by combining the absolute importance and the relative importance as weights in a multiplicative manner, the degree of client private data distribution change and the similarity between the client private data change and the global model's perceived change are comprehensively considered, and the importance of the client's communication to the federal global model's perception of data distribution change is fully reflected.

[0044] Based on the measurement of client importance, naturally, more federal communication demand is allocated to clients that contribute more to the federal global model's perception of data distribution change, which can improve communication efficiency. Therefore, after updating the client importance in each round of federated learning, the client communication control module adjusts the probability of the client uploading the model according to the client's importance. By mapping the client importance monotonically to the federal communication probability, high-importance clients will have a higher probability of uploading the model to participate in federated aggregation, and the model upload of low-importance clients will be correspondingly reduced. Finally, the federal server aggregation module is responsible for updating the client's model gradient information uploaded with probability to the historical federal global model, and sending the updated federal global model to the client.

[0045] Example 1

[0046] Task description:

[0047] The federated learning system contains 10 clients, and the federated learning process lasts for 10 rounds in the experiment. The simulation method of simulating data space heterogeneity is to randomly assign 20% of the corresponding data of each client to the class during the initialization process of federated learning, and the incremental data of the subsequent client comes from the assigned class in the initialization process. The simulation method of simulating the change of the distribution of the private data of the client over time is to change the data class of 4 clients from the 5th round. In the case, the federated process of client 6 in the 5th round is described in detail:

[0048] Step 1: Client local update

[0049] Step 1.1: The client receives the federated global model ω issued by the server 5 Each client of the federated system updates the local model ω where i is the client id belongs to [1,10].

[0050] Step 2: Client importance evaluation

[0051] Step 2.1: First, absolute importance detection is performed. Take client 6 as an example, which has a change in local private data distribution in the 5th round. The local model ω is tested on the local private incremental data set in the 5th round The accuracy is 0.8, and the accuracy on the union of the local private incremental data set of the previous 2 rounds and is 0.85. Further, the absolute value of the difference between the accuracies of the two data sets, abs(0.80.85), reflects the difference between the current round data set and the historical 2 rounds data set, which is 0.05. The absolute change value of the data distribution of client 6 in the 5th round is 0.05.

[0052] Step 2.2: Calculate the relative distribution change degree of client 6. Client model 6 uploaded the local model in the 3rd round before the 5th round, so the cumulative gradient of client from the 3rd round to the 5th round is calculated as denoted as Similarly, the cumulative gradient ω 5 -ω 3 of the federated global model ω 5 and ω 3 at the last upload time point of the client is calculated as Further, the cosine similarity between and is 0.4, and the relative importance measure is 1-0.4=0.6.

[0053] Step 2.3: The importance of client 6 in the 5th round is the product of its relative importance and absolute importance, that is,

[0054] Step 3: Client communication probability control

[0055] Step 3.1: Based on the measurement results of the importance detection module of the client, in each round of the federal process, the communication probability of each client j is updated as where the hyperparameter tau is set to 2, P floor is set to 5%. Then the communication probability of client 6 in the 5th round is 6%.

[0056] Step 3.2: Client 6 generates a random number in [0, 1], if the random value is less than 6%, the gradient information of the local model is uploaded to the federal server, otherwise it does not participate in this round of uploading.

[0057] Step 4: Federal server aggregation

[0058] 4.1 The federal server updates the client gradient information uploaded in this round to the federal global model, assuming that the client set C uploaded in this round is [6, 7, 9], and the uploaded gradients of clients [6, 7, 9] are respectively The new round of federal global model where η is the update learning rate of the federal global model, which is set to 0.01 in the experiment.

[0059] 4.2 The federal server distributes the federal global model ω 6 to all clients, and enters the next round of federal process.

[0060] Experiments show that in the scenario of 20% client distribution changing over time, compared with the baseline methods such as Fed-average and FedPNS, the system can achieve the same performance as the average 40% client participating in federal uploading with the communication cost of the average 30% client uploading, that is, it reduces the uplink communication overhead by 10% under the same performance. It effectively improves the federal communication efficiency and realizes lightweight federated learning in the space-time heterogeneous scene.

[0061] In summary, the application proposes a lightweight federated learning system and method for spatiotemporal data heterogeneous scenarios. In the Internet of Things scenarios such as intelligent medical treatment, the importance of the client is measured by using the private data distribution change information of the client, and the communication probability of the client is dynamically adjusted according to the importance in the federated process, thereby reducing the communication overhead of the federated system. The image acquisition and analysis terminal calculates its importance in each round of federated process by using the private case data distribution change information of the image acquisition and analysis terminal and the gradient information of the private detection model of the image acquisition and analysis terminal. According to the importance measurement result of the image acquisition and analysis terminal, the probability of the image acquisition and analysis terminal participating in the federated communication in each round of federated process is dynamically adjusted. By distributing the communication demand to the more important image acquisition and analysis terminal, the communication amount of the low importance image acquisition and analysis terminal is reduced, the same federated learning performance is achieved with less federated communication cost, and the federated learning communication amount is effectively reduced.

[0062] The above examples are only used to illustrate the technical solutions of the application, and not to limit it. Although the application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalents, but these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A lightweight federated learning method for spatiotemporal data heterogeneous scenarios, characterized in that, The method comprises the following steps: S1, client local update: after receiving the latest federal global model, the federal global model is updated to a more localized model that is more consistent with the local private data distribution by using the private data of the client locally; S2, client importance evaluation: the degree of distribution change of private data and the consistency of gradient information of the client model and the global model gradient information are used to measure the importance of the client participating in the federal communication to the federal global model; the importance measurement formula of client i in t rounds is: is the absolute distribution change of the client local data, is the relative difference between the gradient to be uploaded by the client and the perceived gradient of the federal global model; S3, client communication probability control: according to the importance of the client obtained in step S2, dynamically adjust the client communication probability, judge whether to upload according to the probability, if yes, go to step S4, if not, the federal global model is distributed to all clients; the communication probability update formula of client i is Where tau controls the communication resources allocated by the federal system for the time variation of data distribution, reflecting the sensitivity to distribution variation; P floor control the basic communication probability of the client; S4, federal server aggregation: the federal global model is updated by using the gradient information uploaded by the client participating in the federal update in this round, and the updated federal global model is distributed to all clients, and the next round of federal process is entered until the preset federal round is reached.

2. The lightweight federated learning method for spatiotemporal data heterogeneous scenes of claim 1, wherein, The specific process of step S1 is: on the basis of the federated global model ω t , the private data set of the client i is used to continue training through the gradient descent iterative algorithm in t rounds , to obtain the local model The optimization objective function of the client is Wherein, n is the sample size in f is the loss function of the client i.

3. The lightweight federated learning method for spatiotemporal data heterogeneous scenes of claim 1, wherein, The specific process of step S2 is: S21, the model obtained by the client i after the local update in the tth round is First, calculate the model Top-1 accuracy on the latest data shard Second, calculate the model Accuracy on the last n rounds of data shards Finally, calculate the absolute value of the accuracy difference of the two different time end data on the same model ABS Reflect the absolute distribution change of the local data of the client, denoted as S22, the client caches the local model uploaded to the federation server last time The difference between the local model of this round and the difference between the local model of this round The difference between the local model of this round and the difference between the local model of this round The difference between the local model of this round and the difference between the local model of this round s The difference between the local model of this round and the difference between the local model of this round t The difference between the local model of this round and the difference between the local model of this round The difference between the local model of this round and the difference between the local model of this round The difference between the local model of this round and the difference between the local model of this round S23, combine the absolute change AC with the relative change RC in multiplicative form as a measure of importance of client i at round t:

4. The lightweight federated learning method for spatiotemporal data heterogeneous scenes of claim 1, wherein, The specific process of step S3 is: S31, input the client importance measure value into the communication probability mapping module, the communication probability of the client i is updated as Where tau controls the communication resource allocated by the federation system for the time variation of data distribution, reflecting the sensitivity to distribution variation; P floor Control the basic communication probability of the client, dynamically increase the uplink probability of high importance clients, and reduce the uplink probability of low importance clients; S32, the communication probability control module generates a client one-hot vector to be uploaded in each round of federal process according to a probability, wherein one dimension value is 1 and other dimension values are 0, and the corresponding client is required to upload the gradient information of the local model.

5. The lightweight federated learning method for spatiotemporal data heterogeneous scenes of claim 1, wherein, The specific process of step S4 is: S41, the selected client i uploads the gradient accumulated gradient from s time to t time in the tth round S42, the client set uploaded in this round is C t , the historical federated global model is ω t , the cumulative gradient information of the uploaded client is updated to the federated global model ω S43, the updated federal global model is distributed to all clients.

6. A lightweight federated learning system for spatiotemporal data heterogeneous scenarios, characterized in that, The method comprises the following modules to realize the method according to any one of claims 1-5: A client local update module is configured to update the federal global model to a more localized model that is more consistent with the local private data distribution by using the private data of the client locally after receiving the latest federal global model; A client importance evaluation module is configured to measure the importance of the client participating in the federal communication to the federal global model by using the distribution change degree of the private data and the consistency of the gradient information of the client model and the gradient information of the global model; A client communication probability control module is configured to dynamically adjust the client communication probability according to the client importance obtained by the client importance evaluation module, to determine whether to upload according to a probability, if yes, the federal server aggregation module is entered, if not, the federal global model is distributed to all clients; A federal server aggregation module is configured to update the federal global model by using the gradient information uploaded by the client participating in the federal update in this round, and the updated federal global model is distributed to all clients, and the next round of federal process is entered until the preset federal round is reached.

Images