Federal learning method oriented to privacy and heterogeneous data of Internet of Things and privacy data protection method and system of Internet of Vehicles

Abstract

The invention relates to the technical field of internet of things federated learning, in particular to a federated learning method oriented to internet of things privacy and heterogeneous data and an internet of vehicles privacy data protection method and system, differential privacy is introduced into federated learning, Gaussian noise meeting the differential privacy is added to a client to be uploaded to local parameters of a server, and meanwhile, the local parameters of the client are added to the internet of things privacy and heterogeneous data. In order to prevent excessive accumulation of privacy cost in model iteration training, noise disturbance is added to parameters uploaded by a local model through dynamic differential privacy, and the privacy and availability of the model are balanced to the maximum extent by using an exponential attenuation mechanism of Gaussian noise; the Wasserstein distance between the local parameter and the global parameter is calculated to serve as a regularization item to update the local model, the generalization ability of the model to Non-IID data is improved, and the accuracy of the global model is improved; and when high-dimensional data is processed, dimension reduction processing is performed on a high-dimensional data set by combining PCA with federated learning, so that the data set consistency of each client is improved while the local model training speed is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things (IoT) federated learning, and particularly to a federated learning method for IoT privacy and heterogeneous data, as well as a method and system for protecting privacy data in vehicle-to-everything (V2X) networks. Background Art

[0002] With the advent of the concept of the Internet of Things, the world has entered a new technological era characterized by the interconnection of devices and the extensive collection of data. IoT devices, ranging from smart watches to industrial sensors, continuously collect and transmit data, bringing convenience and improved efficiency to life. However, with the explosion of data volume, how to protect user privacy, ensure data security, and at the same time make full use of this data has become an important issue. Federated learning, as an emerging machine learning paradigm, provides a potential solution to this problem. Federated learning aims to address the problem of data "islands" caused by security isolation and data barriers between different networks and industries, where data cannot be securely shared between different systems, and the performance of machine learning models cannot reach the global optimum. Federated learning uses a distributed machine learning framework to store user privacy data locally and perform model training locally. Only the training parameters are shared between users and the central server to obtain the global optimal model. This way of training machine learning models not only reduces the system overhead of the central server but also solves the data "island" problem, effectively protecting the data privacy of users participating in model training.

[0003] However, studies have shown that it is possible to reconstruct the client dataset by analyzing the model parameters uploaded by the client, thus threatening the privacy and security of participants. Therefore, to prevent local data leakage, the most common privacy protection method in federated learning is to add noise perturbation to the uploaded parameters by the client. However, adding too much noise will lead to a decrease in model usability as the privacy level increases. Therefore, it is necessary to balance the privacy and usability of the model. In addition, due to different device types, the local datasets of IoT clients may be unevenly distributed and feature-shifted, forming non-independent and identically distributed (non-IID) data. Non-IID data will cause local model heterogeneity, and the generalization error of the global model will also increase accordingly, hindering the convergence of the model during the federated learning training process and reducing the overall performance of the model. Therefore, non-IID data poses a significant challenge to federated learning in cross-device IoT scenarios. Summary of the Invention

[0004] Therefore, the present invention provides a federated learning method for IoT privacy and heterogeneous data, as well as a method and system for protecting privacy data in vehicle-to-everything (V2X) networks, to solve the problems of imbalance between data privacy and model usability and model convergence when dealing with non-independent and identically distributed (non-IID) data in existing IoT federated learning.

[0005] According to the design solution provided by the present invention, on the one hand, a federated learning method for Internet of Things privacy and heterogeneous data is provided, and a target model for protecting the privacy data of Internet of Things devices is obtained by using the federated learning method. The federated learning method includes:

[0006] Each client of the Internet of Things trains a local partial model using a local data set, adds noise perturbation to the local partial model parameters based on differential privacy, and uploads the local partial model parameters to the Internet of Things federated learning central server. The local data set consists of heterogeneous data that is not independently and identically distributed, and the differential privacy uses exponentially decaying Gaussian noise to perturb the local partial model parameters;

[0007] The Internet of Things federated learning central server aggregates the received local partial model parameters to obtain global model parameters, uses the Wasserstein distance as a regularization term for the training loss of the local partial model, and updates the local partial model using this regularization term. The Wasserstein distance is used to measure the difference between the global model parameters and the local model parameters.

[0008] As the federated learning method for Internet of Things privacy and heterogeneous data of the present invention, further, adding noise perturbation to the local partial model parameters based on differential privacy includes:

[0009] Use a dynamic noise coefficient to set the degree of exponential noise decay, and the dynamic noise coefficient is set according to the decay rate, the global model training iteration data, and the local partial model training iteration data;

[0010] During the iterative process of local partial model training, the degree of exponential noise decay is added as perturbation noise to the local model training loss, and the local partial model parameters are updated according to the training loss with the added perturbation noise, so as to upload the updated local partial model parameters to the Internet of Things federated learning central server for global model update.

[0011] As the federated learning method for Internet of Things privacy and heterogeneous data of the present invention, further, the process of updating the local partial model using this regularization term is expressed as:

[0012] F i (w) = f i (w) + λD(w i , w), where f i (w) is the loss function of client i as a participant in federated learning, w i is the local partial model parameter, D(w i , w) is the local partial model parameter w iThe Wasserstein distance from the global model parameter w, where λ is the regularization parameter for controlling (w i , w).

[0013] As the federated learning method for Internet of Things privacy and heterogeneous data according to the present invention, further, the federated learning method further includes:

[0014] Pre-dimension reduction processing is performed on the local dataset using the principal component analysis method, so as to use the pre-dimension reduction processed local dataset to perform the training of the local local model during the federated learning process.

[0015] As the federated learning method for Internet of Things privacy and heterogeneous data according to the present invention, further, performing pre-dimension reduction processing on the local dataset using the principal component analysis method includes:

[0016] Extract the feature matrix of the local dataset of each client of the Internet of Things, calculate the covariance matrix of the local dataset of each client according to the feature matrix, update the covariance matrix using Gaussian noise and upload it to the Internet of Things federated learning central server;

[0017] The Internet of Things federated learning central server performs aggregation averaging on each covariance matrix, performs eigenvalue decomposition on the aggregated averaged covariance matrix, intercepts several eigenvectors corresponding to the highest eigenvalues and forms a feature space matrix, and transmits the feature space matrix back to the client;

[0018] The client uses the received feature space matrix to perform dimension reduction update on the local dataset feature matrix.

[0019] As the federated learning method for Internet of Things privacy and heterogeneous data according to the present invention, further, the process of the dimension reduction update is expressed as: where i = 0,..., N - 1, N is the number of Internet of Things clients, X i is the feature matrix of the local dataset of the i-th Internet of Things client, is the feature space matrix.

[0020] On the other hand, the present invention also provides a method for protecting privacy data in a vehicle-to-everything network, including the following content:

[0021] Each vehicle end in the vehicle-to-everything network is used as a client participating in the federated learning, and the cloud server is used as the federated learning central server to execute the above-mentioned federated learning method, and the target model of each vehicle end is obtained, so that each vehicle end uses the target model to protect the vehicle end privacy data when the vehicle-to-everything network data is collaboratively shared.

[0022] On the other hand, the present invention also provides an Internet of Vehicles privacy data protection system, comprising: a cloud server and a number of vehicle terminals communicatively connected to the cloud server, wherein the vehicle terminals and the cloud server implement data collaborative sharing in the Internet of Vehicles through a federated learning method. Among them,

[0023] The vehicle terminal, as a federated learning client, trains the local local model using the local data set, adds noise perturbation to the local local model parameters based on differential privacy, and uploads the local local model parameters to the Internet of Things federated learning central server. The local data set consists of heterogeneous data that is not independently and identically distributed. The differential privacy uses exponentially decaying Gaussian noise to perturb the local local model parameters;

[0024] The cloud server, as the federated learning central server, aggregates the received local local model parameters to obtain global model parameters, uses the Wasserstein distance as a regularization term for the local local model training loss, and updates the local local model using this regularization term. The Wasserstein distance is used to measure the difference between the global model parameters and the local model parameters.

[0025] Advantages of the present invention:

[0026] The present invention introduces differential privacy into federated learning, adds Gaussian noise that satisfies differential privacy to the local parameters uploaded by the client to the server. At the same time, to prevent the excessive accumulation of privacy costs during model iterative training, noise perturbation is added to the local model upload parameters through differential privacy, and the exponential decay mechanism of Gaussian noise is used to balance the privacy and usability of the model to the greatest extent. By calculating the Wasserstein distance between the local parameters and the global parameters as a regularization term to update the local model, the generalization ability of the model to Non-IID data is improved, and the accuracy of the global model is increased; when dealing with high-dimensional data, PCA is combined with federated learning to reduce the dimension of the high-dimensional data set, which not only improves the training speed of the local model, but also the local data set is aggregated and averaged by the server to improve the consistency of each client data set. Further experiments are carried out on non-IID data, and the experimental results show that the solution of this case can significantly improve the accuracy and usability of the model on heterogeneous data while protecting user privacy. As a solution for federated learning data privacy protection and collaboration in the Internet of Things environment, it has good application prospects in the Internet of Vehicles and other Internet of Things environments, and has important theoretical and practical significance for promoting the secure use and intelligent analysis of Internet of Things data. Description of the Drawings

[0027] Figure 1 Schematic diagram of the federated learning process for Internet of Things privacy and heterogeneous data in the embodiment;

[0028] Figure 2 Schematic diagram of the model deviating from the global optimal solution under non-IID data in the embodiment;

[0029] Figure 3 Schematic diagram of the model re-converging to the global optimal solution by calculating the wasserstein distance in the embodiment;

[0030] Figure 4 Schematic diagram of the Internet of Vehicles with differential privacy added in the embodiment;

[0031] Figure 5 Schematic diagram of the data dimensionality reduction process of differential federated learning based on PCA in the embodiment;

[0032] Figure 6 Schematic diagram of the accuracy of FedWDP on the MNIST and CIFAER10 datasets under different privacy budgets in the embodiment;

[0033] Figure 7 Schematic diagram of the accuracy of FedWDP on the MNIST and CIFAER10 datasets under the noise attenuation rate in the embodiment;

[0034] Figure 8 Schematic diagram of the accuracy of PCA-FedWDP on the MNIST dataset under different dimensionality reduction ratios in the embodiment;

[0035] Figure 9 Schematic diagram of the accuracy of PCA-FedWDP on the MNIST dataset under different ratios of ∈1 and ∈2 in the embodiment;

[0036] Figure 10 Schematic diagram of the accuracy comparison of each algorithm on the MNIST and CIFAER10 datasets in the embodiment. Detailed implementation manners

[0037] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and technical solutions.

[0038] Facing the increasing demands for device interconnection and data sharing in the Internet of Things, traditional centralized data processing methods have the risk of privacy leakage and are difficult to adapt to the diversity and distribution of Internet of Things devices. Federated learning, as a distributed machine learning method, provides a new way to solve these problems.

[0039] Federated learning is essentially a distributed machine learning technology. Participating users use their own data to train local models, and the central server aggregates the training parameters uploaded by the participating users to form a global model. Through iteration, an ideal model approaching the results of centralized machine learning is obtained. Federated learning can form training data sharing without aggregating source data, protecting the data privacy of participants during the machine learning process. The solution objective of federated learning, that is, the objective function F(·) of the centralized server, can be expressed as:

[0040]

[0041] where F(w) is the global loss function, w is the global parameter, N is the number of users participating in federated learning, is the total data set, is the data set of the i-th user, and F i (w) is the local loss function of the i-th user.

[0042]

[0043] In the sum of the loss functions f i (w) generated by all instances divided by the total data volume of user i is the average loss function F i (w) of the local client. The loss function is inversely proportional to the model accuracy. Therefore, the optimization of the objective function of machine learning is to make F(w) obtain the minimum value in the domain. Usually, the stochastic gradient descent method SGD is used to update the model parameter w, and the update of the local model parameter w t can be expressed as:

[0044]

[0045] where η is a fixed learning rate. The local model parameters of each participating user are uploaded to the central server to aggregate and update the global model parameters:

[0046]

[0047] However, the heterogeneity of Internet of Things data and the strict requirements for privacy protection pose new challenges to federated learning algorithms. In the embodiments of the present invention, as shown in Figure 1 a federated learning method for Internet of Things privacy and heterogeneous data is provided. The federated learning method is used to obtain a target model for protecting the privacy data of Internet of Things devices. The federated learning method includes:

[0048] S101. Each client of the Internet of Things trains the local local model using the local dataset, adds noise perturbation to the local local model parameters based on differential privacy, and uploads the local local model parameters to the central server of the Internet of Things federated learning. The local dataset consists of heterogeneous data that is not independently and identically distributed. The differential privacy uses exponentially decaying Gaussian noise to perturb the local local model parameters.

[0049] Differential privacy requires that the influence of each single element on the output in the dataset is limited, so that an attacker cannot infer which individual in the dataset makes the query return such a result after observing the query result, thereby providing protection for the privacy of each individual in the dataset, rather than proving the privacy protection of the integrity of the dataset.

[0050] Suppose a randomized mechanism that satisfies differential privacy Then For two adjacent datasets There exists:

[0051]

[0052] where ∈ is the privacy budget and δ is the relaxation term. When δ = 0, it satisfies strict ∈-differential privacy.

[0053] For any k algorithms, they respectively satisfy ∈ 1 -differential privacy, ∈ 2 -differential privacy,..., ∈ k -differential privacy. When they act on the same dataset, it satisfies -differential privacy.

[0054] Divide a dataset into k sets, which are respectively Let A 1 , A 2 ,..., A k be k differential privacy algorithms that respectively satisfy ∈ 1 , ∈ 2 ,..., ∈ k Then The result of satisfies max i∈(1,2,…,k) ∈ i -differential privacy.

[0055] Suppose two adjacent datasets For any domain function s: The sensitivity of s is the maximum range of change when s makes a query on the dataset. Then the sensitivity of s is expressed as:

[0056]

[0057] The sensitivity of s is independent of the dataset and is only determined by the query function itself.

[0058] For There exists:

[0059]

[0060] Among them, satisfies the Gaussian distribution,

[0061] Specifically, adding noise perturbation to the local model parameters based on differential privacy can be designed to include:

[0062] Using the dynamic noise coefficient to set the exponential noise attenuation degree, and the dynamic noise coefficient is set according to the attenuation rate, the global model training iteration data, and the local model training iteration data;

[0063] During the iterative process of local model training, the exponential noise attenuation degree is added as perturbation noise to the local model training loss, and the local model parameters are updated according to the training loss with added perturbation noise, so as to upload the updated local model parameters to the Internet of Things federated learning center server for global model update.

[0064] The client directly sharing the model parameters with the central server may cause the leakage of local model privacy. Therefore, in the FedWDP algorithm of this case, the Gaussian mechanism is introduced to add noise to the uploaded parameters to construct a federated learning model with differential privacy protection.

[0065] To prevent model overfitting, gradient clipping technology is usually adopted in the gradient descent algorithm to ensure that the model parameters do not exceed the gradient threshold C. The gradient clipping for client i in this model can be expressed as:

[0066]

[0067] As the number of iterations increases, excessive privacy costs will accumulate. At the same time, as the number of training rounds increases, the model gradually converges, and the differences in model parameters among clients will become smaller and smaller. In this case, it will be difficult for attackers to extract sensitive information of a specific client from the model parameters. For this, exponential noise attenuation can be used to reduce the privacy investment in the later stage of training. Add a large amount of noise at the initial stage of the model, and reduce the noise perturbation when the model tends to converge. The exponential noise attenuation degree can be defined by the dynamic noise coefficient α:

[0068]

[0069] Among them, \(k\in(0,1)\) is the decay rate, \(t\) is the current global iteration round, \(r\) is the current local iteration round, \(T\) is the final global iteration round, and \(R\) is the final local iteration round.

[0070] Meanwhile, when sampling and training the data of client \(i\) with a batch size of \(B\), Gaussian noise is added:

[0071]

[0072] By adding an exponential noise decay mechanism, the local model parameters of client \(i\) are updated as:

[0073]

[0074] After the local model is updated, the local model parameters need to be uploaded to the server for update. During the upload to the server, all local models satisfy \((\epsilon,\delta)\)-differential privacy. Therefore, by perturbing the local model through local differential privacy, it will be difficult for malicious attackers to infer the user-sensitive information on client \(i\) from the uploaded model parameters \(w\) i among them.

[0075] In the FedWDP algorithm, each client \(i\) conducts sampling training under non-IID data, uses the Wasserstein distance to quantify the difference between the local model and the global model, and updates the model parameters \(w\) i . By Wasserstein optimization, the negative impact of performance degradation caused by model heterogeneity under non-IID data is reduced, the difference between local models is narrowed, global convergence is accelerated, and the model accuracy is improved. At the same time, the local parameter \(w\) i is clipped, and adaptive differential privacy noise is added to perturb it and then sent to the server for global aggregation, effectively preventing malicious attackers from analyzing the model parameters to obtain user-sensitive information. The specific algorithm pseudocode is shown in Algorithm 1.

[0076]

[0077] S102. The Internet of Things federated learning central server aggregates the received local partial model parameters to obtain global model parameters, uses the Wasserstein distance as a regularization term for the training loss of the local partial model, and updates the local partial model using this regularization term. The Wasserstein distance is used to measure the difference between the global model parameters and the local model parameters.

[0078] The Wasserstein distance is a metric used to measure the similarity between two distributions. Compared with the KL divergence and the JS divergence, it has unique advantages. Even if the support sets of the two distributions have little or no overlap, the Wasserstein distance can still reflect the proximity between the two distributions.

[0079] There are two distributions P r and P g . The Wasserstein distance is the lower bound of the expected value that can be obtained among all possible joint distributions composed of P r and P g , and is expressed as:

[0080]

[0081] where ∏(P r , P g ) is the set composed of all joint distributions of the combination of P r and P g . γ is any element in ∏(P r , P g ), and (x, y) is a real sample sampled from γ.

[0082] Since there are heterogeneous local models in federated learning under non-IID data, it may lead to poor training performance of the global model or even non-convergence. To solve this problem, FedWDP redefines the federated learning optimization problem using the Wasserstein distance and trains the local models individually to improve the model performance. The specific Wasserstein optimization process is as Figure 2 shown. Under non-IID data, the distances between the global optimal solution and each local optimal solution of federated learning are not equal. In this case, the averaged model will deviate from the global optimal solution, resulting in the global model not being able to converge to its true global optimal solution.

[0083] To solve this problem, in the embodiments of this case, the problem of improving the model performance is transformed into the problem of solving the global loss function. The loss function of the participating party client i can be expressed as:

[0084]

[0085] Introduce the Wasserstein distance as a regularization term to update the loss function of client i to F i ():

[0086] F i (w) = f i (w) + λD(w i , w)

[0087] Among them, w i is the local model parameter, and D(w i , w) is the Wasserstein distance between the local model parameter w i and the global model parameter w. λ is the regularization parameter that controls (w i , w), which can be adjusted according to the convergence of the model to balance the influence of the global model parameter and the importance of the client's local data. Through Wasserstein distance optimization, the convergence decline caused by model heterogeneity in the FL client model training process can be reduced, and the global loss function can be transformed into:

[0088]

[0089]

[0090] Finally, the client model parameters are updated by the gradient descent method:

[0091]

[0092]

[0093] In this way, as Figure 3 shown, by measuring the difference between the global model and the local model through the Wasserstein distance, the global model can be forced to converge to the global optimal solution again.

[0094] Principal Component Analysis (PCA) can be used for feature extraction and data dimensionality reduction. Its idea is to use the feature space transformation of the statistical properties of the dataset to reduce the dimensionality of a dataset with a high data dimension and mutual correlation. After dimensionality reduction by PCA, the original space is transformed into a new principal component space, and each principal component is uncorrelated.

[0095] Suppose the dataset containing N samples where is the feature space and m is the feature dimension. By transformation, the variable space Z = {z 1 , z 2 ,..., z k} is obtained, satisfying k < m and <z i , z j > = 0. The obtained k-dimensional new variable Z can represent most of the information of the m-dimensional original variable X.

[0096] In the embodiments of this case, the principal component analysis method is further used to pre-dimensionally reduce the local dataset, so as to use the pre-dimensionally reduced local dataset to train the local local model in the federated learning process. Among them, using the principal component analysis method to pre-dimensionally reduce the local dataset can be designed to include:

[0097] Extract the feature matrix of the local datasets of each client of the Internet of Things, calculate the covariance matrix of the local datasets of each client according to the feature matrix, update the covariance matrix with Gaussian noise and upload it to the central server of the Internet of Things federated learning;

[0098] The central server of the Internet of Things federated learning aggregates and averages each covariance matrix, performs eigenvalue decomposition on the aggregated and averaged covariance matrix, intercepts several eigenvectors corresponding to the highest eigenvalues and forms a feature space matrix, and transmits the feature space matrix back to the client;

[0099] The client uses the received feature space matrix to dimensionally reduce and update the local dataset feature matrix.

[0100] To improve the performance of Algorithm 1 in high-dimensional datasets, it is improved by combining PCA to pre-dimensionally reduce the local client datasets, so as to improve the performance of the algorithm in high-dimensional datasets. In order to dimensionally reduce the dataset safely, the dimensionality reduction process of PCA combined with differential federated learning, the algorithm pseudocode can be as shown in Algorithm 2, and the specific steps can be summarized as follows: (1) Represent the feature matrix of the local data of the i-th user as: as:

[0101]

[0102] where is the feature space and m is the number of feature dimensions. Calculate the covariance matrix representing the local data features as:

[0103]

[0104] Then add Gaussian noise to the covariance matrix, where the matrix

[0105]

[0106] Each client uploads the covariance matrix with Gaussian noise added

[0107] (2) The central server will aggregate and average all covariance matrices to obtain Then the server performs eigenvalue decomposition on, intercepts the k eigenvectors corresponding to the highest eigenvalues to form a feature space matrix and transmits it back to the client.

[0108] (3) The client uses Calculate the dimensionality-reduced feature matrix X of the local dataset i ′ .

[0109]

[0110]

[0111] Furthermore, an embodiment of the present invention further provides a method for protecting privacy data in a vehicle networking, including the following content:

[0112] Use each vehicle terminal in the vehicle networking as a client participating in federated learning, and use the cloud server as the central server of federated learning to execute the above-mentioned federated learning method, and obtain the target model of each vehicle terminal, so that each vehicle terminal can use the target model to protect the privacy data of the vehicle terminal when the vehicle networking data is collaboratively shared.

[0113] As Figure 4 shown, in the vehicle networking, in order to protect the privacy information of the vehicle, differential privacy perturbation is added to the parameters uploaded by the vehicle locally. In order to perform dimensionality reduction on the local dataset in a secure manner, as Figure 5 shown, use PCA and combine it with differential federated learning for dimensionality reduction processing.

[0114] Furthermore, based on the above method for protecting privacy data in a vehicle networking, an embodiment of the present invention further provides a system for protecting privacy data in a vehicle networking, including: a cloud server and several vehicle terminals communicatively connected to the cloud server. The vehicle terminals and the cloud server realize collaborative data sharing in the vehicle networking through the federated learning method, where

[0115] The vehicle terminal, as a federated learning client, trains the local local model using the local dataset, adds noise perturbation to the parameters of the local local model based on dynamic differential privacy, and uploads the parameters of the local local model to the central server of the Internet of Things federated learning. The local dataset consists of heterogeneous data that is not independently and identically distributed. The dynamic differential privacy uses exponentially decaying Gaussian noise to perturb the parameters of the local local model;

[0116] The cloud server, as the central server of federated learning, aggregates the received parameters of the local local model to obtain global model parameters, uses the Wasserstein distance as a regularization term for the training loss of the local local model, and updates the local local model using this regularization term. The Wasserstein distance is used to measure the difference between the global model parameters and the local model parameters.

[0117] To verify the effectiveness of the solution in this case, the following further explanation is made in combination with theoretical analysis and experimental data:

[0118] 1. To study the feasibility of the algorithm in this case, the following conducts a theoretical analysis on whether the FedWDP algorithm can meet the convergence conditions and the differential privacy requirements of the Gaussian mechanism under non-IID datasets, and proves the convergence and privacy of the algorithm.

[0119] 1. Convergence analysis

[0120] In personalized training, the Wasserstein divergence is used to calculate the proximal point to accelerate convergence. To facilitate the analysis of the convergence of FedWDP, assume that F(·) is L-smooth, and its upper and lower bounds can be expressed as:

[0121]

[0122] where, when l = 0, F(·) is a convex function; when l = σ > 0, F(·) is a σ-strongly convex function; when l = -L, F(·) is a general non-convex function.

[0123] Property: From the fact that F(·) is L-smooth, we can obtain:

[0124] Proof:

[0125]

[0126] Because So:

[0127]

[0128] Convex function convergence: When l = 0, F(·) is a convex function, and F(·) is sublinearly convergent

[0129] Proof:

[0130] Since F(·) is a convex function, according to the property, we can obtain:

[0131]

[0132] Because,

[0133]

[0134] And So:

[0135]

[0136] We get:

[0137]

[0138] Convergence of σ-strongly convex function: When l = σ > 0, F(·) is a σ-strongly convex function, and F(·) converges linearly

[0139] Proof:

[0140] Since F(·) is a σ-strongly convex function, according to the property, we have:

[0141]

[0142] So the σ-strongly convex function that is L-smooth satisfies:

[0143] We get:

[0144] After T iterations, we have:

[0145] Because So:

[0146] Convergence of general non-convex function: When l = -L, F(·) is a general non-convex function, and F(·) converges sub-linearly

[0147] Proof:

[0148] Since F(·) is a general non-convex function, we have: ||▽F(w k )|| ≤ ∈

[0149] According to the property, we get:

[0150] After T iterations, we have:

[0151]

[0152] We get:

[0153]

[0154] 2. Privacy analysis

[0155] Differential privacy protection is applied to the local model parameters using the Gaussian mechanism. Next, the privacy of the model will be analyzed. For two adjacent datasets Function mechanism And the output o, Through The resulting difference, i.e., the privacy loss Can be expressed as:

[0156]

[0157] mechanism The condition for satisfying (∈,δ)-differential privacy is Then let The differential privacy condition formula can be transformed into an inequality:

[0158]

[0159] Then at the mechanism satisfies (∈,δ)-differential privacy.

[0160] Proof:

[0161]

[0162] Since the probability is always positive,

[0163]

[0164] To ensure bounded by ∈ and with probability at least 1 - δ, then there is:

[0165]

[0166] Let

[0167] Because it is required

[0168] Because Let There will be:

[0169] Because ∈ ≤ 1 and c ≥ 1, so there is

[0170]

[0171] According to the properties of the logarithmic function So There will be:

[0172] II. To verify the effectiveness of the FedWDP algorithm for Non-IID data, the influence of each parameter on the algorithm accuracy is verified through experiments, and the FedWDP and PCA-FedWDP algorithms are compared with the FedProx, FedAvg, and DP-FedAvg algorithms.

[0173] The datasets used in the experiment are the MNIST and CIFAR10 datasets, which are widely used in current federated learning algorithms. MNIST is a handwritten character recognition dataset. The input samples are 28×28 single-channel images with a data dimension of 784. The dataset includes a total of 60,000 training data and 10,000 test data, all of which are handwritten digit images. CIFAR10 is an image classification recognition dataset. The images are divided into 10 categories. The input data samples are 32×32 color images with a data dimension of 3072. The dataset includes a total of 50,000 training data and 10,000 test data. In addition, the number of users is set to 100 and the training dataset is evenly divided, and the server holds all the datasets. The neural network used for model training is CNN, activated by the ReLU and Softmax functions, and the gradient clipping threshold is set to 0.1.

[0174] 1. Analysis of the impact of privacy budget ∈ on the model

[0175] When introducing differential privacy in federated learning, it is necessary to balance the privacy budget and the model accuracy. To balance the model accuracy and the privacy protection effect, the model accuracy of the FedWDP model of the proposed solution in this case under different privacy budgets is tested on the MNIST and CIFAR10 datasets. The experimental results are shown in Table 1 and Figure 6 as shown.

[0176] Table 1 Accuracy of FedWDP on MNIST and CIFAER10 datasets under different privacy budgets

[0177]

[0178] As shown in Table 1 and Figure 6 the experimental results show that the FedWDP model of the proposed solution in this case can also achieve a high accuracy in the case of a low budget. At the same time, as the privacy budget increases, the model accuracy continuously improves. In the case where the differential privacy noise is infinite, the accuracy of the FedWDP model tested on the MNIST and CIFAR10 datasets reaches 95.4% and 54.3%.

[0179] 2. Analysis of the impact of decay rate κ on the model

[0180] In the FedWDP model, to reduce the accumulation of privacy costs and improve the model accuracy, the noise added in each round is exponentially decayed, and the decay rate κ is defined to control the noise decay rate. The experimental results are shown in Table 2 and Figure 7 as shown.

[0181] Table 2 Accuracy of FedWDP on MNIST and CIFAER10 datasets under the noise decay rate

[0182]

[0183] As shown in Table 2 and Figure 7 the experimental results show that with the increase of the decay rate, the FedWDP model of the proposed solution can improve the model accuracy and the training efficiency of the model.

[0184] 3. Analysis of the influence of the data dimension reduction ratio θ on the model

[0185] After the dataset is dimensionally reduced by PCA, the ratio of the dimension of the new dataset to the dimension of the original dataset is defined as θ. Under the same privacy budget, the MNIST dataset is dimensionally reduced, and θ is set to 1, 0.8, 0.5, and 0.1. The experimental results are as Figure 8 shown. As shown by Figure 8 the experimental results, reducing θ to a certain extent will improve the model accuracy, but when θ is reduced to a certain threshold, the model accuracy will decline. This may be because a certain degree of data dimension reduction can weaken the influence of noise and improve the generalization ability of the model, thereby improving the model accuracy; however, too low dimension reduction will lose important information in the original dataset, resulting in a decrease in the test accuracy.

[0186] 4. ∈ 1 and ∈ 2 Ratio analysis of the influence on the model

[0187] According to the sequential composability satisfied by ∈ 1 and ∈ 2 the overall privacy budget of the algorithm is ∈ = ∈ 1 + ∈ 2 , and the experimental results are as Figure 9 shown. The experiment shows that when ∈ is fixed, by adjusting the ratio of ∈ 1 and ∈ 2 , it is found that when ∈ 1 : ∈ 2 = 0.5, the accuracy reaches the highest at 87.3%. Decreasing or increasing the ratio will result in a decline in accuracy. This may be because increasing or decreasing the ratio will make either ∈ 1 or ∈ 2 too low, which will damage the usability of the model.

[0188] 5. Comparative experiment analysis

[0189] To verify the effectiveness of the PCA-FedWDP algorithm under Non-IID data, the FedProx, FedAvg, and DP-FedAvg algorithms under the same privacy conditions are compared. The experimental results are shown in Table 3 and Figure 10 shown.

[0190] Table 3 Comparison of the accuracies of various algorithms on the MNIST and CIFAER10 datasets

[0191]

[0192] As can be seen from the experimental results in Table 3 and Figure 10 under the same privacy conditions, the algorithm in this case has higher accuracy and efficiency.

[0193] Through the above theoretical analysis and experimental data, it shows that the FedWDP model in the solution of this case can ensure the convergence of the model and the privacy of data under the cooperation of Internet of Things data, and can ensure the model accuracy on the premise of realizing privacy protection.

[0194] Unless otherwise specifically stated, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0195] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0196] The units and method steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation is not considered to exceed the scope of the present invention.

[0197] Those of ordinary skill in the art can understand that all or part of the steps in the above method can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, such as: read-only memory, disk or optical disc, etc. Optionally, all or part of the steps of the above embodiments can also be implemented using one or more integrated circuits. Correspondingly, each module / unit in the above embodiments can be implemented in the form of hardware or in the form of a software function module. The present invention is not limited to any specific form of the combination of hardware and software.

[0198] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present invention, which are used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any technician familiar with the technical field of the present invention can still modify the technical solutions described in the foregoing embodiments, or can easily think of changes, or make equivalent replacements for some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A federated learning method for privacy and heterogeneous data in the Internet of Things, characterized by: A federated learning method is used to obtain a target model for protecting privacy data of IoT devices. The federated learning method includes: Each IoT client trains a local model using a local data set, adds noise perturbations to the local model parameters based on dynamic differential privacy, and uploads the local model parameters to the IoT federated learning central server. The local data set consists of heterogeneous data that are not independent and identically distributed, and the dynamic differential privacy uses exponentially decaying Gaussian noise to perturb the local model parameters. The IoT federated learning central server aggregates the received local model parameters to obtain global model parameters, and uses Wasserstein distance as a regularization term for the local model training loss, and uses the regularization term to update the local model. The Wasserstein distance is used to measure the difference between the global model parameters and the local model parameters.

2. The method for federated learning of IoT privacy and heterogeneous data according to claim 1, characterized in that: Add noise perturbations to local model parameters based on dynamic differential privacy, including: The exponential noise attenuation degree is set by using a dynamic noise coefficient, wherein the dynamic noise coefficient is set according to the attenuation rate, the global model training iteration data, and the local local model training iteration data; During the iterative process of local model training, the exponential noise attenuation degree is added as disturbance noise to the local model training loss, and the local model parameters are updated according to the training loss with the disturbance noise added, so as to upload the updated local model parameters to the IoT federated learning center server for global model update.

3. The method for federated learning of IoT privacy and heterogeneous data according to claim 1 or 2, characterized in that: The process of updating the local model using this regularization term is expressed as: F i (w) = f i (w)+λD(w i ,w), where f i (w) is the loss function of client i as a participant in federated learning, w i is the local model parameter, D(w i ,w) is the local model parameter w i The Wasserstein distance between the global model parameter w and λ is the control (w i ,w) regularization parameter.

4. The method for federated learning of IoT privacy and heterogeneous data according to claim 1, characterized in that: The federated learning method further comprises: The principal component analysis method is used to pre-reduce the dimension of the local dataset so that the local local model training in the federated learning process can be performed using the local dataset after dimension reduction.

5. The method for federated learning of IoT privacy and heterogeneous data according to claim 4, characterized in that: Use principal component analysis to pre-reduce the dimensionality of the local data set, including: Extract the feature matrix of each IoT client's local data set, calculate the covariance matrix of each client's local data set based on the feature matrix, use Gaussian noise to update the covariance matrix and upload it to the IoT federated learning central server; The IoT federated learning central server aggregates and averages each covariance matrix, performs eigenvalue decomposition on the aggregated and averaged covariance matrix, extracts several eigenvectors with the highest corresponding eigenvalues ​​and constructs a feature space matrix, which is then transmitted back to the client. The client uses the received feature space matrix to perform dimension reduction update on the feature matrix of the local dataset.

6. The method for federated learning of IoT privacy and heterogeneous data according to claim 5, characterized in that: The process of dimensionality reduction and updating is expressed as: Where i = 0, ..., N-1, N is the number of IoT clients, X i is the feature matrix of the local data set of the ith IoT client, is the feature space matrix.

7. A method for protecting privacy data in an Internet of Vehicles, characterized in that: Contains the following: Each vehicle terminal in the Internet of Vehicles acts as a client participating in federated learning, and the cloud server acts as a central server for federated learning to execute the federated learning method described in claim 1, and obtains the target model of each vehicle terminal, so that each vehicle terminal can use the target model to protect the privacy data of the vehicle terminal when the Internet of Vehicles data is collaboratively shared.

8. A privacy data protection system for Internet of Vehicles, characterized in that: It includes: a cloud server and several vehicle terminals connected to the cloud server, wherein the vehicle terminals and the cloud server realize data collaborative sharing in the vehicle network through a federated learning method, wherein: The vehicle side, as a federated learning client, uses a local data set to train a local local model, adds noise perturbations to the local local model parameters based on dynamic differential privacy, and uploads the local local model parameters to the IoT federated learning central server. The local data set consists of heterogeneous data that are not independent and identically distributed, and the dynamic differential privacy uses exponentially decaying Gaussian noise to perturb the local local model parameters. The cloud server, as a central server for federated learning, aggregates the received local model parameters to obtain global model parameters, and uses Wasserstein distance as a regularization term for the local model training loss, and uses the regularization term to update the local model. The Wasserstein distance is used to measure the difference between the global model parameters and the local model parameters.

9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 6 can be implemented.

Images