Decentralized learning method based on differential privacy protection for heterogeneous data

Abstract

The present invention belongs to the field of distributed machine learning, and relates to a decentralized learning method based on differential privacy protection for heterogeneous data. It includes: building a decentralized scenario, setting global parameters, and completing user settings; users send model parameters to neighbor users, calculate self-gradients and cross-gradients, clip self-gradients and cross-gradients respectively, add noise to self-gradients and cross-gradients, and pass cross-gradients to neighbors; each user obtains a corrected gradient based on self-gradients and received cross-gradients after similarity correction; update the model, broadcast the updated momentum and model to neighbors, and perform weighted averaging based on the received momentum and model; repeat the above steps until the model of each user converges or reaches the upper limit of the global iteration number. The method disclosed in the present invention obtains a model that effectively processes heterogeneous data within a limited number of learning rounds in a decentralized environment, and protects user data privacy during the learning process to meet differential privacy requirements.

Description

Technical Field

[0001] The present invention belongs to the field of distributed machine learning, and specifically relates to a decentralized learning method based on differential privacy protection for heterogeneous data. Background Art

[0002] With the rapid development of artificial intelligence technology and the popularization of smart devices, a huge Internet of Things architecture has been formed. The number of devices and the amount of data generated in the real network have increased significantly, which has brought great troubles to traditional user training and learning. Distributed learning technologies such as federated learning, with their unique advantages of cross-device collaboration, disperse the huge time and computing power consumption brought by centralized learning, and gradually become the mainstream training method. However, with the continuous growth of data volume and network scale, the demand for central servers in communication, computing and storage capabilities continues to increase, and the pressure on central servers is increasing. Once it fails, it may cause the collapse of the entire distributed system, forming a bottleneck for federated learning. In order to solve the limitations of high-quality central servers, decentralized learning uses cooperation between node neighbors to effectively distribute the training load to each node in the system, effectively alleviating the challenges brought by centralized architecture and meeting the needs of real networks.

[0003] Although studies have shown that decentralized learning can achieve comparable results to federated learning, data heterogeneity is still unavoidable. With the increase in the number of devices, the diversification of participating users, and the increasing personalization of devices, the data differences between users are not independent and identically distributed, which will hinder the convergence of the model and reduce the overall effect. Although many technologies such as FedProx, SCAFFOLD, and FedFTG have solved the problem of data heterogeneity in federated learning, the solution in decentralized learning is not yet mature. Technologies such as cross-gradient aggregation and gradient tracking have made some progress in solving the problem of data heterogeneity in decentralized learning, but these methods are prone to fall into local optimal solutions to a certain extent and are insufficient in fusing global information.

[0004] In addition, the privacy issue of decentralized learning remains a major challenge, because by analyzing the updates exchanged by the client (such as gradients, model parameters), attackers may still be able to infer the client's private information. Research has shown that attackers can recover the user's personal privacy data from the gradients publicly shared by the client. In order to reduce privacy threats in decentralized learning, some studies have applied differential privacy to wireless decentralized learning methods to ensure differential privacy protection in wireless transmission. However, few decentralized learning methods apply differential privacy to data heterogeneity scenarios, and how to balance model performance and privacy protection remains to be solved. Summary of the invention

[0005] In response to the technical problems raised above, the present invention provides a decentralized learning method based on differential privacy protection for heterogeneous data. Through this method, in a decentralized environment with heterogeneous data, each user can learn a model that approximates the global optimality containing global information, and can achieve excellent results in processing non-independent and identically distributed data; at the same time, by introducing noise based on differential privacy when users exchange parameters and gradients, each user's local privacy data can be effectively protected; it can reasonably balance the model effect and the degree of privacy protection.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] The decentralized learning method based on differential privacy protection for heterogeneous data includes the following steps:

[0008] (1) Initialization: Build a decentralized scenario, generate a decentralized network diagram and a network topology of user connection relationships; set global parameters; complete user settings, load data for each user node, and initialize user model parameters and momentum parameters;

[0009] (2) Self-gradient and cross-gradient calculation: The user sends model parameters to neighbor users, calculates self-gradients based on its own data, and calculates cross-gradients based on the model parameters of its neighbors. The self-gradients and cross-gradients are clipped respectively, and noise is added to the self-gradients and cross-gradients according to the differential privacy mechanism. The cross-gradients with added noise are passed to the neighbors.

[0010] (3) Generating a correction gradient: The user calculates the similarity between the self-gradient and the received cross-gradient based on the self-gradient and the received cross-gradient, and generates a correction gradient by weighted correction based on the similarity;

[0011] (4) Momentum parameter and model parameter update: update the momentum parameter according to the correction gradient, use the updated momentum parameter to update the user's local model parameter, broadcast the updated momentum parameter and model parameter to neighbors, receive the momentum parameter and model parameter from neighbors, and perform weighted averaging on the received momentum parameter and model parameter;

[0012] (5) Each user performs multiple rounds of iterations according to steps (2) to (4). After all users have finished learning, the models of all users are averaged to obtain the global model.

[0013] Furthermore, the step (1) specifically includes the following steps:

[0014] (1.1) Building a decentralized scenario: In the decentralized scenario, the user set , including N users in total;

[0015] Decentralized network diagram , where W is the edge weight matrix of the network topology, , the edge weight matrix W satisfies A doubly random matrix, where represents the edge weight between user i and user j, reflecting the degree of association between users i and j. If users i and j are not neighbors, then , the neighbor set of each user i in the set N is expressed as , ;

[0016] (1.2) Set global parameters: set the global maximum number of iterations T and model learning rate , momentum update coefficient , single training data length B, gradient clipping threshold C, privacy budget , gradient noise size , where the gradient noise size ;

[0017] (1.3) Complete user settings: For each user i in the set N, load each user i’s own dataset , initialize the model parameters , Initialize momentum parameters .

[0018] Furthermore, in step (2), each user i sends the model parameters to its neighbors in round t. After receiving the model parameters of all neighbors, the user performs self-gradient calculation, cross-gradient calculation and transmission.

[0019] Furthermore, in step (2), the method for self-gradient calculation includes:

[0020] According to the single training data length B, randomly sample from all user data to obtain a single batch of data , ;

[0021] User i sends model parameters to neighbors in round t , for a single batch of data Compute self-gradients:

[0022] ;

[0023] in is the self-gradient of user i in round t, Express beg The partial guide, is the average cross entropy loss function for a single batch of data, , ;

[0024] For a single batch of data The i-th single data in .

[0025] Furthermore, in step (2), the method for clipping and adding noise to the self-gradient is:

[0026] Self-gradient clipping: ;

[0027] in, is the self-gradient of user i in round t after clipping; is the initial self-gradient of user i in round t; is the initial self-gradient norm; C is the gradient clipping threshold;

[0028] Self-gradient noise: ;

[0029] in, is the self-gradient after adding noise; The variance is of dimensional Gaussian noise matrix; for dimensional identity matrix.

[0030] Furthermore, in step (2), the cross-gradient calculation method includes:

[0031] According to the single training data length B, a single batch of data is randomly sampled from all user data , ;

[0032] User i uses the model parameters received from neighbor j in round t For a single batch of data Calculate cross gradients;

[0033] ;

[0034] in is the cross-gradient of user i to neighbor j in round t, Express beg The partial derivative of

[0035] is the average cross entropy loss function for a single batch of data, , ;

[0036] For a single batch of data The i-th single data in .

[0037] Furthermore, in step (2), the method for clipping and adding noise to the cross gradient is:

[0038] Cross-gradient clipping: ;

[0039] in, is the cross-gradient of user i to neighbor j in round t after cropping; is the initial cross-gradient of user i to neighbor j in round t; is the initial cross-gradient norm, C is the gradient clipping threshold;

[0040] Cross-gradient noise addition: ;

[0041] in, is the cross gradient after noise addition; The variance is of dimensional Gaussian noise matrix; for dimensional identity matrix;

[0042] User i adds the cross gradient after noise Pass it to neighbor j.

[0043] Furthermore, the specific method of step (3) is as follows:

[0044] (3.1) Wait for the current user i to complete the calculation of self-gradient, wait for the current user to receive all neighbor cross-gradients, and generate a cross-gradient set ;

[0045] (3.2) Calculate the similarity factor between each cross gradient and self gradient based on vector similarity , ;

[0046] in, Represents neighbor cross gradient and clipped from gradient The inner product of Represents neighbor cross gradient and clipped from gradient The product of the norms of

[0047] (3.3) For each calculated similarity factor , perform similarity linear mapping:

[0048] ;

[0049] in, Represents the similarity coefficient after mapping, Represents the absolute value of vector similarity, and are two positive coefficients, determining the range of the linear mapping;

[0050] Normalized to get similarity weight :

[0051] ;

[0052] in, is the maximum similarity coefficient after mapping, and the final similarity weights are limited to ;

[0053] (3.4) The cross-gradient is weighted by the decentralized topological edge weights, and the self-gradient correction with similarity weights is used to calculate the corrected gradient of user j relative to user i :

[0054] ;

[0055] in, represents the edge weight between users i and j; for all neighbors The correction gradient of user i is summed to get the final correction gradient of user i .

[0056] Furthermore, the specific method of step (4) is as follows:

[0057] (4.1) Momentum and parameter update: Update the momentum parameter using the corrected gradient:

[0058] ;

[0059] in, is the updated momentum parameter of user i in round t, is the momentum coefficient, which controls the degree of momentum participation and affects the convergence speed of the model; represents the momentum parameter of user i in round t-1;

[0060] Use the updated momentum parameter Update model parameters:

[0061] ;

[0062] in, is the updated model parameter of user i in round t-1, represents the momentum parameter of user i in round t-1; is the model learning rate;

[0063] (4.2) Broadcast the updated momentum parameter to neighbors With model parameters ;

[0064] (4.3) According to the momentum parameters received from all neighbors Momentum Weighting , based on the updated model parameters received from all neighbors Weighting model parameters .

[0065] Furthermore, in step (5), until the given learning round ends or the absolute difference between the training loss before and after the user for 10 consecutive rounds is less than When , the user's learning process ends; after all users have finished learning, the models of all users are averaged to obtain the global model.

[0066] Beneficial technical effects of the present invention: The method disclosed in the present invention can obtain a model for effectively processing heterogeneous data within a limited number of learning rounds in a decentralized environment, and protect user data privacy during the learning process to meet differential privacy requirements.

[0067] Through this method, in a decentralized environment with heterogeneous data, each user can learn a model that is close to the global optimal model and contains global information, and can achieve excellent results in processing non-independent and identically distributed data. At the same time, by introducing noise based on differential privacy when users exchange parameters and gradients, each user's local privacy data can be effectively protected. It can reasonably balance the model effect and the degree of privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0069] Figure 1 A flow chart of a decentralized learning method based on differential privacy protection for heterogeneous data disclosed in an embodiment of the present invention;

[0070] Figure 2 This is a diagram showing the effect of the model loss function with the iteration of rounds in an embodiment of the present invention;

[0071] Figure 3 This is a diagram showing the accuracy of the model on test data in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0073] The present invention provides a decentralized learning method based on differential privacy protection for heterogeneous data. Figure 1 As shown, the following steps are included:

[0074] (1) Initialization: Build a decentralized scenario, generate a decentralized network diagram and a network topology of user connection relationships; set global parameters; complete user settings, load data for each user node, and initialize user model parameters and momentum parameters;

[0075] (2) Self-gradient and cross-gradient calculation: The user sends model parameters to neighbor users, calculates self-gradients based on its own data, and calculates cross-gradients based on the model parameters of its neighbors. The self-gradients and cross-gradients are clipped respectively, and noise is added to the self-gradients and cross-gradients according to the differential privacy mechanism. The cross-gradients with added noise are passed to the neighbors.

[0076] (3) Generating a correction gradient: The user calculates the similarity between the self-gradient and the received cross-gradient based on the self-gradient and the received cross-gradient, and generates a correction gradient by weighted correction based on the similarity;

[0077] (4) Momentum parameter and model parameter update: update the momentum parameter according to the correction gradient, use the updated momentum parameter to update the user's local model parameter, broadcast the updated momentum parameter and model parameter to neighbors, receive the momentum parameter and model parameter from neighbors, and perform weighted averaging on the received momentum parameter and model parameter;

[0078] (5) Each user performs multiple rounds of iterations according to steps (2) to (4). After all users have finished learning, the models of all users are averaged to obtain the global model.

[0079] In this embodiment, the step (1) specifically includes the following steps:

[0080] (1.1) Building a decentralized scenario: In the decentralized scenario, the user set , including N users in total;

[0081] Decentralized network diagram , where W is the edge weight matrix of the network topology, , the edge weight matrix W satisfies A doubly random matrix, where represents the edge weight between user i and user j, reflecting the degree of association between users i and j. If users i and j are not neighbors, then , the neighbor set of each user i in the set N is expressed as , ;

[0082] (1.2) Set global parameters: set the global maximum number of iterations T and model learning rate , momentum update coefficient , single training data length B, gradient clipping threshold C, privacy budget , gradient noise size , where the gradient noise size ;

[0083] The gradient noise size ; The size of gradient noise is inversely proportional to the length of single training data and the privacy budget, and is proportional to the gradient clipping threshold;

[0084] (1.3) Complete user settings: For each user i in the set N, load each user i’s own dataset , initialize the model parameters , Initialize momentum parameters Specifically, the user's own data set is image data or text data; the model in the present invention is a commonly used neural network model, such as a convolutional neural network model, a VGG model, and the like.

[0085] In step (2), each user i sends model parameters to its neighbors in round t After receiving the model parameters of all neighbors, the user performs self-gradient calculation, cross-gradient calculation and transmission.

[0086] In this embodiment, the method for calculating the self-gradient includes:

[0087] According to the single training data length B, randomly sample from all user data to obtain a single batch of data , ;

[0088] User i sends model parameters to neighbors in round t , for a single batch of data Compute self-gradients:

[0089] ;

[0090] in is the self-gradient of user i in round t, Express beg The partial guide, is the average cross entropy loss function for a single batch of data, , ;

[0091] For a single batch of data The i-th single data in .

[0092] In this embodiment, the method of clipping and adding noise to the self-gradient is:

[0093] Self-gradient clipping: ;

[0094] in, is the self-gradient of user i in round t after clipping; is the initial self-gradient of user i in round t; is the initial self-gradient norm; C is the gradient clipping threshold;

[0095] Self-gradient noise: ;

[0096] in, is the self-gradient after adding noise; The variance is of dimensional Gaussian noise matrix; for dimensional identity matrix, where only the elements on the diagonal are 1 and the rest are 0.

[0097] In this embodiment, the cross-gradient calculation method includes:

[0098] According to the single training data length B, a single batch of data is randomly sampled from all user data , ; Specifically, if the task is image classification, a single batch of data It is the image data. If it is a text classification task, a single batch of data It is text data.

[0099] User i uses the model parameters received from neighbor j in round t For a single batch of data Calculate cross gradients;

[0100] ;

[0101] in is the cross-gradient of user i to neighbor j in round t, Express beg The partial derivative of

[0102] is the average cross entropy loss function for a single batch of data, , ;

[0103] For a single batch of data The i-th single data in .

[0104] In this embodiment, the specific method of step (3) is as follows:

[0105] (3.1) Wait for the current user i to complete the calculation of self-gradient, wait for the current user to receive all neighbor cross-gradients, and generate a cross-gradient set ;

[0106] (3.2) Calculate the similarity factor between each cross gradient and self gradient based on vector similarity , ;

[0107] in, Represents neighbor cross gradient and clipped from gradient The inner product of Represents neighbor cross gradient and clipped from gradient The product of the norms of

[0108] (3.3) For each calculated similarity factor , perform similarity linear mapping:

[0109] ;

[0110] in, Represents the similarity coefficient after mapping, Represents the absolute value of vector similarity, and are two positive coefficients, determining the range of the linear mapping;

[0111] Normalized to get similarity weight :

[0112] ;

[0113] in, is the maximum similarity coefficient after mapping, and the final similarity weights are limited to ;

[0114] (3.4) The cross-gradient is weighted by the decentralized topological edge weights, and the self-gradient correction with similarity weights is used to calculate the corrected gradient of user j relative to user i :

[0115] ;

[0116] in, represents the edge weight between users i and j; for all neighbors The correction gradient of user i is summed to get the final correction gradient of user i .

[0117] In this embodiment, the specific method of step (4) is as follows:

[0118] (4.1) Momentum and parameter update: Update the momentum parameter using the corrected gradient:

[0119] ;

[0120] in, is the updated momentum parameter of user i in round t, is the momentum coefficient, which controls the degree of momentum participation and affects the convergence speed of the model; represents the momentum parameter of user i in round t-1 (initial );

[0121] Use the updated momentum parameter Update model parameters:

[0122] ;

[0123] in, is the updated model parameter of user i in round t-1, represents the momentum parameter of user i in round t-1; is the model learning rate;

[0124] (4.2) Broadcast the updated momentum parameter to neighbors With model parameters ;

[0125] (4.3) According to the momentum parameters received from all neighbors Momentum Weighting , based on the updated model parameters received from all neighbors Weighting model parameters .

[0126] , in step (5), until the given learning round ends or the absolute difference between the training loss before and after the user for 10 consecutive rounds is less than When , the user's learning process ends; after all users have finished learning, the models of all users are averaged to obtain the global model.

[0127] In order to evaluate the performance of the method proposed in the present invention, a simulation experiment was conducted on the method proposed in the present invention. A grid topology was used, and the number of users was set to 10 and 20. A handwritten digit recognition dataset was used, and the data heterogeneity state of the real network was simulated according to the Dirichlet distribution. Users participated in the training using the Lenet model. The global maximum number of iterations was set to 150, the model learning rate was 0.003, and the gradient clipping threshold was set to 0. , the model single training data length B = 256, momentum update coefficient , Privacy Budget , gradient noise size The experimental results are as follows Figure 2-Figure 3As shown in the figure; PCCDL is the method proposed by the present invention, and the other methods are comparative methods that process the same scenario as this experiment. DP-DPSGD and MUFFLIATO are existing decentralized learning algorithms based on differential privacy mechanisms, and DP-NETFLEET and DP-CGA are methods obtained by integrating differential privacy with existing decentralized learning algorithms that process heterogeneity. N Users, T The upper limit of round iterations, the model training of the method proposed in this invention can reach Compared with the best comparison method, the present invention can reduce the required convergence rounds by 4% in the scenario of 10 users, and achieve a 2.5% improvement in accuracy; in the scenario of 20 users, it can reduce the required convergence rounds by 15%, and achieve a 4.8% improvement in accuracy.

[0128] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A decentralized learning method based on differential privacy protection for heterogeneous data, characterized by: The steps include: (1) Initialization: Build a decentralized scenario, generate a decentralized network diagram and a network topology of user connection relationships; set global parameters; complete user settings, load data for each user node, and initialize user model parameters and momentum parameters; (2) Self-gradient and cross-gradient calculation: The user sends model parameters to neighbor users, calculates self-gradients based on its own data, and calculates cross-gradients based on the model parameters of its neighbors. The self-gradients and cross-gradients are clipped respectively, and noise is added to the self-gradients and cross-gradients according to the differential privacy mechanism. The cross-gradients with added noise are passed to the neighbors. (3) Generating a correction gradient: The user calculates the similarity between the self-gradient and the received cross-gradient based on the self-gradient and the received cross-gradient, and generates a correction gradient by weighted correction based on the similarity; (4) Momentum parameter and model parameter update: update the momentum parameter according to the correction gradient, use the updated momentum parameter to update the user's local model parameter, broadcast the updated momentum parameter and model parameter to neighbors, receive the momentum parameter and model parameter from neighbors, and perform weighted averaging on the received momentum parameter and model parameter; (5) Each user performs multiple rounds of iterations according to steps (2) to (4). After all users have finished learning, the models of all users are averaged to obtain the global model.

2. The decentralized learning method based on differential privacy protection for heterogeneous data according to claim 1 is characterized in that: The step (1) specifically includes the following steps: (1.1) Building a decentralized scenario: In the decentralized scenario, the user set , including N users in total; Decentralized network diagram , where W is the edge weight matrix of the network topology, , the edge weight matrix W satisfies A doubly random matrix, where represents the edge weight between user i and user j, reflecting the degree of association between users i and j. If users i and j are not neighbors, then , the neighbor set of each user i in the set N is expressed as , ; (1.2) Set global parameters: set the global maximum number of iterations T and model learning rate , momentum update coefficient , single training data length B, gradient clipping threshold C, privacy budget , gradient noise size , where the gradient noise size ; (1.3) Complete user settings: For each user i in the set N, load each user i’s own dataset , initialize the model parameters , Initialize momentum parameters .

3. The decentralized learning method based on differential privacy protection for heterogeneous data according to claim 1 is characterized in that: In step (2), each user i sends the model parameters to its neighbors in round t After receiving the model parameters of all neighbors, the user performs self-gradient calculation, cross-gradient calculation and transmission.

4. The decentralized learning method based on differential privacy protection for heterogeneous data according to claim 3 is characterized in that: In step (2), the method for self-gradient calculation includes: According to the single training data length B, randomly sample from all user data to obtain a single batch of data , ; User i sends model parameters to neighbors in round t , for a single batch of data Compute self-gradients: ； in is the self-gradient of user i in round t, Express beg The partial guide, is the average cross entropy loss function for a single batch of data, , ; For a single batch of data The i-th single data in .

5. The decentralized learning method based on differential privacy protection for heterogeneous data according to claim 3 is characterized in that: In step (2), the method for clipping and adding noise to the self-gradient is: Self-gradient clipping: ; in, is the self-gradient of user i in round t after clipping; is the initial self-gradient of user i in round t; is the initial self-gradient norm; C is the gradient clipping threshold; Self-gradient noise: ; in, is the self-gradient after adding noise; The variance is of dimensional Gaussian noise matrix; for dimensional identity matrix.

6. The decentralized learning method based on differential privacy protection for heterogeneous data according to claim 3 is characterized in that: In step (2), the cross-gradient calculation method includes: According to the single training data length B, a single batch of data is randomly sampled from all user data , ; User i uses the model parameters received from neighbor j in round t For a single batch of data Calculate cross gradients; ； in is the cross-gradient of user i to neighbor j in round t, Express beg The partial derivative of is the average cross entropy loss function for a single batch of data, , ; For a single batch of data The i-th single data in .

7. The decentralized learning method based on differential privacy protection for heterogeneous data according to claim 3 is characterized in that: In step (2), the method for clipping and adding noise to the cross gradient is: Cross-gradient clipping: ; in, is the cross-gradient of user i to neighbor j in round t after cropping; is the initial cross-gradient of user i to neighbor j in round t; is the initial cross-gradient norm, C is the gradient clipping threshold; Cross-gradient noise addition: ; in, is the cross gradient after noise addition; The variance is of dimensional Gaussian noise matrix; for dimensional identity matrix; User i adds the cross gradient after noise Pass it to neighbor j.

8. The decentralized learning method based on differential privacy protection for heterogeneous data according to claim 1, characterized in that: The specific method of step (3) is as follows: (3.1) Wait for the current user i to complete the calculation of self-gradient, wait for the current user to receive all neighbor cross-gradients, and generate a cross-gradient set ; (3.2) Calculate the similarity factor between each cross gradient and self gradient based on vector similarity , ; in, Represents neighbor cross gradient and clipped from gradient The inner product of Represents neighbor cross gradient and clipped from gradient The product of the norms of (3.3) For each calculated similarity factor , perform similarity linear mapping: ； in, Represents the similarity coefficient after mapping, Represents the absolute value of vector similarity, and are two positive coefficients, determining the range of the linear mapping; Normalized to get similarity weight : ； in, is the maximum similarity coefficient after mapping, and the final similarity weights are limited to ; (3.4) The cross-gradient is weighted by the decentralized topological edge weights, and the self-gradient correction with similarity weights is used to calculate the corrected gradient of user j relative to user i : ； in, represents the edge weight between users i and j; for all neighbors The correction gradient of user i is summed to get the final correction gradient of user i .

9. The decentralized learning method based on differential privacy protection for heterogeneous data according to claim 8, characterized in that: The specific method of step (4) is as follows: (4.1) Momentum and parameter update: Update the momentum parameter using the corrected gradient: ； in, is the updated momentum parameter of user i in round t, is the momentum coefficient, which controls the degree of momentum participation and affects the convergence speed of the model; represents the momentum parameter of user i in round t-1; Use the updated momentum parameter Update model parameters: ； in, is the updated model parameter of user i in round t-1, represents the momentum parameter of user i in round t-1; is the model learning rate; (4.2) Broadcast the updated momentum parameter to neighbors With model parameters ; (4.3) According to the momentum parameters received from all neighbors Momentum Weighting , based on the updated model parameters received from all neighbors Weighting model parameters .

10. The decentralized learning method based on differential privacy protection for heterogeneous data according to claim 1, characterized in that: In step (5), the training process continues until the given learning round ends or the absolute difference between the training losses before and after the user for 10 consecutive rounds is less than When , the user's learning process ends; After all users have finished learning, the models of all users are averaged to obtain the global model.

Images