Privacy protection method and device for distributed learning

Abstract

The application provides a privacy protection method and device for distributed learning, and the method comprises the following steps: sending first data to a target client in a distributed system, wherein the first data is used to indicate a target gradient of a second data block corresponding to a first data block in a global model in a local model, the target client is deployed with the local model, and the first data block is a data block with a privacy sensitivity arranged in a front preset number after all data blocks in the global model are arranged in an ascending order of the privacy sensitivity; receiving second data sent by the target client, and updating the global model according to the second data, wherein the second data is obtained by updating the second data block based on the target gradient. By uploading part of parameters in the local model for updating the global model, an attacker cannot extract information in a local data set used for training the local model through the parameters, so as to alleviate the problem of leakage of privacy information of users in the distributed learning.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a privacy protection method and apparatus for distributed learning. Background Technology

[0002] Distributed learning technology can, to a certain extent, ensure the free flow of data under the premise of security and compliance, solving the data silo problem caused by the inadequacy of a single data provider, the prohibition of brute-force exchange, and the unwillingness to contribute value. Distributed learning systems typically consist of multiple clients participating in training. Its essence is that the participants collaborate to complete machine learning tasks. The main process of distributed learning is as follows: clients use their local data to train models, and then use centralized or distributed synchronization aggregation algorithms to aggregate the new model parameters into the global model. During this process, each client only uploads model parameters; the training data does not leave its local machine. Each client can also independently use the model for data prediction, effectively solving the data silo problem and improving data processing capabilities.

[0003] However, this approach still presents significant privacy concerns. Although clients participating in distributed learning do not need to directly exchange training data, the parameters sent by clients can still indirectly lead to privacy breaches: an honest and curious client can use the acquired model gradient information to launch a training data reconstruction attack to recover the real training data from other clients. The specific attack method is as follows: After obtaining the real model gradient sent by a client, the attacker randomly initializes virtual data and labels and inputs them into the current global model to obtain virtual gradients. Then, the attacker optimizes the virtual data and labels to minimize the distance between the virtual and real gradients; after multiple rounds of iterative optimization, an approximate approximation of the client's real data and labels is recovered.

[0004] Therefore, in distributed learning scenarios, how to fully guarantee the availability of the global model while preventing privacy leaks caused by model parameter exchange is an urgent problem to be solved.

[0005] To address the aforementioned privacy issues, various protection techniques have been proposed, such as homomorphic encryption, Secure Multi-Party Computation (MPC), and differential privacy. These schemes all protect user privacy to some extent. Among these protection schemes, homomorphic encryption leverages the homomorphic property, allowing clients to directly aggregate ciphertext parameters to obtain the correct aggregated global model without needing to obtain the plaintext model parameters from other clients. Secure Multi-Party Computation requires participants to send their secret data to a predefined function for collaborative computation. Even if one or more parties are attacked, the participants' data remains undisclosed, ensuring the correctness of the computation. Mohassel et al. proposed the SecureML method, where participants distribute their private data between two non-colluding clients, and these two clients then train a global model on the encrypted joint data of the participants following the MPC protocol. However, this algorithm has high computational complexity, and the communication overhead of encrypted parameters is significant. Differential privacy methods introduce a certain amount of noise perturbation into the model's parameters or gradients by the client, ensuring that the true training data cannot be reconstructed from the perturbed model parameters or input. Due to its strong theoretical support and relatively simple integration with existing distributed learning algorithms, differential privacy has been widely applied. Shokri et al. proposed adding noise to the gradient to achieve differential privacy. However, this algorithm requires a large number of clients to participate in training simultaneously to ensure that the accuracy of the aggregated global model is not significantly affected. Furthermore, differential privacy only theoretically guarantees that member information is not leaked and cannot effectively resist training data reconstruction attacks. Truex et al. combined differential privacy with multi-party computation to reduce noise, ensure a balance between model accuracy and privacy, and resist the threat of collusion among users.

[0006] Homomorphic encryption algorithms have significant limitations in computational performance for engineering implementation, making them difficult to widely apply as model sizes increase. Privacy protection methods based on secure multi-party computation require a certain number of users to simultaneously train and aggregate models online, imposing substantial communication overhead on the client. Existing differential privacy-based protection schemes require adding significant noise to gradients during user model training to ensure privacy, making it difficult to balance accuracy and privacy. While combining differential privacy with secure multi-party computation improves accuracy, the requirement for a certain number of users to simultaneously train and aggregate results in each round of secure multi-party computation makes it unsuitable for scenarios with asynchronous client updates. Summary of the Invention

[0007] The present invention provides a privacy protection method and apparatus for distributed learning, which is used to solve the problem of leakage of user data privacy information in distributed learning in the prior art.

[0008] This invention provides a privacy protection method for distributed learning, applied to a server in a distributed system, wherein the server deploys a global model, including:

[0009] Send first data to the target client in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The target client deploys the local model. The first data block is a data block whose privacy sensitivity is sorted in ascending order by all data blocks in the global model, and the privacy sensitivity is sorted into the first preset number of data blocks.

[0010] The system receives second data sent by the target client and updates the global model based on the second data, which is obtained by updating the second data block based on the target gradient.

[0011] The methods for obtaining the second data include:

[0012] The following iterative process is executed to update the global model until the updated global model converges;

[0013] The parameters corresponding to the first data block in the converged global model are sent to the target client so that the target client can obtain the second data based on the parameters corresponding to the first data block in the converged global model.

[0014] The cyclic process includes:

[0015] The first data block is sent to the target client so that the target client can determine the target parameter corresponding to the second data block in the local model based on the parameter corresponding to the first data block;

[0016] The system receives an updated second data block sent by the target client. Based on the parameters corresponding to the updated second data block, it updates the parameters corresponding to the first data block in the global model to update the global model. The updated second data block is obtained by the target client selecting the target gradient corresponding to the second data block from the gradient of the parameters in the local model based on the received first data, and updating the target parameters corresponding to the second data block based on the target gradient. The gradient of the parameters is obtained by the target client training the local model using the local dataset.

[0017] According to the present invention, a privacy protection method for distributed learning is applied to a server in a distributed system, wherein the server deploys a global model, and the method for obtaining the privacy sensitivity of all data blocks in the global model includes:

[0018] Calculate the privacy sensitivity of each data block in the global model on an auxiliary dataset, the privacy sensitivity being used to characterize the privacy data leakage risk of the data block, the auxiliary dataset being determined based on a public dataset or a local dataset of a privacy-insensitive client.

[0019] According to the present invention, a privacy protection method for distributed learning is applied to a server in a distributed system, wherein the server deploys a global model, and calculates the privacy sensitivity of each data block in the global model on an auxiliary dataset, including:

[0020] The global model is trained based on the auxiliary dataset, and the gradients of the parameters in the global model are obtained;

[0021] The gradient of the data block in the global model is determined based on the gradient of the parameters in the global model.

[0022] Calculate the mean of all samples in the auxiliary dataset;

[0023] The privacy sensitivity of each data block is determined based on the mean.

[0024] According to the present invention, a privacy protection method for distributed learning is applied to a server in a distributed system, wherein the server deploys a global model. When the global model is a neural network model, before calculating the privacy sensitivity of each data block in the global model on an auxiliary dataset, the method further includes:

[0025] Based on the number of neurons in each layer of the global model, the parameters of the global model are sequentially divided into data blocks of the same size, layer by layer. The number of parameters contained in each data block is the same as the greatest common divisor of the number of neurons in each layer.

[0026] According to the present invention, a privacy protection method for distributed learning is applied to a server in a distributed system. The server deploys a global model. The method involves updating the parameters corresponding to the first data block in the global model based on the updated parameters corresponding to the second data block, thereby updating the global model. The method includes:

[0027] Using a weighted average method, the parameters corresponding to the first data block in the global model are updated based on the parameters corresponding to the updated second data block, thereby updating the global model.

[0028] This invention also provides a privacy protection method for distributed learning, applied to a target client in a distributed system, wherein the target client deploys a local model, including:

[0029] The system receives first data sent by a server in the distributed system. The first data is used to indicate the target gradient of a second data block in the local model that corresponds to a first data block in the global model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0030] Send second data to the server so that the server updates the global model based on the second data, wherein the second data is obtained by updating the second data block based on the target gradient, and the server deploys the global model;

[0031] The methods for obtaining the second data include:

[0032] Execute the following iterative process until the updated global model converges;

[0033] Based on the parameters corresponding to the first data block in the converged global model sent by the server, the second data is obtained;

[0034] The cyclic process includes:

[0035] Obtain the first data block from the server, and determine the target parameters corresponding to the second data block in the local model based on the parameters corresponding to the first data block;

[0036] The local model is trained using a local dataset to obtain the gradients of the parameters in the local model;

[0037] Based on the first data, select the target gradient corresponding to the second data block from the gradient of the parameters, and update the target parameters corresponding to the second data block based on the target gradient;

[0038] The updated second data block is sent to the server so that the server updates the parameters corresponding to the first data block in the global model based on the parameters corresponding to the updated second data block.

[0039] The present invention also provides a privacy protection device for distributed learning, which is applied to a server in a distributed system. The server is deployed with a global model, including: a first sending module and a first receiving module.

[0040] The first sending module is used to send first data to the target client in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The target client deploys the local model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0041] The first receiving module is used to receive the second data sent by the target client and update the global model according to the second data, wherein the second data is obtained by updating the second data block based on the target gradient;

[0042] The methods for obtaining the second data include:

[0043] The following iterative process is executed to update the global model until the updated global model converges;

[0044] The parameters corresponding to the first data block in the converged global model are sent to the target client so that the target client can obtain the second data based on the parameters corresponding to the first data block in the converged global model.

[0045] The cyclic process includes:

[0046] The first data block is sent to the target client so that the target client can determine the target parameter corresponding to the second data block in the local model based on the parameter corresponding to the first data block;

[0047] The system receives an updated second data block sent by the target client. Based on the parameters corresponding to the updated second data block, it updates the parameters corresponding to the first data block in the global model to update the global model. The updated second data block is obtained by the target client selecting the target gradient corresponding to the second data block from the gradient of the parameters in the local model based on the received first data, and updating the target parameters corresponding to the second data block based on the target gradient. The gradient of the parameters is obtained by the target client training the local model using the local dataset.

[0048] The present invention also provides a privacy protection device for distributed learning, applied to a target client in a distributed system, the target client having a local model deployed thereon, including: a second receiving module and a second sending module;

[0049] The second receiving module is used to receive first data sent by the server in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0050] The second sending module is used to send second data to the server so that the server updates the global model based on the second data. The second data is obtained by updating the second data block based on the target gradient. The server deploys the global model.

[0051] The methods for obtaining the second data include:

[0052] Execute the following iterative process until the updated global model converges;

[0053] Based on the parameters corresponding to the first data block in the converged global model sent by the server, the second data is obtained;

[0054] The cyclic process includes:

[0055] Obtain the first data block from the server, and determine the target parameters corresponding to the second data block in the local model based on the parameters corresponding to the first data block;

[0056] The local model is trained using a local dataset to obtain the gradients of the parameters in the local model;

[0057] Based on the first data, select the target gradient corresponding to the second data block from the gradient of the parameters, and update the target parameters corresponding to the second data block based on the target gradient;

[0058] The updated second data block is sent to the server so that the server updates the parameters corresponding to the first data block in the global model based on the parameters corresponding to the updated second data block.

[0059] The present invention also provides an electronic device, including a processor and a memory storing a computer program, wherein the processor, when executing the program, implements the privacy protection method for distributed learning as described above.

[0060] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the privacy protection method for distributed learning as described above.

[0061] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the privacy protection method for distributed learning as described above.

[0062] The privacy protection method and apparatus for distributed learning provided by this invention update the global model deployed on the server by uploading some parameters from the local model deployed on the target client. This prevents attackers from obtaining information about the local dataset used to train the local model by extracting the parameters, thereby mitigating the problem of user privacy information leakage in distributed learning. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0064] Figure 1 This is one of the flowcharts illustrating the privacy protection method for distributed learning provided by this invention;

[0065] Figure 2 This is the second flowchart illustrating the privacy protection method for distributed learning provided by this invention.

[0066] Figure 3 This is the third flowchart of the privacy protection method for distributed learning provided by this invention;

[0067] Figure 4 This is a schematic diagram of the process for calculating privacy sensitivity provided by the present invention;

[0068] Figure 5 This is a schematic diagram of the process of training a local model on a client side, as provided by the present invention.

[0069] Figure 6 This is a schematic diagram of a privacy protection device for federated learning provided by the present invention;

[0070] Figure 7 This is one of the structural schematic diagrams of the privacy protection device for distributed learning provided by the present invention;

[0071] Figure 8 This is the second schematic diagram of the privacy protection device for distributed learning provided by the present invention;

[0072] Figure 9 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0074] This invention provides a privacy protection method for distributed learning, which effectively balances data privacy, model performance, and system overhead. Addressing the issues of high computational complexity, large communication overhead, and difficulty in scalability and application in existing distributed learning privacy protection methods, this invention improves the way global model parameters are passed during distributed learning. While ensuring data privacy, it reduces the impact on the accuracy of the global model, thus improving the performance of the privacy protection method. To address the communication overhead caused by data exchange between clients, this invention divides the global model parameters into data blocks and uploads only a portion of these blocks to reduce communication overhead between clients. To address the issue of privacy leakage of raw client data, this invention proposes a method for calculating the privacy sensitivity of data blocks to quantify the risk of gradient leakage of privacy information. A subset is selected from all the data blocks into which parameters are divided to reduce the sensitivity of uploaded parameters. Before parameters leave the client's data domain, the invention resolves the data leakage problem caused by parameters, maximizing the protection of client data privacy. The specific implementation is as follows:

[0075] Figure 1 This is one of the flowcharts illustrating the privacy protection method for distributed learning provided by this invention, such as... Figure 1 As shown, the method includes:

[0076] Step 110: Send first data to the target client in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The target client deploys the local model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0077] Step 120: Receive the second data sent by the target client, and update the global model according to the second data, wherein the second data is obtained by updating the second data block based on the target gradient;

[0078] The methods for obtaining the second data include:

[0079] The following iterative process is executed to update the global model until the updated global model converges;

[0080] The parameters corresponding to the first data block in the converged global model are sent to the target client so that the target client can obtain the second data based on the parameters corresponding to the first data block in the converged global model.

[0081] The cyclic process includes:

[0082] The first data block is sent to the target client so that the target client can determine the target parameter corresponding to the second data block in the local model based on the parameter corresponding to the first data block;

[0083] The system receives an updated second data block sent by the target client. Based on the parameters corresponding to the updated second data block, it updates the parameters corresponding to the first data block in the global model to update the global model. The updated second data block is obtained by the target client selecting the target gradient corresponding to the second data block from the gradient of the parameters in the local model based on the received first data, and updating the target parameters corresponding to the second data block based on the target gradient. The gradient of the parameters is obtained by the target client training the local model using the local dataset.

[0084] It should be noted that the execution entity of the above method can be a server in a distributed system, which deploys a global model, which can be a neural network model.

[0085] Optionally, the privacy protection method for distributed learning provided by the present invention is for machine learning tasks in a distributed system. The distributed system consists of multiple clients, one of which is randomly selected as a server, and the server randomly selects multiple clients as target clients. In the distributed system, all clients except the one acting as the server have local models deployed. The local model can be a neural network model that is the same as the global model.

[0086] The server sends first data to the target client in the distributed system. This first data can be specifically used to indicate the target gradient of the second data block in the local model of the target client, which corresponds to the first data block in the global model. The first data block can be specifically selected by arranging the privacy sensitivity of all data blocks in the global model in ascending order, and then selecting the first preset number of data blocks with the highest privacy sensitivity. For example, the first k% of the preset number of data blocks can be selected as the first data block.

[0087] The second data block can be determined based on the location information of the first data block in the global model. Specifically, based on the layer number of the global model where the first data block is located, a data block of the same layer number in the local model deployed in the target client is found, and this data block is the second data block corresponding to the first data block.

[0088] The server receives the second data sent by the target client and updates the global model based on the second data. Specifically, the second data can be obtained by updating the second data block according to the target gradient.

[0089] Optionally, the second data may specifically be the parameters corresponding to the first data block in the converged global model, which may be obtained in the following way:

[0090] The following loop process is executed to update the global model deployed on the server until the updated global model converges. Then, the parameters corresponding to the first data block in the converged global model are sent to the target client.

[0091] The target client sends the parameters corresponding to the first data block as the second data to the server, and the server updates the global model based on the second data.

[0092] This cyclical process can specifically include:

[0093] The server arranges all data blocks in the global model in ascending order of privacy sensitivity, selects the top k% of data blocks as the first data block, and sends a block list containing the location information of these data blocks in the global model to all clients. This location information can specifically be the layer number and sequence number of the data block in the global model.

[0094] The target client finds the first data block from the received block list and initializes the parameters (i.e., target parameters) of the second data block at the same position in the deployed local model according to the parameters corresponding to the first data block, so as to update the local model. The target client is one of the clients randomly selected by the server to participate in the training.

[0095] The target clients each train their own local models using their local datasets to obtain gradient updates for the parameters.

[0096] The target client selects the target gradient corresponding to the second data block from the gradients of the parameters in the local model based on the block list, updates the target parameters of the second data block based on the target gradient, and uploads the updated second data block to the server.

[0097] The server updates the parameters corresponding to the first data block in the global model based on the received updated second data block, until the global model converges.

[0098] Alternatively, the convergence of the global model can be determined in the following way:

[0099] Input the auxiliary dataset into the global model. If the accuracy of the global model remains unchanged or the loss function of the global model reaches the preset value, the result is considered.

[0100] The auxiliary dataset can specifically be a subset of samples that are related to the target task (e.g., classification, prediction) and have a similar distribution (e.g., independent and identically distributed) selected by the server from a public dataset or a local dataset corresponding to a client that is not sensitive to privacy. The dimensions of the auxiliary dataset are then standardized to fit the global model input.

[0101] Existing schemes based on secure multi-party computation protocols require a certain number of clients to simultaneously train and aggregate the model online in order to correctly obtain the aggregation results of the global model. Therefore, they are not suitable for updates in asynchronous distributed learning. In this invention, each client can participate in training completely independently.

[0102] The privacy protection method for distributed learning provided by this invention updates the global model deployed on the server by uploading some parameters from the local model deployed on the target client. This prevents attackers from obtaining information about the local dataset used to train the local model by extracting the parameters, thereby mitigating the problem of user privacy information leakage in distributed learning.

[0103] Furthermore, in one embodiment, the method for obtaining the privacy sensitivity of all data blocks in the global model may specifically include:

[0104] Calculate the privacy sensitivity of each data block in the global model on an auxiliary dataset, the privacy sensitivity being used to characterize the privacy data leakage risk of the data block, the auxiliary dataset being determined based on a public dataset or a local dataset of a privacy-insensitive client.

[0105] Optionally, the server randomly initializes a global model and calculates the privacy sensitivity of each data block in the global model on an auxiliary dataset. This includes, but is not limited to, calculating the p-norm of the Jacobian determinant of each data block in the global model on each sample in the auxiliary dataset, and calculating the privacy sensitivity of the data block based on the p-norm to measure the risk of privacy information leakage of the data block. The auxiliary dataset is determined based on a public dataset or a local dataset of a privacy-insensitive client.

[0106] The privacy protection method for distributed learning provided by this invention utilizes privacy sensitivity to quantify the risk of privacy information leakage in data locally on the client side, ensuring that the processed data is less likely to leak model parameters containing privacy information. This prevents external attackers or curious clients from inferring the original data of other clients. By using the Jacobian determinant to measure the privacy sensitivity of parameters, and by reducing the privacy sensitivity of parameters exchanged between clients, the risk of privacy leakage of client training data is reduced.

[0107] Furthermore, in one embodiment, the method is applied to a server in a distributed system, the server deploying a global model, and calculating the privacy sensitivity of each data block in the global model, including:

[0108] The global model is trained based on the auxiliary dataset, and the gradients of the parameters in the global model are obtained;

[0109] The gradient of the data block in the global model is determined based on the gradient of the parameters in the global model.

[0110] Calculate the mean of all samples in the auxiliary dataset;

[0111] The privacy sensitivity of each data block is determined based on the mean.

[0112] Optionally, using the auxiliary dataset D S Train the global model and obtain the gradient G of the parameters in the trained global model, where the gradient of the b-th data block (i.e., the data block located in the b-th layer of the global model) is g. b .

[0113] Using auxiliary dataset D s Calculate the Jacobian determinant of the gradient of the b-th data block.

[0114] The method for calculating the determinant of Jacobi is as follows:

[0115]

[0116] Where θ is the parameter of the global model, θ b θ is the parameter corresponding to the b-th data block, y is the true label of data X, and l(X,y,θ) is the loss function calculated after data X is tested by the global model.

[0117] For auxiliary dataset D S Each sample X in i Calculate the Jacobian determinant of each data block. The p-norm of the sample is calculated, and the mean of all samples is used as the privacy sensitivity of each data block.

[0118]

[0119] Where m is the auxiliary dataset D S The number of samples in the data, the p-norm is chosen to be the F-norm, the F-norm refers to the matrix... The square root of the sum of the squares of the absolute values ​​of all elements in the equation.

[0120] This invention reduces the amount of parameters exchanged between clients by transmitting data blocks that partially meet privacy requirements. This lowers the communication and computational overhead during data transmission. Existing differential privacy-based protection methods only theoretically prevent attackers from inferring whether a client participated in training, but cannot defend against reconstruction attacks targeting the training data itself. This invention, by reducing the amount of privacy information contained in the model parameters transmitted between clients, ensures that even if parameters are leaked, the local training data cannot be reconstructed by attackers.

[0121] Furthermore, in one embodiment, before calculating the privacy sensitivity of each data block in the global model on the auxiliary dataset, the method may further include:

[0122] Based on the number of neurons in each layer of the global model, the parameters of the global model are sequentially divided into data blocks of the same size, layer by layer. The number of parameters contained in each data block is the same as the greatest common divisor of the number of neurons in each layer.

[0123] Optionally, since the global model uses a neural network model, which consists of multiple stacked layers, each layer is composed of multiple parallel neuron nodes, and the weights of these neuron nodes together constitute the parameters of the global model, the greatest common divisor of the number of neurons in each layer is calculated. The parameters of the global model in the server are then divided into data blocks of the same size layer by layer in sequence, with each block containing the same number of parameters as the calculated greatest common divisor.

[0124] This invention divides the parameters in the global model into fine-grained units of data blocks, eliminating the impact of uneven parameter numbers on privacy measurement results and ensuring the reliability of privacy measurement results and the usability of the global model.

[0125] Further, in one embodiment, updating the parameters corresponding to the first data block in the global model based on the updated parameters corresponding to the second data block, in order to update the global model, may specifically include:

[0126] Using a weighted average method, the parameters corresponding to the first data block in the global model are updated based on the parameters corresponding to the updated second data block, thereby updating the global model.

[0127] Optionally, the present invention may employ a weighted average method to update the parameters corresponding to the first data block in the global model based on the updated parameters corresponding to the second data block, thereby updating the global model.

[0128] The weighted average aggregation method specifically includes the following steps: For each client's uploaded second data block, the server calculates the proportion of each client's local data volume to the sum of all clients' local data volumes, uses this as the weight of each client's second data block, and performs a weighted average on the parameters corresponding to the second data blocks uploaded by all clients. Then, it adds this average to the parameters corresponding to the first data block in the global model to obtain the updated global model.

[0129] This invention provides a privacy protection method for distributed learning, improving the way global model parameters are passed during distributed learning. While ensuring data privacy, it reduces the impact on the accuracy of the global model, improving the performance of the privacy protection method. Addressing the communication overhead caused by data exchange between clients, this invention divides the global model parameters into data blocks and uploads only a portion of these blocks to reduce communication overhead between clients. Regarding the issue of privacy leakage of the client's original data, this invention quantifies the risk of gradient leakage by calculating the privacy sensitivity of data blocks. A subset is selected from all the parameter data blocks to reduce the sensitivity of uploaded parameters. By resolving the data leakage problem caused by parameters before they leave the client's data domain, this invention maximizes the protection of client data privacy and effectively achieves a balance between data privacy, model performance, and system overhead.

[0130] Figure 2 This is the second flowchart illustrating the privacy protection method for distributed learning provided by this invention, as shown below. Figure 2 As shown, the method includes:

[0131] Step 210: Receive first data sent by the server in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0132] Step 220: Send second data to the server so that the server updates the global model based on the second data. The second data is obtained by updating the second data block based on the target gradient. The server deploys the global model.

[0133] The methods for obtaining the second data include:

[0134] Execute the following iterative process until the updated global model converges;

[0135] Based on the parameters corresponding to the first data block in the converged global model sent by the server, the second data is obtained;

[0136] The cyclic process includes:

[0137] Obtain the first data block from the server, and determine the target parameters corresponding to the second data block in the local model based on the parameters corresponding to the first data block;

[0138] The local model is trained using a local dataset to obtain the gradients of the parameters in the local model;

[0139] Based on the first data, select the target gradient corresponding to the second data block from the gradient of the parameters, and update the target parameters corresponding to the second data block based on the target gradient;

[0140] The updated second data block is sent to the server so that the server updates the parameters corresponding to the first data block in the global model based on the parameters corresponding to the updated second data block.

[0141] It should be noted that the subject executing the above method can be a target client in a distributed system, which has a local model deployed thereon, and this local model can be a neural network model.

[0142] Optionally, the target client receives first data sent by the server in the distributed system. This first data can be specifically used to indicate the target gradient of the second data block in the local model of the target client, which corresponds to the first data block in the global model. The first data block can be specifically selected by arranging the privacy sensitivity of all data blocks in the global model in ascending order, and then selecting the first preset number of data blocks with the highest privacy sensitivity. For example, the first k% of the preset number of data blocks can be selected as the first data block.

[0143] The second data block can be determined based on the location information of the first data block in the global model. Specifically, based on the layer number of the global model where the first data block is located, a data block of the same layer number in the local model deployed in the target client is found, and this data block is the second data block corresponding to the first data block.

[0144] The target client sends the second data to the server.

[0145] The server updates the global model based on this second data, which can specifically be obtained by updating the second data block based on the target gradient.

[0146] Optionally, the second data may specifically be the parameters corresponding to the first data block in the converged global model, which may be obtained in the following way:

[0147] The following loop process is executed until the updated global model converges. The target client takes the parameters corresponding to the first data block in the converged global model sent by the server as the second data and sends it to the server so that the server can update the global model based on the second data.

[0148] This cyclical process can specifically include:

[0149] The server arranges all data blocks in the global model in ascending order of privacy sensitivity, selects the top k% of data blocks as the first data block, and sends a block list containing the location information of these data blocks in the global model to all clients. This location information can specifically be the layer number of the data block in the global model.

[0150] The target client finds the first data block from the received block list and initializes the parameters (i.e., target parameters) of the second data block at the same position in the deployed local model according to the parameters corresponding to the first data block, so as to update the local model. The target client is one of the clients randomly selected by the server to participate in the training.

[0151] The target clients each train their own local models using their local datasets to obtain gradient updates for the parameters.

[0152] The target client selects the target gradient corresponding to the second data block from the gradients of the parameters in the local model based on the block list, updates the target parameters of the second data block based on the target gradient, and uploads the updated second data block to the server.

[0153] The server updates the parameters corresponding to the first data block in the global model based on the received updated second data block, until the global model converges.

[0154] Alternatively, the server can determine whether the global model has converged in the following way:

[0155] Input the auxiliary dataset into the global model. If the accuracy of the global model remains unchanged or the loss function of the global model reaches the preset value, the result is considered.

[0156] The auxiliary dataset can specifically be a subset of samples that are related to the target task (e.g., classification, prediction) and have a similar distribution (e.g., independent and identically distributed) selected by the server from a public dataset or a local dataset corresponding to a client that is not sensitive to privacy. The dimensions of the auxiliary dataset are then standardized to fit the global model input.

[0157] The privacy protection method for distributed learning provided by this invention can also be applied to scenarios in federated learning where the server and client need to protect the exchanged parameter updates. This is implemented collaboratively by the client and server in each training round. On the server's global model, the risk of privacy information leakage from parameter data blocks is measured using a privacy sensitivity metric. The client only uploads privacy-insensitive data blocks from its local model parameters, thus ensuring the privacy and security of local data while reducing communication overhead.

[0158] Figure 3 This is the third flowchart of the privacy protection method for distributed learning provided by this invention, as shown below. Figure 3 As shown, it includes:

[0159] Step 11, Auxiliary Dataset Preprocessing: Randomly select a client as the server. The server selects a subset of samples from the public dataset or a client dataset that is not sensitive to privacy. These samples are related to the target task and have a similar distribution as the auxiliary dataset. The server then unifies the dimensions of the auxiliary dataset to adapt it to the global model input.

[0160] Step 12, Data Block Division: The neural network model corresponding to the global model consists of multiple stacked layers, each layer composed of multiple parallel neurons. The weights of these neurons collectively form the parameters of the global model. Calculate the greatest common divisor (GCD) of the number of neurons in each layer, and divide the parameters of the global model layer by layer into data blocks of the same size. The number of parameters contained in each block is the same as the calculated GCD.

[0161] Step 13, Calculate Sensitivity: The server randomly initializes a global model, and calculates the p-norm of the Jacobian determinant of each data block on the auxiliary dataset based on this global model. The mean p-norm of all samples in the auxiliary dataset is calculated and used as the privacy sensitivity of the data block to measure the risk of privacy information leakage of the data block.

[0162] Figure 4 This is a schematic diagram of the process for calculating privacy sensitivity provided by the present invention, such as... Figure 4 As shown, step 13 can specifically include:

[0163] Step 131, using auxiliary dataset D S Train the global model and obtain the gradient G of the parameters in the trained global model, where the gradient of the b-th data block (i.e., the data block located in the b-th layer of the global model) is g. b .

[0164] Step 132, using auxiliary dataset D S Calculate the Jacobian determinant of the gradient of the b-th data block.

[0165] The method for calculating the determinant of Jacobi is as follows:

[0166]

[0167] Where θ is the parameter of the global model, θ b θ is the parameter corresponding to the b-th data block, y is the true label of data X, and l(X,y,θ) is the loss function calculated after data X is tested by the global model.

[0168] Step 133, for the auxiliary dataset D S Each sample X in i Calculate the Jacobian determinant of each data block. The p-norm of the sample is calculated, and the mean of all samples is used as the privacy sensitivity of each data block.

[0169]

[0170] Where m is the auxiliary dataset D S The number of samples in the data, the p-norm is chosen to be the F-norm, the F-norm refers to the matrix... The square root of the sum of the squares of the absolute values ​​of all elements in the equation.

[0171] Step 14, Select Data Blocks: Arrange all data blocks in the global model in ascending order of privacy sensitivity, select the top k% of data blocks, and send the block list containing the location information of these data blocks (i.e., the first database) in the global model to all clients.

[0172] Step 15, Client-side model training: Before each round of training begins, a certain number of clients choose to participate in this round of training. The client downloads the latest parameter information of the first data block from the server, updates the target parameters of the corresponding second data block in the local model accordingly, and then trains the local model using the local dataset to obtain the gradient update of the parameters in the local model.

[0173] Figure 5 This is a schematic diagram of the client-side training of a local model provided by the present invention, as shown below. Figure 5 As shown, step 15 can specifically include:

[0174] Step 151: The server randomly selects some clients C = {C1, C2, ... C...} m}, and notify the client to participate in the training.

[0175] Step 152, Client C i Based on the block list in step 14, download the latest parameter information of the first data block from the server, and update the target parameters corresponding to the second data block of the local model accordingly.

[0176] Step 153, Client C i Train a local model using a local dataset to obtain gradient updates for the parameters.

[0177] Step 16, Upload Data Block: Based on the block list received in Step 14, the client participating in this round of training selects the target gradient of the second data block in the local model, updates the second database based on the target gradient, and uploads the updated second data block to the server.

[0178] Step 17, Data Block Update: The server aggregates data blocks from each client using a centralized synchronization method (such as weighted averaging) to update the parameters of the global model data blocks. Repeat steps 15 to 17 until the global model converges.

[0179] For example, Figure 6 This is a schematic diagram of a privacy protection device for federated learning provided by the present invention. This device can be specifically used to execute the privacy protection method for distributed learning provided by the present invention, and is applied to scenarios in federated learning where the server and client need to protect the exchanged parameter updates, including:

[0180] The system includes an auxiliary dataset preprocessing module, a parameter partitioning module, a sensitivity calculation module, a data block selection module, a client-side model training module, a data block upload module, and a parameter update module.

[0181] In the auxiliary dataset preprocessing module, the server selects a subset of samples that are relevant to the target task and have a similar distribution from public datasets or client datasets that are not sensitive to privacy as auxiliary datasets, and unifies their dimensions to adapt to the model input. After the settings are completed, the auxiliary dataset is output as the input to the sensitivity calculation module.

[0182] In the parameter partitioning module, the server divides the parameters of the server model into data blocks of the same size layer by layer in sequence, and uses the list of data blocks as input to the sensitivity calculation module and the client model training module.

[0183] The server randomly initializes a global model in the sensitivity calculation module, and calculates the privacy sensitivity of each data block based on the global model, which is the mean p-norm of the Jacobian determinant over all samples in the auxiliary dataset, and uses it as input to the data block selection module.

[0184] In the data block selection module, the server selects the k% of data blocks with the lowest privacy sensitivity and uses their location information in the model to form a list of low-sensitivity blocks, which serves as input for the data block upload module and the parameter update module.

[0185] In the client-side model training module, the server randomly selects some clients to participate in training. The client downloads the latest parameter information of the data block from the server, updates the parameters of the corresponding data block in its local model accordingly, and then trains its local model using the local dataset to obtain the gradient update of the parameters, which serves as the input to the data block upload module.

[0186] In the data block upload module, the client selects the corresponding data block from the gradient of the local model based on the received block list, and uses it as input to the parameter update module.

[0187] In the parameter update module, the server aggregates the data blocks from each client using a weighted average method to update the global model, and divides the updated parameters into parameter data blocks, which serve as input to the client model training module.

[0188] The privacy protection method for distributed learning provided by this invention updates the global model deployed on the server by uploading some parameters from the local model deployed on the target client. This prevents attackers from obtaining information about the local dataset used to train the local model by extracting the parameters, thereby mitigating the problem of user privacy information leakage in distributed learning.

[0189] The privacy protection device for distributed learning provided by the present invention will be described below. The privacy protection device for distributed learning described below can be referred to in correspondence with the privacy protection method for distributed learning described above.

[0190] Figure 7 This is one of the structural schematic diagrams of a privacy protection device for distributed learning provided by the present invention, applied to a server in a distributed system, wherein the server deploys a global model, such as... Figure 7 As shown, it includes:

[0191] The first transmitting module 710 and the first receiving module 711;

[0192] The first sending module 710 is used to send first data to the target client in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The target client deploys the local model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0193] The first receiving module 711 is used to receive the second data sent by the target client and update the global model according to the second data, wherein the second data is obtained by updating the second data block based on the target gradient;

[0194] The methods for obtaining the second data include:

[0195] The following iterative process is executed to update the global model until the updated global model converges;

[0196] The parameters corresponding to the first data block in the converged global model are sent to the target client so that the target client can obtain the second data based on the parameters corresponding to the first data block in the converged global model.

[0197] The cyclic process includes:

[0198] The first data block is sent to the target client so that the target client can determine the target parameter corresponding to the second data block in the local model based on the parameter corresponding to the first data block;

[0199] The system receives an updated second data block sent by the target client. Based on the parameters corresponding to the updated second data block, it updates the parameters corresponding to the first data block in the global model to update the global model. The updated second data block is obtained by the target client selecting the target gradient corresponding to the second data block from the gradient of the parameters in the local model based on the received first data, and updating the target parameters corresponding to the second data block based on the target gradient. The gradient of the parameters is obtained by the target client training the local model using the local dataset.

[0200] The privacy protection device for distributed learning provided by this invention updates the global model deployed on the server by uploading some parameters from the local model deployed on the target client. This prevents attackers from obtaining information about the local dataset used to train the local model by extracting the parameters, thereby mitigating the problem of user privacy information leakage in distributed learning.

[0201] Figure 8 This is the second schematic diagram of the privacy protection device for distributed learning provided by the present invention, applied to a target client in a distributed system. The target client deploys a local model, such as... Figure 8 As shown, it includes:

[0202] The second receiving module 810 and the second transmitting module 811;

[0203] The second receiving module 810 is used to receive first data sent by the server in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0204] The second sending module 811 is used to send second data to the server so that the server updates the global model according to the second data. The second data is obtained by updating the second data block based on the target gradient. The server deploys the global model.

[0205] The methods for obtaining the second data include:

[0206] Execute the following iterative process until the updated global model converges;

[0207] Based on the parameters corresponding to the first data block in the converged global model sent by the server, the second data is obtained;

[0208] The cyclic process includes:

[0209] Obtain the first data block from the server, and determine the target parameters corresponding to the second data block in the local model based on the parameters corresponding to the first data block;

[0210] The local model is trained using a local dataset to obtain the gradients of the parameters in the local model;

[0211] Based on the first data, select the target gradient corresponding to the second data block from the gradient of the parameters, and update the target parameters corresponding to the second data block based on the target gradient;

[0212] The updated second data block is sent to the server so that the server updates the parameters corresponding to the first data block in the global model based on the parameters corresponding to the updated second data block.

[0213] The privacy protection device for distributed learning provided by this invention updates the global model deployed on the server by uploading some parameters from the local model deployed on the target client. This prevents attackers from obtaining information about the local dataset used to train the local model by extracting the parameters, thereby alleviating the problem of user privacy information leakage in distributed learning.

[0214] Figure 9 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 9 As shown, the electronic device may include a processor 910, a communication interface 911, a memory 912, and a bus 913, wherein the processor 910, the communication interface 911, and the memory 912 communicate with each other via the bus 913. The processor 910 can call logical instructions in the memory 912 to execute the following methods:

[0215] Send first data to the target client in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The target client deploys the local model. The first data block is a data block whose privacy sensitivity is sorted in ascending order by all data blocks in the global model, and the privacy sensitivity is sorted into the first preset number of data blocks.

[0216] The system receives second data sent by the target client and updates the global model based on the second data, which is obtained by updating the second data block based on the target gradient.

[0217] The methods for obtaining the second data include:

[0218] The following iterative process is executed to update the global model until the updated global model converges;

[0219] The parameters corresponding to the first data block in the converged global model are sent to the target client so that the target client can obtain the second data based on the parameters corresponding to the first data block in the converged global model.

[0220] The cyclic process includes:

[0221] The first data block is sent to the target client so that the target client can determine the target parameter corresponding to the second data block in the local model based on the parameter corresponding to the first data block;

[0222] The system receives an updated second data block sent by the target client. Based on the parameters corresponding to the updated second data block, it updates the parameters corresponding to the first data block in the global model to update the global model. The updated second data block is obtained by the target client selecting the target gradient corresponding to the second data block from the gradient of the parameters in the local model based on the received first data, and updating the target parameters corresponding to the second data block based on the target gradient. The gradient of the parameters is obtained by the target client training the local model using the local dataset.

[0223] or,

[0224] The system receives first data sent by a server in the distributed system. The first data is used to indicate the target gradient of a second data block in the local model that corresponds to a first data block in the global model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0225] Send second data to the server so that the server updates the global model based on the second data, wherein the second data is obtained by updating the second data block based on the target gradient, and the server deploys the global model;

[0226] The methods for obtaining the second data include:

[0227] Execute the following iterative process until the updated global model converges;

[0228] Based on the parameters corresponding to the first data block in the converged global model sent by the server, the second data is obtained;

[0229] The cyclic process includes:

[0230] Obtain the first data block from the server, and determine the target parameters corresponding to the second data block in the local model based on the parameters corresponding to the first data block;

[0231] The local model is trained using a local dataset to obtain the gradients of the parameters in the local model;

[0232] Based on the first data, select the target gradient corresponding to the second data block from the gradient of the parameters, and update the target parameters corresponding to the second data block based on the target gradient;

[0233] The updated second data block is sent to the server so that the server updates the parameters corresponding to the first data block in the global model based on the parameters corresponding to the updated second data block.

[0234] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer power supply (which may be a personal computer, server, or network power supply, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0235] Furthermore, this invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when these instructions are executed by a computer, the computer can perform the privacy protection methods for distributed learning provided in the above-described method embodiments, such as including:

[0236] Send first data to the target client in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The target client deploys the local model. The first data block is a data block whose privacy sensitivity is sorted in ascending order by all data blocks in the global model, and the privacy sensitivity is sorted into the first preset number of data blocks.

[0237] The system receives second data sent by the target client and updates the global model based on the second data, which is obtained by updating the second data block based on the target gradient.

[0238] The methods for obtaining the second data include:

[0239] The following iterative process is executed to update the global model until the updated global model converges;

[0240] The parameters corresponding to the first data block in the converged global model are sent to the target client so that the target client can obtain the second data based on the parameters corresponding to the first data block in the converged global model.

[0241] The cyclic process includes:

[0242] The first data block is sent to the target client so that the target client can determine the target parameter corresponding to the second data block in the local model based on the parameter corresponding to the first data block;

[0243] The system receives an updated second data block sent by the target client. Based on the parameters corresponding to the updated second data block, it updates the parameters corresponding to the first data block in the global model to update the global model. The updated second data block is obtained by the target client selecting the target gradient corresponding to the second data block from the gradient of the parameters in the local model based on the received first data, and updating the target parameters corresponding to the second data block based on the target gradient. The gradient of the parameters is obtained by the target client training the local model using the local dataset.

[0244] or,

[0245] The system receives first data sent by a server in the distributed system. The first data is used to indicate the target gradient of a second data block in the local model that corresponds to a first data block in the global model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0246] Send second data to the server so that the server updates the global model based on the second data, wherein the second data is obtained by updating the second data block based on the target gradient, and the server deploys the global model;

[0247] The methods for obtaining the second data include:

[0248] Execute the following iterative process until the updated global model converges;

[0249] Based on the parameters corresponding to the first data block in the converged global model sent by the server, the second data is obtained;

[0250] The cyclic process includes:

[0251] Obtain the first data block from the server, and determine the target parameters corresponding to the second data block in the local model based on the parameters corresponding to the first data block;

[0252] The local model is trained using a local dataset to obtain the gradients of the parameters in the local model;

[0253] Based on the first data, select the target gradient corresponding to the second data block from the gradient of the parameters, and update the target parameters corresponding to the second data block based on the target gradient;

[0254] The updated second data block is sent to the server so that the server updates the parameters corresponding to the first data block in the global model based on the parameters corresponding to the updated second data block.

[0255] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the privacy protection methods for distributed learning provided in the above embodiments, for example including:

[0256] Send first data to the target client in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The target client deploys the local model. The first data block is a data block whose privacy sensitivity is sorted in ascending order by all data blocks in the global model, and the privacy sensitivity is sorted into the first preset number of data blocks.

[0257] The system receives second data sent by the target client and updates the global model based on the second data, which is obtained by updating the second data block based on the target gradient.

[0258] The methods for obtaining the second data include:

[0259] The following iterative process is executed to update the global model until the updated global model converges;

[0260] The parameters corresponding to the first data block in the converged global model are sent to the target client so that the target client can obtain the second data based on the parameters corresponding to the first data block in the converged global model.

[0261] The cyclic process includes:

[0262] The first data block is sent to the target client so that the target client can determine the target parameter corresponding to the second data block in the local model based on the parameter corresponding to the first data block;

[0263] The system receives an updated second data block sent by the target client. Based on the parameters corresponding to the updated second data block, it updates the parameters corresponding to the first data block in the global model to update the global model. The updated second data block is obtained by the target client selecting the target gradient corresponding to the second data block from the gradient of the parameters in the local model based on the received first data, and updating the target parameters corresponding to the second data block based on the target gradient. The gradient of the parameters is obtained by the target client training the local model using the local dataset.

[0264] or,

[0265] The system receives first data sent by a server in the distributed system. The first data is used to indicate the target gradient of a second data block in the local model that corresponds to a first data block in the global model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks.

[0266] Send second data to the server so that the server updates the global model based on the second data, wherein the second data is obtained by updating the second data block based on the target gradient, and the server deploys the global model;

[0267] The methods for obtaining the second data include:

[0268] Execute the following iterative process until the updated global model converges;

[0269] Based on the parameters corresponding to the first data block in the converged global model sent by the server, the second data is obtained;

[0270] The cyclic process includes:

[0271] Obtain the first data block from the server, and determine the target parameters corresponding to the second data block in the local model based on the parameters corresponding to the first data block;

[0272] The local model is trained using a local dataset to obtain the gradients of the parameters in the local model;

[0273] Based on the first data, select the target gradient corresponding to the second data block from the gradient of the parameters, and update the target parameters corresponding to the second data block based on the target gradient;

[0274] The updated second data block is sent to the server so that the server updates the parameters corresponding to the first data block in the global model based on the parameters corresponding to the updated second data block.

[0275] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0276] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer power supply (which may be a personal computer, server, or network power supply, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0277] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A privacy protection method for distributed learning, characterized in that, A server used in a distributed system, wherein the server deploys a global model, including: Send first data to the target client in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The target client deploys the local model. The first data block is a data block whose privacy sensitivity is sorted in ascending order by all data blocks in the global model, and the privacy sensitivity is sorted into the first preset number of data blocks. The system receives second data sent by the target client and updates the global model based on the second data, which is obtained by updating the second data block based on the target gradient. The methods for obtaining the second data include: The following iterative process is executed to update the global model until the updated global model converges; The parameters corresponding to the first data block in the converged global model are sent to the target client so that the target client can obtain the second data based on the parameters corresponding to the first data block in the converged global model. The cyclic process includes: The first data block is sent to the target client so that the target client can determine the target parameter corresponding to the second data block in the local model based on the parameter corresponding to the first data block; The system receives an updated second data block sent by the target client. Based on the parameters corresponding to the updated second data block, it updates the parameters corresponding to the first data block in the global model to update the global model. The updated second data block is obtained by the target client selecting the target gradient corresponding to the second data block from the gradient of the parameters in the local model based on the received first data, and updating the target parameters corresponding to the second data block based on the target gradient. The gradient of the parameters is obtained by the target client training the local model using the local dataset.

2. The privacy protection method for distributed learning according to claim 1, characterized in that, A server used in a distributed system, wherein the server deploys a global model, and the methods for obtaining the privacy sensitivity of all data blocks in the global model include: Calculate the privacy sensitivity of each data block in the global model on an auxiliary dataset, the privacy sensitivity being used to characterize the privacy data leakage risk of the data block, the auxiliary dataset being determined based on a public dataset or a local dataset of a privacy-insensitive client.

3. The privacy protection method for distributed learning according to claim 2, characterized in that, A server applied in a distributed system, wherein the server deploys a global model, calculates the privacy sensitivity of each data block in the global model on an auxiliary dataset, including: The global model is trained based on the auxiliary dataset, and the gradients of the parameters in the global model are obtained; The gradient of the data block in the global model is determined based on the gradient of the parameters in the global model. Calculate the mean of all samples in the auxiliary dataset; The privacy sensitivity of each data block is determined based on the mean.

4. The privacy protection method for distributed learning according to claim 2, characterized in that, In a server deployed in a distributed system, where the global model is a neural network model, the method further includes, before calculating the privacy sensitivity of each data block in the global model on an auxiliary dataset: Based on the number of neurons in each layer of the global model, the parameters of the global model are sequentially divided into data blocks of the same size, layer by layer. The number of parameters contained in each data block is the same as the greatest common divisor of the number of neurons in each layer.

5. The privacy protection method for distributed learning according to claim 1, characterized in that, An application in a distributed system, wherein the server deploys a global model, and the step of updating the parameters corresponding to the first data block in the global model based on the updated parameters corresponding to the second data block to update the global model includes: Using a weighted average method, the parameters corresponding to the first data block in the global model are updated based on the parameters corresponding to the updated second data block, thereby updating the global model.

6. A privacy protection method for distributed learning, characterized in that, A target client applied in a distributed system, wherein the target client has a local model deployed, including: The system receives first data sent by a server in the distributed system. The first data is used to indicate the target gradient of a second data block in the local model that corresponds to a first data block in the global model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks. Send second data to the server so that the server updates the global model based on the second data, wherein the second data is obtained by updating the second data block based on the target gradient, and the server deploys the global model; The methods for obtaining the second data include: Execute the following iterative process until the updated global model converges; Based on the parameters corresponding to the first data block in the converged global model sent by the server, the second data is obtained; The cyclic process includes: Obtain the first data block from the server, and determine the target parameters corresponding to the second data block in the local model based on the parameters corresponding to the first data block; The local model is trained using a local dataset to obtain the gradients of the parameters in the local model; Based on the first data, select the target gradient corresponding to the second data block from the gradient of the parameters, and update the target parameters corresponding to the second data block based on the target gradient; The updated second data block is sent to the server so that the server updates the parameters corresponding to the first data block in the global model based on the parameters corresponding to the updated second data block.

7. A privacy protection device for distributed learning, characterized in that, A server used in a distributed system, the server having a global model deployed thereon, including: a first sending module and a first receiving module; The first sending module is used to send first data to the target client in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The target client deploys the local model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks. The first receiving module is used to receive the second data sent by the target client and update the global model according to the second data, wherein the second data is obtained by updating the second data block based on the target gradient; The methods for obtaining the second data include: The following iterative process is executed to update the global model until the updated global model converges; The parameters corresponding to the first data block in the converged global model are sent to the target client so that the target client can obtain the second data based on the parameters corresponding to the first data block in the converged global model. The cyclic process includes: The first data block is sent to the target client so that the target client can determine the target parameter corresponding to the second data block in the local model based on the parameter corresponding to the first data block; The system receives an updated second data block sent by the target client. Based on the parameters corresponding to the updated second data block, it updates the parameters corresponding to the first data block in the global model to update the global model. The updated second data block is obtained by the target client selecting the target gradient corresponding to the second data block from the gradient of the parameters in the local model based on the received first data, and updating the target parameters corresponding to the second data block based on the target gradient. The gradient of the parameters is obtained by the target client training the local model using the local dataset.

8. A privacy protection device for distributed learning, characterized in that, A target client applied in a distributed system, the target client having a local model deployed, including: a second receiving module and a second sending module; The second receiving module is used to receive first data sent by the server in the distributed system. The first data is used to indicate the target gradient of the second data block in the local model that corresponds to the first data block in the global model. The first data block is a data block whose privacy sensitivity is arranged in ascending order after all data blocks in the global model are sorted. The privacy sensitivity is then arranged in the first preset number of data blocks. The second sending module is used to send second data to the server so that the server updates the global model based on the second data. The second data is obtained by updating the second data block based on the target gradient. The server deploys the global model. The methods for obtaining the second data include: Execute the following iterative process until the updated global model converges; Based on the parameters corresponding to the first data block in the converged global model sent by the server, the second data is obtained; The cyclic process includes: Obtain the first data block from the server, and determine the target parameters corresponding to the second data block in the local model based on the parameters corresponding to the first data block; The local model is trained using a local dataset to obtain the gradients of the parameters in the local model; Based on the first data, select the target gradient corresponding to the second data block from the gradient of the parameters, and update the target parameters corresponding to the second data block based on the target gradient; The updated second data block is sent to the server so that the server updates the parameters corresponding to the first data block in the global model based on the parameters corresponding to the updated second data block.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the privacy protection method for distributed learning as described in any one of claims 1 to 5 or claim 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the privacy protection method for distributed learning as described in any one of claims 1 to 5 or claim 6.

Images