A signal processing method based on non-random privacy-preserving distributed deep learning

Abstract

The present invention discloses a signal processing method based on non-random privacy-preserving distributed deep learning. Non-random privacy-preserving deep learning signal processing can protect private information to a certain extent without significantly increasing the computational and communication burdens. Specifically, in this method, each processor only needs to add an additional privacy quantization operation to protect privacy. This operation involves only a simple judgment operation, and the number of data bits transmitted is less than or equal to the original data. Therefore, it can protect privacy without significantly increasing the computational burden and reducing the communication burden. Non-random privacy-preserving deep learning signal processing can effectively ensure learning accuracy while protecting private information to a certain extent. Specifically, according to line 4 of Algorithm 2, by reasonably selecting the parameters η and s, the corresponding data transmission accuracy can be achieved, thereby protecting the accuracy of distributed deep learning.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and network security technology, and specifically relates to a signal processing method based on non-random privacy-preserving distributed deep learning. Background Art

[0002] In recent years, with the widespread use of big data, the amount of data used to train artificial intelligence models has grown exponentially. The scale and complexity of this data far exceeds the limits of what a single computer can handle, necessitating the design of distributed, large-scale machine learning architectures. Therefore, to advance the development of my country's artificial intelligence field and improve the robustness and efficiency of machine learning systems, there is an urgent need to address the challenges of distributed machine learning in the context of big data, including model building, data sharing, collaborative computing optimization, and performance analysis. Large-scale machine learning systems often require coordinating multiple machines in a network to achieve distributed, autonomous deep learning and complete a given training task. However, in distributed deep learning networks, since the main processors in the network often carry sensitive private information, privacy leaks often occur during data transmission. Therefore, algorithms need to be designed to minimize the leakage of private information.

[0003] Existing privacy-preserving methods for distributed deep learning can be divided into two categories: homomorphic encryption-based methods and random perturbation-based methods. Homomorphic encryption-based methods require encoding data, converting it into ciphertext for transmission and computation, and then decoding it back into usable data. Due to the characteristics of homomorphic encryption algorithms, they often significantly increase the computational and communication burden of the system. Another privacy-preserving method based on random perturbation aims to protect privacy by adding random noise during network training. However, these methods often face a significant trade-off between privacy and training effectiveness: protecting the network's privacy information comes at the expense of training performance.

[0004] Existing homomorphic encryption-based methods require encoding data, converting it into ciphertext for transmission and computation, and then decoding it back to usable data. Due to the characteristics of homomorphic encryption algorithms, they often significantly increase the computational and communication burden of the system. Another privacy-preserving method based on random perturbations aims to protect privacy by adding random noise during network training. However, such methods often face a significant trade-off between privacy and training effectiveness: protecting the network's private information comes at the expense of training performance.

[0005] In contrast, non-random privacy can effectively guarantee the accuracy of distributed deep learning while protecting private information to a certain extent, while also reducing the computational and communication burdens. It is noted that existing research on non-random privacy in distributed deep learning systems is still in its early stages. Therefore, the distributed deep learning method based on non-random privacy protection proposed in this paper is of great significance to the study of data security issues in distributed learning. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a signal processing method based on non-random privacy-preserving distributed deep learning, which can protect privacy information to a certain extent without significantly increasing the computing and communication burdens.

[0007] A signal processing method based on non-random privacy-preserving distributed deep learning, comprising:

[0008] Step 0: Deploy the deep learning model to multiple processors and assign different task threads. Each thread interacts with a communication protocol that satisfies non-random privacy based on a distributed dataset.

[0009] Step 1: Perform non-random privacy distributed deep learning local training on the deep learning model of each processor, including:

[0010] Step 11: First, input the distributed dataset D, initialize the learning rate γ, the non-random privacy parameter η, the total number of training rounds T, the size of each batch of data n, and define the total data size as |D|; where, Indicates the total number of sampling rounds of the data set in each training round; the current model weight is represented as θ i ;

[0011] Step 12: Set training round t=1;

[0012] Step 13: Set the number of sampling rounds k=1;

[0013] Step 14: Randomly sample batches of n data D from the distributed dataset D without replacement t,k ∈D; according to the sampled data D t,k Calculate the loss function L(θ i ,D t,k ) and the corresponding gradient The processor calculates the updated model weights:

[0014]

[0015] Step 15: Conduct non-random private distributed communication with neighboring processors in the distributed network and perform aggregate updates on the model weights. Specifically:

[0016] According to the non-random privacy parameter η, the parameter s is selected so that the inequality η≥2+log210 s The maximum value established;

[0017] For each processor i in the network:

[0018] Computing model weights based on quantized non-random privacy as follows:

[0019]

[0020] Among them, θ max is the maximum value range of each dimension of the model weight, θ min The minimum value of the range of values for each dimension of the model weight;

[0021] Send privacy model weights to neighboring processors in the network

[0022] Receive model weights from each neighbor j Where j∈N i , N i Represents the set of all neighboring processors of processor i; and defines the number of its neighboring processors as |N i |;

[0023] Each processor aggregates all received model weights and updates its own current model weights:

[0024] Step 16: Determine whether k=K. If not, add 1 to k and go to step 14. If yes, go to step 17.

[0025] Step 17: Determine whether t=T is satisfied. If not, add 1 to the value of t; if so, proceed to the next step.

[0026] Step 2: Set the model weight θ i Configured to the deep learning model, each processor processes the input signal accordingly based on the deep learning model.

[0027] Preferably, the data set D is image data, and in step 2, each processor processes the input image signal based on a deep learning model.

[0028] Preferably, the data set D is distributed sensor data, and in step 2, each processor processes the input distributed sensor data based on a deep learning model.

[0029] Preferably, the data set D is frequency domain signal data, and in step 2, each processor processes the input frequency domain signal based on a deep learning model.

[0030] Preferably, the initial model weights are set based on experience or randomly.

[0031] The present invention has the following beneficial effects:

[0032] 1. Non-random privacy-focused deep learning signal processing can protect private information to a certain extent without significantly increasing computational and communication burdens. Specifically, in this method, each processor only needs to add an additional privacy quantization operation to protect privacy. This operation involves only simple judgment calculations, and the number of data bits transmitted is less than or equal to the original data. Therefore, it can protect privacy without significantly increasing computational burdens and reducing communication burdens.

[0033] 2. Non-random privacy-focused deep learning signal processing can effectively ensure learning accuracy while protecting privacy information to a certain extent. Specifically, by properly selecting the parameters η and s, the corresponding degree of data transmission accuracy can be achieved, thereby protecting the accuracy of distributed deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the non-random privacy-preserving distributed training process of the present invention;

[0035] Figure 2 A communication topology diagram for distributed deep learning methods;

[0036] Figure 3 This is a graph showing the training accuracy of different non-random privacy parameters when the learning rate is 0.2 and epoch=50, where: Figure 3 (a) represents the accuracy (90%) when s=2,η=9, Figure 3 (b) shows the accuracy (95%) when s=3,η=12, Figure 3 (c) shows the accuracy (96%) when s = 4, η = 16;

[0037] Figure 4 This is a schematic diagram of the distributed neural network image classification model;

[0038] Figure 5 Schematic diagram of collaborative signal estimation in distributed sensor networks;

[0039] Figure 6 Schematic diagram of robust distributed phase recovery of signals. DETAILED DESCRIPTION

[0040] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0041] This paper provides a non-randomized privacy-preserving deep learning method for distributed datasets. Essentially, the deep learning model is deployed across multiple processors and assigned different task threads. Each thread, based on the distributed dataset, interacts with a non-randomized privacy-preserving communication protocol through local updates, thereby achieving distributed training tasks that meet non-randomized privacy protection.

[0042] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following examples illustrate the implementation environment of the present invention:

[0043] Hardware environment: Computer with Ubuntu system installed, multiple processors (CPU, GPU);

[0044] Software requirements: Python 3.0 or later, PyTorch 1.6 or later, OpenMPI 4.0 or later, and Bluefog interface.

[0045] This paper aims to design a privacy-preserving distributed deep learning method to solve data-driven signal processing problems. It recovers or fits a suitable model from a large amount of data to achieve data-driven image classification and recognition, signal analysis and other tasks. Such signal processing problems can be mathematically transformed into training the optimal model weights θ from the dataset D, and making the expected loss function of the entire model Minimize the problem, where each processor i can only obtain its own private dataset D i ∈D. Since each processor uses the data set D i There is a risk of privacy leakage, and privacy protection mechanisms need to be incorporated into distributed learning methods.

[0046] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0047] Step 0: Configure the deep learning model to multiple processors and assign different task threads. Each thread is based on a distributed dataset and interacts with a communication protocol that satisfies non-random privacy through local updates, thereby achieving distributed training tasks that meet non-random privacy protection.

[0048] Step 1: Perform non-random privacy distributed deep learning local training on the deep learning model of each processor, including:

[0049] Step 11: First, input the distributed dataset D, initialize the learning rate γ, the non-random privacy parameter η, the total number of training rounds T, the size of each batch of data n, and define the total data size as |D|; where, Indicates the total number of sampling rounds of the data set in each training round; the current model weight is represented as θ i , the initial model weights are set based on experience or randomly.

[0050] Step 12: Set training round t=1;

[0051] Step 13: Set the number of sampling rounds k=1;

[0052] Step 14: Randomly sample batches of n data D from the distributed dataset D without replacement t,k ∈D; according to the sampled data D t,k Calculate the loss function L(θ i ,D t,k ) and the corresponding gradient The weights are updated locally according to the appropriate optimizer. For example, the stochastic gradient descent optimizer is used. The processor calculates the updated model weights:

[0053]

[0054] Step 15: Conduct non-random private distributed communication with neighboring processors in the distributed network and perform aggregate updates on the model weights. Specifically:

[0055] According to the non-random privacy parameter η, the parameter s is selected so that the inequality η≥2+log210 s The maximum value established;

[0056] For each processor i in the network:

[0057] Computing model weights based on quantized non-random privacy as follows:

[0058]

[0059] Among them, θ max is the maximum value range of each dimension of the model weight, θ min The minimum value of the range of values for each dimension of the model weight;

[0060] Send privacy model weights to neighboring processors in the network

[0061] Receive model weights from each neighbor j Where j∈N i , N i Represents the set of all neighboring processors of processor i; and defines the number of its neighboring processors as |N i |;

[0062] Each processor aggregates all received model weights and updates its own current model weights:

[0063] Step 16: Determine whether k=K. If not, add 1 to k and go to step 14. If yes, go to step 17.

[0064] Step 17: Determine whether t=T is satisfied. If not, add 1 to the value of t; if so, proceed to the next step.

[0065] In steps 14-16 above, the processor performs distributed training locally. In the code implementation, the distributed dataset is read using the PyTorch `torch.utils.data.distributed.DistributedSampler` statement and the model to be trained is imported. The Bluefog API `bfrun-np 8python train.py` statement is used to run eight local processes representing distributed training on the Bluefog server.

[0066] In step 15 above, according to the non-random privacy distributed communication mechanism, each processor i communicates with its neighbor processor j∈N i Communication is carried out to achieve the aggregation of model weights on each processor in the distributed deep learning system; in the design of the privacy protection mechanism of this process, the Q function is used to segment the value range of the model weights. Given a non-random privacy parameter η, the minimum quantization interval of non-random privacy q≤2 can be obtained. η (θ max -θ min ). For model weights Using quantized non-random privacy processing with precision s, we can obtain the relationship between the non-random privacy parameter η and the precision s: η≥2+log210 s During the communication process, the bf.neighbor_allreduce instruction of the Bluefog interface is used to realize the communication between different processors and the aggregation of model weights. The index topology used is as follows: Figure 2 shown.

[0067] Step 2: Set the model weight θ i Configured to the deep learning model, each processor processes the input signal accordingly based on the deep learning model.

[0068] The effectiveness of the method of the present invention will be tested through experiments below.

[0069] In the simulation experiment, a convolutional neural network is trained using the MNIST dataset. The number of training rounds is set to 50, the size of each batch is set to 64, and the learning rate is set to 0.2. Figure 3 The experimental results shown in the figure show a trade-off between privacy protection and algorithm training performance. When s = 2 and η = 9, the maximum random leakage value is the lowest, resulting in better privacy protection. However, the quantization precision is low at this point, leading to lower training accuracy. Gradually increasing the value of η increases the maximum random leakage value, weakening the privacy protection effect. Meanwhile, higher quantization precision leads to higher training accuracy. Therefore, during training, the trade-off between privacy protection and training performance should be comprehensively considered, and training parameters should be appropriately selected.

[0070] The simulation experiment only uses specific data sets and models, and selects different non-random privacy parameters for comparative experiments.

[0071] Implementation Case 1: Distributed Neural Network Image Classification

[0072] The purpose of distributed neural network image classification is to fit and train the best image classification rules based on a large number of sample images and label information. It can be widely used in license plate recognition, smart workshops, face recognition and other fields. In this problem, the image classification rules to be trained can be represented by a neural network. Figure 4 As shown in FIG, the input image is subjected to a series of operations such as convolution, pooling, activation, and full connection of the neural network to obtain the output classification result. The present invention can be applied to the training and optimization of the neural network model. In step 1, the initial weight of the model is input as θ 0,K , and use the mathematical expression of the neural network model to replace the loss function to be optimized The distributed deep learning method proposed in this invention can realize the distributed neural network image classification task while protecting non-random privacy, and significantly reduce the number of communications and the number of communication bits, thereby improving communication efficiency.

[0073] Implementation Case 2: Distributed Collaborative Signal Estimation in Multi-Sensor Networks

[0074] The goal of distributed cooperative signal estimation in multi-sensor networks is to estimate the true state of the signal source using the data collected online by multiple sensors in the network, such as Figure 5 As shown in Figure 1, this problem is widely used in the context of multi-robot collaborative detection and target positioning. Due to environmental interference and sensor errors, there are often errors in the sensor's observation signal. Therefore, it is necessary to coordinate multiple sensors and collect multiple sets of data for collaborative estimation. Let x be the signal source state to be estimated, y be the actual observation value of the sensor, H i is the transformation matrix from the signal source state to the sensor observation value, which represents the relationship that should be satisfied between the state and the observation value. Further selecting the least squares loss form, the loss function to be optimized can be expressed as follows:

[0075]

[0076] Subsequently, based on the actual sensor observations, the signal source state is continuously estimated online, enabling collaborative signal estimation in distributed sensor networks while preserving privacy. This method can significantly improve communication efficiency between sensors and reduce communication bandwidth congestion, especially in large-scale sensor systems.

[0077] Implementation Case 3: Robust Distributed Phase Recovery of Signals

[0078] In the fields of astronomical observation, X-ray imaging, diffraction optics, etc., the properties of the frequency domain signal collected by the sensor can be expressed in terms of intensity and phase. Phase recovery is to recover the phase information from the intensity measurement information with disturbance, such as Figure 6 The task of recovering the phase x from observations with additive white Gaussian noise can be modeled as minimizing the intensity-based empirical loss:

[0079]

[0080] where w j is a measure of Gaussian white noise, y j The actual observed value. This problem is a non-convex problem in some cases, so it is complicated to handle. The method proposed in this invention can solve the problem of robust distributed phase recovery of signals when the problem is non-convex.

[0081] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A signal processing method based on non-random privacy-preserving distributed deep learning, characterized in that: include: Step 0: Deploy the deep learning model to multiple processors and assign different task threads. Each thread interacts with a communication protocol that satisfies non-random privacy based on a distributed dataset. Step 1: Perform non-random privacy distributed deep learning local training on the deep learning model of each processor, including: Step 11: First, input the distributed dataset D, initialize the learning rate γ, the non-random privacy parameter η, the total number of training rounds T, the size of each batch of data n, and define the total data size as |D|; where, Indicates the total number of sampling rounds of the data set in each training round; the current model weight is represented as θ i ; Step 12: Set training round t=1; Step 13: Set the number of sampling rounds k=1; Step 14: Randomly sample batches of n data D from the distributed dataset D without replacement t,k ∈D; according to the sampled data D t,k Calculate the loss function L(θ i ,D t,k ) and the corresponding gradient The processor calculates the updated model weights: Step 15: Conduct non-random private distributed communication with neighboring processors in the distributed network and perform aggregate updates on the model weights. Specifically: According to the non-random privacy parameter η, the parameter s is selected so that the inequality η≥2+log210 s The maximum value established; For each processor i in the network: Computing model weights based on quantized non-random privacy as follows: Among them, θ max is the maximum value range of each dimension of the model weight, θ min The minimum value of the range of values for each dimension of the model weight; Send privacy model weights to neighboring processors in the network Receive model weights from each neighbor j Where j∈N i , N i Represents the set of all neighboring processors of processor i; and defines the number of its neighboring processors as |N i |; Each processor aggregates all received model weights and updates its own current model weights: Step 16: Determine whether k=K. If not, add 1 to k and go to step 14. If yes, go to step 17. Step 17: Determine whether t=T is satisfied. If not, add 1 to the value of t; if so, proceed to the next step. Step 2: Set the model weight θ i Configured to the deep learning model, each processor processes the input signal accordingly based on the deep learning model.

2. The signal processing method based on non-random privacy-preserving distributed deep learning according to claim 1, characterized in that: The data set D is image data. In step 2, each processor processes the input image signal based on a deep learning model.

3. The signal processing method based on non-random privacy-preserving distributed deep learning according to claim 1, characterized in that: The data set D is distributed sensor data. In step 2, each processor processes the input distributed sensor data based on a deep learning model.

4. The signal processing method based on non-random privacy-preserving distributed deep learning according to claim 1, characterized in that: The data set D is frequency domain signal data. In step 2, each processor processes the input frequency domain signal based on a deep learning model.

5. The signal processing method based on non-random privacy-preserving distributed deep learning according to claim 1, characterized in that: The initial model weights are set based on experience or randomly.

Images