Efficient collaborative reasoning method and system for privacy protection

Abstract

The invention discloses a privacy protection-oriented efficient collaborative reasoning method and system, and belongs to the field of data security. According to the method, quantization noise obeying single-peak random distribution is added into collaborative reasoning intermediate data, the size of intermediate transmission data is reduced through quantization, and the privacy security of the intermediate data is protected through noise addition. According to the method, the model slices on the server in collaborative reasoning are finely adjusted through retraining, so that the accuracy of collaborative reasoning is improved. According to the method, the communication overhead of collaborative reasoning can be reduced and the data privacy of collaborative reasoning can be protected at the same time.

Description

Technical Field

[0001] The present invention belongs to the field of data security, and specifically relates to an efficient collaborative reasoning method and system for privacy protection. Background Art

[0002] Collaborative reasoning distributes a neural network reasoning task to multiple devices to execute, thereby accelerating the execution of the reasoning task. Figure 1 As shown in the figure, in a typical client-server collaborative reasoning architecture, the neural network is split into two parts, one on the client and the other on the server. The client uses the input data and the neural network slices on it to perform reasoning and obtain intermediate data; the intermediate data is passed to the server and used as the input of the neural network slices on it for reasoning to obtain the final reasoning result. Finally, the final reasoning result is sent back to the client device.

[0003] However, collaborative reasoning faces two major challenges: first, the large size of intermediate data leads to high communication overhead; second, attackers can launch model inversion attacks through intermediate data to restore model input data, resulting in privacy leakage. To address the problem of large intermediate data, quantization is used to reduce the size of intermediate data, but quantization cannot protect data privacy. To address the problem of privacy leakage, adding noise to intermediate data is used to protect data privacy, but adding noise cannot reduce the size of intermediate data. To address the two major challenges of high communication overhead and privacy leakage at the same time, one method is to first add noise to the intermediate data and then quantize the intermediate data. However, both adding noise and quantization will reduce the inference accuracy of the model.

[0004] Existing technical disclosures mainly involve the compression of neural network model parameters and activation values. The Chinese invention patent with application number CN201810456429.X discloses a neural network data compression and its related calculation method and device, the main content of which is fixed-point quantization of neural network model parameters and activation values. The Chinese invention patent with application number CN202410935680.X discloses a deep neural network compression method based on joint dynamic pruning, which prunes and compresses neural network parameters. The application number CN202410584793.X discloses a method, device, computer equipment and storage medium for deep neural network compression, which sparsely prunes the weight matrix of the neural network and compresses the weight matrix after sparse pruning by the product quantization method. The above method does not involve intermediate data compression and privacy protection for collaborative reasoning. Summary of the invention

[0005] The purpose of the present invention is to provide an efficient collaborative reasoning method and system for privacy protection, which can simultaneously reduce the communication overhead in collaborative reasoning and protect the data privacy of collaborative reasoning.

[0006] The technical solution adopted by the present invention to achieve the above object is as follows:

[0007] An efficient collaborative inference method for privacy protection, comprising the following steps:

[0008] In the establishment stage, the client sends a random number to the server;

[0009] In the retraining stage, the collaborative inference model slices are respectively deployed on the client and the server; the client performs random quantization encoding on the input collaborative inference intermediate data, and sends the encoded quantization data together with the corresponding labels to the server; the server decodes the quantization data using the random number obtained in the establishment stage and generates the decoded intermediate data, and fine-tunes the model slice on the server using the decoded intermediate data and the corresponding labels;

[0010] In the usage stage, the client performs random quantization encoding on the input collaborative inference intermediate data, and sends the encoded quantization data to the server; the server decodes the quantization data, and uses the decoded data to perform inference through the fine-tuned collaborative inference model, and returns the inference result to the client.

[0011] Further, in the establishment stage, the client uses the intermittent period of the collaborative inference task to send multiple batches of random numbers to the server.

[0012] Further, the loss function for fine-tuning the model slice on the server is:

[0013] L * = L + α||V1(U) - V1(U′)||

[0014] Wherein, V1 is the model slice on the server, L is the loss function originally used for training the model slice, α is a balance parameter, U is the collaborative inference intermediate data before encoding, and U′ is the decoded intermediate data.

[0015] Further, L is a cross-entropy function.

[0016] Further, the random numbers sent by the client to the server are R = {r i -l i , 0 ≤ i < N} and X = {x i ~f(x), 0 ≤ i < N}.

[0017] Further, the steps for the client to perform random quantization encoding on the input collaborative inference intermediate data include:

[0018] Input the collaborative inference intermediate data U = {u i , 0 ≤ i < N}, the clipping value B, and the privacy budget P;

[0019] Sample a vector X = {x i ~ f(x), 0 ≤ i < N} from a unimodal random distribution f(x, μ, σ(P)), where μ is the mean of f(x, μ, σ) and σ(P) is the standard deviation of f(x, μ, σ(P)), and σ(P) is a function of P;

[0020] Sample Z = {z i ~ Unif(0, 1), 0 ≤ i < N} from a uniform distribution in the interval (0, 1);

[0021] Clip the collaborative inference intermediate data U to obtain the clipped data where u i ∈ U;

[0022] Uniformly sample a point (x i , y i ) under the curve of the unimodal random distribution f(x), where y i = z i f(x i ), x i ∈ X, z i ∈ Z;

[0023] If x i < 0, then calculate y i = f(μ) - y i ;

[0024] Calculate r i = |f -1 (y i )|, where f -1 is the inverse function of f(x, μ, σ);

[0025] Calculate l i = -|f -1 (f(μ) - y i )|;

[0026] Encode u i ∈ U to obtain Return the quantized data M = {m i , 0 ≤ i < N}.

[0027] Furthermore, the client performs Laplace-based stochastic quantization encoding on the input collaborative inference intermediate data.

[0028] Furthermore, the steps for the client to perform Laplace-based stochastic quantization encoding on the input collaborative inference intermediate data include:

[0029] Input the collaborative inference intermediate data U = {u i , 0 ≤ i < N}, the clipping value B, and the privacy budget P;

[0030] Calculate the scale parameter Sample a vector X = {x ~ f(x), 0 ≤ i < N} from a Laplace distribution with location parameter 0 and scale parameter A; i ~ f(x), 0 ≤ i < N};

[0031] Sample Z = {z i ~ Unif(0, 1), 0 ≤ i < N} from a uniform distribution in the interval (0, 1);

[0032] Clip the collaborative inference intermediate data U to obtain the clipped data where u i ∈ U;

[0033] Uniformly sample a point (x , y i , y i ) under the curve of a Laplace distribution with location parameter 0 and scale parameter A, where

[0034] If x i < 0, then calculate

[0035] Calculate r i = -Aln(2Ay i );

[0036] Calculate

[0037] Encode u i ∈ U to obtain Return the quantized data M = {m i , 0 ≤ i < N}.

[0038] Furthermore, the steps for the server to decode the quantized data include:

[0039] For the quantized data M = {m i , 0 ≤ i < N}, use the random numbers R = {r i - l i , 0 ≤ i < N} and X = {x i ~ f(x), 0 ≤ i < N} to perform decoding calculation u i ' = m i (r i - l i ) + x i , to obtain the decoded intermediate data U' = {u i ', 0 ≤ i < N}.

[0040] An efficient collaborative inference system for privacy protection includes:

[0041] The client is used to send random numbers to the server during the establishment phase; during the retraining phase, the input collaborative reasoning intermediate data is randomly quantized and encoded, and the encoded quantized data and corresponding labels are sent to the server; during the use phase, the input collaborative reasoning intermediate data is randomly quantized and encoded, and the encoded quantized data is sent to the server;

[0042] The server is used to receive random numbers from the client during the establishment phase; receive quantized data and labels sent by the client during the retraining phase, use the random numbers obtained during the establishment phase to decode the quantized data and generate decoded intermediate data, and then use the decoded intermediate data and labels to fine-tune the collaborative reasoning model slices on the server; during the use phase, receive and decode the quantized data sent by the client, use the decoded data to perform reasoning through the fine-tuned collaborative reasoning model, and return the reasoning results to the client.

[0043] The technical effects achieved by the present invention are as follows:

[0044] 1. Privacy protection: By adding quantization noise that obeys a given unimodal random distribution to the collaborative reasoning intermediate data, this method can ensure compliance with differential privacy requirements and effectively prevent attackers from recovering the original input data from the decoded data through model inversion attacks, thereby ensuring the privacy of the data.

[0045] 2. Reduce communication overhead: Due to the use of random quantization technology, this method reduces the number of bits during data transmission. The quantized data requires fewer bits than the original data, thereby greatly reducing communication overhead. This advantage is particularly prominent when a large amount of collaborative reasoning intermediate data needs to be transmitted.

[0046] 3. No impact on collaborative reasoning performance: During the establishment phase, the client sends multiple batches of random numbers serving multiple tasks to the server during the intermittent period, so that the process does not affect the actual collaborative reasoning task performance, which ensures a balance between privacy protection and performance.

[0047] 4. Improve collaborative reasoning accuracy: During the retraining phase, the server decodes the randomly quantized intermediate data and compensates for the accuracy loss caused by random quantization by fine-tuning the model slices. In this way, the server can improve the accuracy of reasoning and ensure the efficiency and accuracy of collaborative reasoning tasks even when the data is quantized. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a schematic diagram of a typical client-server collaborative reasoning architecture;

[0049] Figure 2Schematic diagram of the efficient collaborative inference process for privacy protection of the present invention;

[0050] Figure 3 Schematic diagram of the efficient collaborative inference process for privacy protection based on the Laplace distribution in the embodiment; Detailed implementation manners

[0051] To make the technical features, advantages or technical effects in the above technical solutions of the present invention more obvious and understandable, the following will be described in detail in conjunction with the accompanying drawings.

[0052] An efficient collaborative inference method for privacy protection proposed by the present invention has an inference process as Figure 2 shown, including three stages: a establishment stage, a retraining stage and a testing stage (i.e., the formal usage stage).

[0053] Establishment stage: The client sends random numbers required to decode the quantized data to the server. The client uses the idle period of the collaborative inference task to send multiple batches of random numbers serving multiple collaborative inference tasks to the server, so that the establishment stage does not affect the performance of the collaborative inference.

[0054] Retraining stage: The slices of the collaborative inference model are respectively deployed on the client and the server. The client performs random quantization encoding on the input collaborative inference intermediate data and sends it to the server together with the corresponding label values; for each collaborative inference task, the server needs to consume some random numbers to decode the randomly quantized collaborative inference intermediate data from the client. The server uses the decoded intermediate data and the corresponding label values to fine-tune the model slice on the server to make up for the accuracy loss caused by random quantization, thereby improving the accuracy of the collaborative inference. The collaborative inference model can be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN) / long short-term memory network (LSTM), etc.

[0055] Testing stage: Based on the fine-tuned pre-trained model, the server uses the randomly quantized collaborative inference intermediate data from the client to complete the collaborative inference task, and then returns the inference result to the client.

[0056] The encoding and decoding processes of random quantization involved in the retraining stage and the testing stage are as follows:

[0057] 1. Encoding stage:

[0058] a. Input the collaborative inference intermediate data U = {u i , 0 ≤ i < N} (N is the number of elements of the collaborative inference intermediate data), as well as the clipping value B and the privacy budget P;

[0059] b. Sample the vector X = {x from the unimodal random distribution f(x, μ, σ(P))i ~f(x), 0 ≤ i < N}, where μ is the mean of f(x, μ, σ), σ(P) is the standard deviation of f(x, μ, σ(P)), σ(P) is a function of P, f(x) is a probability distribution function, and x i is a random variable sampled from f(x);

[0060] c. Sample Z = {z i ~Unif(0, 1), 0 ≤ i < N} from the uniform distribution in the interval (0, 1), where Unif(0, 1) represents a random number uniformly distributed between 0 and 1;

[0061] d. Clip the collaborative inference intermediate data U to obtain the clipped data where u i ∈ U, ‖U‖ ∞ represents the maximum absolute value of the elements in U, and 0.5B is a clipping threshold;

[0062] e. Uniformly sample a point (x i , y i ) under the curve of the unimodal random distribution f(x), where y i = z i f(x i ), x i ∈ X (X is a set of random variables sampled from a unimodal random distribution), and z i ∈ Z (Z is a set of uniformly distributed random numbers);

[0063] f. If x i < 0, then calculate y i = f(μ) - y i ;

[0064] g. Calculate r i = |f -1 (y i )|, where f -1 is the inverse function of f(x, μ, σ);

[0065] h. Calculate l i = -|f -1 (f(μ) - y i )|;

[0066] i. Encode u i ∈ U Return the vector M = {m i , 0 ≤ i < N}.

[0067] 2. Decoding stage:

[0068] a. Input M = {m i, 0 ≤ i < N}, R = {r i -l i , 0 ≤ i < N} and X = {x i ~f(x), 0 ≤ i < N};

[0069] b. Calculate u i ’ = m i (r i -l i ) + x i , and return U’ = {u i ’, 0 ≤ i < N}.

[0070] The above process satisfies that u i ’ - u i follows a given unimodal random distribution, that is, quantization noise following a given unimodal random distribution is added to the collaborative inference intermediate data. Since adding quantization noise following a given unimodal random distribution can meet the requirements of differential privacy, it can prevent attackers from recovering the original input data from U’ through model inversion attacks. Among them, the encoding stage is the quantization process. Assuming that the original collaborative inference intermediate data is represented by 32 bits, the number of bits reduced in the quantization process is 32 - log2(m max -m min +1), where m max is the largest element in vector M, and m min is the smallest element in vector M.

[0071] Based on the method proposed in the present invention, the present invention specifically discloses an efficient privacy - preserving collaborative inference method based on Laplace distribution through an embodiment. As Figure 3 shown, it includes three stages: a setup stage, a retraining stage, and a testing stage.

[0072] 1. Setup stage: The client sends the random numbers R = {r i -l i , 0 ≤ i < N} and X = {x i ~f(x), 0 ≤ i < N} required to decode the quantization data M to the server. The client uses the idle period of the collaborative inference task to send multiple batches of random numbers serving multiple collaborative inference tasks to the server, so that the setup stage does not affect the performance of collaborative inference.

[0073] 2. Retraining Phase: The collaborative inference model slices are separately deployed on the client and the server. The client performs Laplace-based random quantization encoding on the input collaborative inference intermediate data U to generate quantized data M, and sends M and the corresponding label values to the server. The server uses the random numbers R and X received during the establishment phase to decode M to generate the decoded intermediate data U', and uses the decoded intermediate data U' and the corresponding label values to fine-tune the model slice on the server to compensate for the accuracy loss caused by random quantization, thereby improving the accuracy of collaborative inference. The loss function for fine-tuning the model slice on the server is:

[0074] L * = L + α‖V1(U) - V1(U ′ )‖

[0075] where V1 is the model slice on the server, L is the loss function originally used for training the model slice (such as the cross-entropy function), and α is the balancing parameter.

[0076] 3. Testing Phase: The server uses the randomly quantized collaborative inference intermediate data from the client to complete the collaborative inference task, and then returns the inference result to the client.

[0077] In the above retraining phase and testing phase, for each collaborative inference task, the client performs Laplace-based random quantization encoding on the collaborative inference intermediate data U = {u i , 0 ≤ i < N}, that is, calculates M = {m i , 0 ≤ i < N}, where and sends M to the server; the server uses the previously received random numbers R and X to decode M, that is, calculates m i , where u i ' = m i (r i - l i ) + x i . Then the server uses the decoded intermediate data U' as the input data and uses the model slice on it to perform inference, obtains the final inference result and sends it back to the client.

[0078] In this embodiment, the random quantization based on the Laplace distribution means adding quantization noise obeying the Laplace distribution to the collaborative inference intermediate data to achieve the purpose of both quantizing the intermediate data and protecting the privacy of the intermediate data. The encoding and decoding processes of the random quantization based on the Laplace distribution are as follows:

[0079] 1. Encoding Phase:

[0080] a. Input the collaborative inference intermediate data U = {u i , 0 ≤ i < N}, the clipping value B, and the privacy budget P;

[0081] b. Calculate Sample vector X = {x ~ f(x), 0 ≤ i < N} from the Laplace distribution with location parameter 0 and scale parameter A i ;

[0082] c. Sample Z = {z i ~ Unif(0, 1), 0 ≤ i < N} from the uniform distribution in the interval (0, 1);

[0083] d. Clip the collaborative inference intermediate data where u i ∈ U;

[0084] e. Uniformly sample a point (x , y i , y i ) under the curve of the Laplace distribution with location parameter 0 and scale parameter A, where x i ∈ X, z i ∈ Z;

[0085] f. If x i < 0, then calculate

[0086] g. Calculate r i = -Aln2Ay i ;

[0087] h. Calculate

[0088] i. Encode u i ∈ U and return vector M = {m , 0 ≤ i < N}. i ;

[0089] 3. Decoding stage:

[0090] a. Input M = {m i , 0 ≤ i < N}, R = {r i -l i , 0 ≤ i < N} and X = {x i ~ f(x), 0 ≤ i < N};

[0091] b. Calculate u i ’ = m i (r i -l i ) + x i , and return U’ = {u i ’, 0 ≤ i < N}.

[0092] The above encoding and decoding process satisfies u i '-u i It obeys the Laplace distribution, that is, Laplace quantization noise is added to the collaborative reasoning intermediate data. Since the process of adding Laplace noise complies with (ε, 0) differential privacy, it can prevent attackers from recovering the original input data from U' through model inversion attacks.

[0093] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.

Claims

1. An efficient collaborative reasoning method for privacy protection, characterized in that: The following steps are involved: In the establishment phase, the client sends a random number to the server; In the retraining phase, the collaborative reasoning model slices are deployed on the client and server respectively; the client randomly quantizes and encodes the input collaborative reasoning intermediate data, and sends the encoded quantized data together with the corresponding label to the server; the server decodes the quantized data using the random number obtained in the establishment phase and generates decoded intermediate data, and uses the decoded intermediate data and corresponding labels to fine-tune the model slices on the server; In the use phase, the client randomly quantizes and encodes the input collaborative reasoning intermediate data, and sends the encoded quantized data to the server; The server decodes the quantized data, uses the decoded data to perform reasoning through the fine-tuned collaborative reasoning model, and returns the reasoning results to the client.

2. The method according to claim 1, characterized in that During the establishment phase, the client uses the intervals between collaborative reasoning tasks to send multiple batches of random numbers to the server.

3. The method according to claim 1, characterized in that The loss function for the model slice on the fine-tuning server is: L * =L+α‖V1(U)-V1(U ′ )‖ Among them, V1 is the model slice on the server, L is the loss function originally used to train the model slice, α is the balance parameter, U is the collaborative reasoning intermediate data before encoding, and U ′ It is the intermediate data after decoding.

4. The method according to claim 3, characterized in that L is the cross entropy function.

5. The method according to claim 1, characterized in that The client sends the random numbers R = {r i -l i , 0 ≤ i < N} and X = {x i ~f(x), 0 ≤ i < N} to the server.

6. The method according to claim 5, characterized in that The steps for the client to randomly quantize and encode the input collaborative reasoning intermediate data include: Input collaborative inference intermediate data U = {u i , 0 ≤ i < N}, clipping value B, privacy budget P; Sample a vector X = {x i ~ f(x), 0 ≤ i < N} from a unimodal random distribution f(x, μ, σ(P)), where μ is the mean of f(x, μ, σ) and σ(P) is the standard deviation of f(x, μ, σ(P)), and σ(P) is a function of P; Sample Z from a uniform distribution in the interval (0,1) = {z i ~Unif(0,1),0≤i <N}; Cut the collaborative reasoning intermediate data U to obtain the cut data where u i ∈U; Uniformly sample a point (x) under the curve of a unimodal random distribution f(x) i ,y i ), where y i =z i f(x i ), x i ∈X,z i ∈ Z; If x i <0, then calculate y i =f(μ)-y i ; Calculate r i =|f -1 (y i )|, where f -1 is the inverse function of f(x,μ,σ); Calculate l i =-|f -1 (f(μ)-y i )|; To u i ∈U is encoded to obtain Return quantized data M = {m i ,0≤i <N}。 7. The method according to claim 6, characterized in that The client performs Laplace-based random quantization encoding on the input collaborative reasoning intermediate data.

8. The method according to claim 7, characterized in that The steps for the client to perform Laplace-based random quantization encoding on the input collaborative reasoning intermediate data include: Input collaborative inference intermediate data U = {u i , 0 ≤ i < N}, clipping value B, privacy budget P; Calculate scale parameters From a Laplace distribution with location parameter 0 and scale parameter A The sampling vector X = {x i ~f(x),0≤i <N}; Sample Z from a uniform distribution in the interval (0,1) = {z i ~Unif(0,1),0≤i <N}; Cut the collaborative reasoning intermediate data U to obtain the cut data where u i ∈U; Laplace distribution with location parameter 0 and scale parameter A Uniform sampling under the curve of sampling point (x i ,y i ),in x i ∈X,z i ∈ Z; If x i <0, then calculate Calculate r i =-Aln2Ay i ; calculate To u i ∈U is encoded to obtain Return quantized data M = {m i ,0≤i <N}。 9. The method according to claim 6 or 8, characterized in that The steps for the server to decode the quantized data include: For the quantized data M = {m i , 0 ≤ i < N}, use the random numbers R = {r i -l i , 0 ≤ i < N} and X = {x i ~f(x), 0 ≤ i < N} to perform decoding calculations u i ’ = m i (r i -l i ) + x i , and obtain the decoded intermediate data U’ = {u i ’, 0 ≤ i < N}.

10. An efficient collaborative reasoning system for privacy protection, used to implement the method described in any one of claims 1 to 9, characterized in that: include: The client is used to send a random number to the server during the establishment phase; In the retraining phase, the input collaborative reasoning intermediate data is randomly quantized and encoded, and the encoded quantized data and corresponding labels are sent to the server; In the use phase, the input collaborative reasoning intermediate data is randomly quantized and encoded, and the encoded quantized data is sent to the server; The server is used to receive the random number from the client during the establishment phase; In the retraining phase, the quantized data and labels sent by the client are received, the random numbers obtained in the establishment phase are used to decode the quantized data and generate decoded intermediate data, and then the decoded intermediate data and labels are used to fine-tune the collaborative inference model slices on the server; During the use phase, the quantized data sent by the client is received and decoded, and the decoded data is used to perform reasoning through the fine-tuned collaborative reasoning model, and the reasoning results are returned to the client.

Images