Privacy-preserving optimization method based on stochastic proximal optimization

Abstract

The present invention discloses a privacy protection optimization method based on random proximal optimization, which mainly solves the problem in the prior art that the performance of the neural network model decreases after differential privacy noise is added. Its implementation scheme is: obtaining an image data set and preprocessing it; selecting an existing neural network model including an input layer, a hidden layer, and an output layer; inputting the preprocessed image data set into the neural network model; using a proximal privacy protection method to perform multiple rounds of training on the neural network including gradient clipping, adding differential privacy noise, calculating variance reduction terms, and using proximal endpoints to update the weight matrix to obtain the optimal weight matrix, providing a parameter basis for the subsequent construction of a classification recognition image processing neural network model with high performance and privacy protection. Under the premise of effectively ensuring data privacy security, the present invention improves the convergence speed and accuracy of image recognition, is particularly suitable for image data sets with uneven classification quality, and can be used for medical image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning and image processing, and in particular relates to a privacy protection optimization method that can be used for medical image classification. Background Art

[0002] With the rapid development of artificial intelligence (AI), particularly the application of deep learning in the medical field, medical image analysis has become an important tool for assisting clinical diagnosis. By training on large amounts of medical image data, neural network models can effectively identify, classify, and predict disease conditions, offering broad application prospects in fields such as radiology, pathology, and ophthalmology. However, the unique nature of medical imaging data presents numerous challenges in its application, particularly data privacy and algorithm effectiveness. The use of medical data must be conducted within a strict privacy protection framework, placing additional demands on AI algorithms. To address this, differential privacy has become a mainstream privacy protection method. By adding noise to disrupt sensitive information, differential privacy prevents the leakage of patient privacy when the data is used for model training.

[0003] In the field of machine learning, commonly used optimization algorithms include SGD, Adagrad, SAGA, etc. These algorithms have certain effects in image classification tasks, but they still have certain shortcomings when processing large-scale medical imaging data.

[0004] Patent document CN202111009795.9 discloses a differential privacy-preserving deep learning algorithm that adaptively allocates a dynamic privacy budget. It first trains on a dataset and uses the LRP algorithm to calculate the average feature correlation to obtain the correlation ratio. The DPSGD algorithm is then used on this basis for differential privacy protection. This method converges slowly on large-scale datasets, and even after introducing the differential privacy mechanism, it is difficult to maintain the model's efficiency, resulting in a significant decrease in classification accuracy. At the same time, its classification and recognition performance is poor when processing datasets with varying quality, such as medical imaging datasets. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and provide a privacy-preserving optimization method based on stochastic proximal optimization, so as to improve the convergence speed and accuracy of image recognition while effectively ensuring data privacy and security; and to enhance the classification and recognition effect of data sets with uneven quality.

[0006] To achieve the above object, the technical solution of the present invention includes the following steps:

[0007] (1) Obtain an image dataset D containing n samples, each sample includes a feature vector and the corresponding label y i′∈{-1,+1}.

[0008] (2) Perform preprocessing of the image dataset D to a uniform size to obtain the preprocessed dataset D′:

[0009] D′={(X1,y1),(X2,y2),…,(X i ,y i ),…,(X n ,y n )},i∈(1,n)

[0010] Where (X i ,y i ) represents the feature vector and corresponding label of the j-th sample in the preprocessed training set D′;

[0011] (3) Selecting an existing multi-layer perceptron MLP neural network model P including an input layer, a hidden layer, and an output layer, wherein the input layer is used to receive pixel information of the image, the hidden layer is used to gradually extract and process the features of the image, and the output layer outputs the probability of the category to which the image belongs;

[0012] (4) Input the preprocessed training set D′ into the neural network model P and perform random proximal privacy protection training on it:

[0013] (4a) Setting the batch size data Initialize the weight matrix θ (t) =0, then initialize batch data B t The average gradient And set the maximum number of iterations T in a single round and the maximum number of training rounds H;

[0014] (4b) Calculate batch data B using the gradient clipping method t The average gradient where t∈{,1,2,…,T};

[0015] (4c) In the average gradient Add differential privacy noise ξ t , get the average gradient after adding noise

[0016] (4d) Calculate batch data B t The global gradient g full , and subtract the average gradient Get the variance reduction term v t ;

[0017] (4e) Based on the variance reduction term v t Use the proximal endpoint method to update the network's weight matrix parameters θ (t) Get the final weight matrix of the current iteration

[0018] (4f) Repeat the above steps (4b) to (4e) until the number of iterations t reaches the maximum number of iterations T or the final weight matrix of the current iteration Satisfy the convergence conditions and output the weight matrix θ of the current round j , where j∈{1,2,…,H} represents the jth round of training, and H is the maximum number of training rounds;

[0019] (4g) Repeat the above steps (4b) to (4f) until j reaches the maximum number of training rounds H or the weight matrix θ j When the accuracy conditions are met, the final round weight matrix, i.e., the optimal weight matrix θ, is output, providing a parameter basis for the subsequent construction of a classification and recognition image processing neural network model with high performance and privacy protection.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] First, the present invention uses gradient clipping to limit the noise added to batch gradients, thereby reducing the noise interference on the neural network, so that the neural network still has high performance. This effect is especially obvious when the neural network faces high privacy requirements and needs to continuously add noise.

[0022] Secondly, the present invention adds a variance reduction term to the average batch gradient of the updated weight matrix, thereby enabling the neural network to be updated more stably, enhancing the generalization ability of the neural network and improving its recognition accuracy.

[0023] Third, the present invention directly embeds the regularization adjustment process into the update process of the weight matrix through the proximal endpoint method, which not only improves the overall training speed of the neural network, but also can promote sparsity adjustment of the parameters of each update, so that the training process of the neural network is more focused on the weight matrix, further improving the training speed of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Flowchart for the implementation of the present invention;

[0025] Figure 2 This is a sub-flowchart for training the neural network model in the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] Reference Figure 1 , the implementation steps of this example are as follows:

[0028] Step 1. Obtain an image dataset.

[0029] From the existing public database, obtain an image dataset D containing n samples, each sample includes a feature vector and the corresponding label y i ′∈{-1,+1}, i∈(1,n), the feature vector is the key mathematical representation used to describe the image in the image dataset, and the corresponding label is the information used to identify the image category.

[0030] Step 2. Preprocess the acquired dataset D.

[0031] The preprocessing is a series of adjustments made to the image data before using it for neural network training and other operations. The specific implementation is as follows:

[0032] 2.1) Use each sample (X i ′,y i ′)’s eigenvector X i 'Each row of the feature matrix X is combined to obtain X. The feature matrix X is to facilitate the processing of each eigenvector X i ′, and the mathematical matrix obtained by integrating them together;

[0033] 2.2) Each element x of the feature matrix X iq Normalize to the interval [0,1] to obtain the processed feature matrix X′:

[0034]

[0035] where x iq is the element in the i-th row and q-th column of the feature matrix X; μ q ,σ q are the mean and standard deviation of all elements in the qth column of the feature matrix X; the element x iq Normalization to the interval [0,1] is a data preprocessing technique used to convert the element x ij The value of is scaled to a specific range;

[0036] 2.3) The original label y i The value of ′ is converted from an integer to a floating point number, and the label of -1 is changed to 0 to obtain the label vector y i , in order to reduce the amount of calculation during the training process and provide precision guarantee for the calculation of subsequent model recognition accuracy;

[0037] 2.4) Output each row of the processed feature matrix X′ as a processed feature vector X i , and then with the transformed label vector y i The combined dataset D′ is:

[0038] D′={(X1,y1),(X2,y2),…,(X i ,y i ),…,(X n ,y n )}.

[0039] Where (X i ,y i ) represents the feature vector and corresponding label of the j-th sample in the preprocessed training set D′.

[0040] Step 3. Select an existing neural network model.

[0041] An existing multi-layer perceptron MLP neural network model P is selected, which includes an input layer, a hidden layer and an output layer. The input layer is used to receive pixel information of the image, the hidden layer is used to gradually extract and process the features of the image, and the output layer outputs the probability of the category to which the image belongs. The multi-layer perceptron MLP is a neural network with a multi-layer structure, nonlinear processing capabilities and approximation characteristics.

[0042] Step 4. Input the dataset D′ into the neural network model P and train it using the random proximal privacy protection method to obtain the optimal weight matrix θ.

[0043] Reference Figure 2 , this step is implemented as follows:

[0044] 4.1) After encapsulating the data in dataset D′ using the Tensor function, the data is loaded in batches using the DataLoader function and input into the neural network model P;

[0045] 4.2) Initialize the weight matrix and average gradient, set the hyperparameters and batch data required for training:

[0046] Setting batch size data And the maximum number of iterations in a single round T, the maximum number of training rounds H, the initialization weight matrix θ (t)=0, then initialize batch data B t The average gradient t∈{1,2,…,T}, where the maximum number of iterations T and the maximum number of training rounds H in a single round are set to prevent the neural network from being unable to stop during training, causing the program to crash. The batch size data B is set t , because in the subsequent training, compared with the gradient calculation of all data, only D 、 Batch data B in t Gradient calculations are faster;

[0047] 4.3) Calculate the average gradient of the batch and perform gradient clipping:

[0048] (4.3.1) Calculate batch data B using the gradient clipping method t Each sample (X r ,y r )’s gradient g r：

[0049]

[0050] where θ (t-1) is the weight matrix obtained at the (t-1)th iteration; r∈{1,2,…,b}, b is the batch size; λ1 is the 1-norm regularization parameter, which is used to adjust the sparsity of the parameters of each update; λ2 is the 2-norm regularization parameter, which is used to prevent overfitting of the neural network;

[0051] (4.3.2) Calculate the gradient g of a single sample r The 2-norm of And compare it with the set gradient clipping threshold C to limit the noise added to the average gradient, thereby reducing the noise interference of the neural network:

[0052] If ‖g r ‖2>C, then gradient clipping is performed first to obtain the clipped single sample gradient g r ', then execute step (5.2.3);

[0053]

[0054] If ‖g r ‖2≤C, then directly execute step (5.2.3);

[0055] (4.3.3) Update batch data B t The average gradient

[0056]

[0057] in For batch data B t-1 The average gradient of

[0058] 4.4) In the average gradient Add differential privacy noise ξ t , get the average gradient after adding noise To achieve differential privacy protection:

[0059] (4.4.1) Calculate the global sensitivity S according to the batch size b and the gradient clipping threshold C, which reflects the sensitivity of the training dataset D' to the addition of noise:

[0060]

[0061] (4.4.2) Calculate the noise scale σ based on the global sensitivity S:

[0062]

[0063] Where n is the total number of image data samples; ε is the privacy budget, which determines the strength of the privacy protection of the neural network;

[0064] (4.4.3) Generate differential privacy noise ξ that follows a normal distribution based on the noise scale σ t :

[0065]

[0066] Among them E n is the identity matrix of order n;

[0067] (4.4.4) According to the average gradient and differential privacy noise ξ t , calculate the average gradient after adding noise

[0068]

[0069] 4.5) Calculate the variance reduction term for subsequent weight matrix updates:

[0070] (4.5.1) Calculate batch data B t The global gradient g full :

[0071]

[0072] where θ (t-1) is the weight matrix obtained at the (t-1)th iteration; r∈{1,2,…,b}, b is the batch size; λ1 is the 1-norm regularization parameter; λ2 is the 2-norm regularization parameter;

[0073] (4.5.2) Calculate the variance reduction term v t :

[0074]

[0075] 4.6) Based on the variance reduction term v t Use the proximal endpoint method to update the network's weight matrix parameters θ (t) Get the final weight matrix of the current iteration

[0076] (4.6.1) According to the average gradient after adding noise and with variance reduction term v t , use the following formula to update the weight matrix parameters:

[0077]

[0078] Among them, η is the learning rate, which determines the step size of each iterative update; θ (t-1) is the weight matrix obtained in the (t-1)th iteration, θ (t) is the weight matrix obtained by the t-th iteration update, and λ2 is the 2-norm regularization parameter used to prevent the neural network from overfitting;

[0079] (4.6.2) for θ (t) Each component Perform 1-norm regularized soft threshold update to obtain

[0080]

[0081] in is θ (t) The kth component of , k∈{1,2,…,n}, η is the learning rate, λ1 is the 1-norm regularization parameter,

[0082] By updating the soft threshold of the 1-norm regularization, sparsity can be adjusted at each update, making the neural network training process more focused on the weight matrix;

[0083] (4.6.3) Yes Recombine in order to get the final weight matrix of the current iteration

[0084]

[0085] in for The kth component of ;

[0086] 4.7) Repeat the iteration to get the weight matrix θ of the current round j, where \(j\in\{1,2,\ldots,H\}\) represents the \(j\)-th round of training, and \(H\) is the maximum number of training rounds:

[0087] (4.7.1) Compare the current iteration number \(t\) with the maximum iteration number \(T\):

[0088] If \(t < T\), then execute step (4.7.2);

[0089] If \(t\geq T\), then output the weight matrix of the current round

[0090] (4.7.2) Calculate the loss function of the neural network model \(P\) according to the final weight matrix of the current iteration

[0091]

[0092] where \((X r , y r ) is the \(r\)-th sample in data \(B t ;

[0093] (4.7.3) Compare with the convergence accuracy \(\rho\):

[0094] If then output the weight matrix of the current round

[0095] If then return to step 4.3;

[0096] where the convergence accuracy \(\rho\) determines the effectiveness of updating the optimal weight matrix \(\theta\) by the proximal point method;

[0097] 4.8) Repeat training to obtain the optimal weight matrix \(\theta\):

[0098] (4.8.1) Compare the current training round number \(j\) with the maximum training round number \(H\):

[0099] If \(j < H\), then execute step (4.8.2);

[0100] If \(j\geq H\), then output the optimal weight matrix \(\theta=\theta j [[ID=6I]], j\in\{1,2,\ldots,H\};

[0101] (4.8.2) Calculate the accuracy \(\alpha(\theta j ) of the neural network model \(P\) according to the weight matrix \(\theta j ) of the current round:

[0102]

[0103] Among them, X′ is the feature matrix processed in step 2.1), y i is the i-th label i∈(1,2,…,n) of the dataset D′; is the indicator function, when the condition sign(X′·θ j ) i =y i When it was established, otherwise

[0104] (4.8.3) Set the convergence accuracy threshold β based on the required neural network recognition accuracy. The value of β determines the recognition accuracy of the classification recognition neural network.

[0105] (4.8.4) j ) is compared with the set convergence accuracy threshold β:

[0106] If α(θ j )≥β, then the optimal weight matrix θ=θ is output j ;

[0107] If α(θ j )<β, then return to step 4.3).

[0108] It should be noted that the step numbers in the specification and claims of the present invention are only for the purpose of clearly describing the embodiments of the present invention and facilitating understanding, and the order of the step numbers is not limited.

Claims

1. A privacy-preserving optimization method based on random proximal optimization, characterized in that: The steps include: (1) Obtain an image dataset D containing n samples, each sample includes a feature vector and the corresponding label y i ′∈{-1,+1}; (2) Perform preprocessing of the image dataset D to a uniform size to obtain the preprocessed dataset D′: D′={(X1,y1),(X2,y2),…,(X i ,y i ),…,(X n ,y n )},i∈(1,n) Where (X i ,y i ) represents the feature vector and corresponding label of the j-th sample in the preprocessed training set D′; (3) Selecting an existing multi-layer perceptron MLP neural network model P including an input layer, a hidden layer, and an output layer, wherein the input layer is used to receive pixel information of the image, the hidden layer is used to gradually extract and process the features of the image, and the output layer outputs the probability of the category to which the image belongs; (4) Input the preprocessed training set D′ into the neural network model P and perform random proximal privacy protection training on it: (4a) Setting the batch size data Initialize the weight matrix θ (t) =0, then initialize batch data B t The average gradient And set the maximum number of iterations T in a single round and the maximum number of training rounds H; (4b) Calculate batch data B using the gradient clipping method t The average gradient where t∈{,1,2,…,T}; (4c) In the average gradient Add differential privacy noise ξ t , get the average gradient after adding noise (4d) Calculate batch data B t The global gradient g full , and subtract the average gradient Get the variance reduction term v t ; (4e) Based on the variance reduction term v t Use the proximal endpoint method to update the network's weight matrix parameters θ (t) Get the final weight matrix of the current iteration The implementation is as follows: (4e1) According to the average gradient after adding noise and with variance reduction term v t , use the following formula to update the weight matrix parameters: Among them, η is the learning rate, θ (t-1) is the weight matrix obtained in the t-1th iteration, θ (t) is the weight matrix obtained by the t-th iteration update, λ2 is the 2-norm regularization parameter; (4e2) for θ (t) Each component Perform 1-norm regularized soft threshold update to obtain in is θ (t) The kth component of , k∈{1,2,…,n}, η is the learning rate, and λ1 is the 1-norm regularization parameter; (4e3) Yes Recombine in order to get the final weight matrix of the current iteration in for The kth component of ; (4f) Repeat the above steps (4b) to (4e) until the number of iterations t reaches the maximum number of iterations T or the final weight matrix of the current iteration Satisfy the convergence conditions and output the weight matrix of the current round Where j∈{1,2,…,H} represents the jth round of training, and H is the maximum number of training rounds; (4g) Repeat the above steps (4b) to (4f) until j reaches the maximum number of training rounds H or the weight matrix θ j When the accuracy conditions are met, the final round weight matrix, i.e., the optimal weight matrix θ, is output, providing a parameter basis for the subsequent construction of a classification and recognition image processing neural network model with high performance and privacy protection.

2. The method according to claim 1, characterized in that In step (2), the image dataset D is preprocessed to a uniform size, which is achieved as follows: (2a) Each sample (X i ′,y i ′)’s eigenvector X i 'composes each row of X, and obtains the feature matrix X; (2b) Each element x of the feature matrix X iq Normalize to the interval [0,1] to obtain the processed feature matrix X′: where x iq is the element in the i-th row and q-th column of the feature matrix X; μ q ,σ q are the mean and standard deviation of all elements in the qth column of the feature matrix X; (2c) The original label y i The value of ′ is converted from an integer to a floating point number, and the label of -1 is changed to 0 to obtain the label vector y i ; (2d) Output each row of the processed feature matrix X′ as a processed feature vector X i , and then with the transformed label vector y i The combined dataset D′ is: D′={(X1,y1),(X2,y2),…,(X i ,y i ),…,(X n ,y n )}。 3. The method according to claim 1, characterized in that In step (4b), the batch data B is calculated using the gradient clipping method t The average gradient The implementation is as follows: (4b1) Calculate batch data B t Each sample (X r ,y r )’s gradient g r , where θ (t-1) is the weight matrix obtained at the t-1th iteration, λ1 is the 1-norm regularization parameter, λ2 is the 2-norm regularization parameter, r∈{1,2,…,b}, b is the batch size; (4b2) Calculate the gradient g of a single sample r The 2-norm of And compare it with the set gradient clipping threshold C: If ‖g r ‖2>C, then gradient clipping is performed first to obtain the clipped single sample gradient g r ', then execute step (4b3): If ‖g r ‖2≤C, then directly execute step (4b3); (4b3) Update batch data B t The average gradient in For batch data B t-1 The average gradient of .

4. The method according to claim 1, wherein Step (4c) averages the gradient Add differential privacy noise ξ t , get the average gradient after adding noise The implementation is as follows: (4c1) Calculate the global sensitivity S: Where b is the batch size and C is the gradient clipping threshold; (4c2) Calculate the noise scale σ based on the global sensitivity S: Where ε is the privacy budget; (4c3) Generate differential privacy noise ξ that follows a normal distribution according to the noise scale σ t : Among them E n is the identity matrix of order n; (4c4) According to the average gradient and differential privacy noise ξ t , calculate the average gradient after adding noise 5. The method according to claim 1, wherein Calculate batch data B in step (4d) t The global gradient g full , the formula is as follows: Among them, (X r ,y r ) is data B t The rth sample r∈{1,2,…,b} in , where b is the batch size; θ (t-1) is the weight matrix obtained in the t-1th iteration; λ1 is the L1 regularization parameter, and λ2 is the L2 regularization parameter.

6. The method according to claim 1, wherein The final weight matrix that satisfies the current iteration in step (4f) The convergence conditions are set as follows: (4f1) According to the final weight of the current iteration Matrix calculation of the loss function of the neural network model P Where (X r ,y r ) is data B t The rth sample r∈{1,2,…,b}, b is the batch size, λ1 is the 1-norm regularization parameter, and θ2 is the 2-norm regularization parameter; (4f2) Compare with the set convergence accuracy ρ: if but Satisfy the convergence conditions; if but The convergence condition is not met.

7. The method according to claim 1, characterized in that Step (4g) satisfies the weight matrix θ of the current round j The accuracy conditions are set as follows: (4g1) According to the weight matrix θ of the current round j Matrix, calculate the accuracy α(θ of the neural network model P j ): Where X′ is the feature matrix processed in the step, y i is the i-th label i∈(1,2,…,n) of the dataset D′, is the indicator function, when the condition sign(X′·θ j ) i =y i When it was established, otherwise (4g2) j ) is compared with the set convergence accuracy threshold β: If α(θ j )≥β, then θ j Satisfy the accuracy requirements; If α(θ j )<β, then θ j The accuracy condition is not met.

Images