Neural network privacy reasoning method and device based on winograd sparse convolution and differentiable activation function approximation

Abstract

The invention belongs to the technical field of artificial intelligence information security, particularly discloses a neural network privacy reasoning method and device based on winograd sparse convolution and differentiable activation function approximation, and aims at solving the problems that existing neural network privacy reasoning is high in calculation time delay and large in communication cost. And in combination with a multi-granularity weight pruning algorithm, the sparsity of a linear convolutional layer is maximized, and the communication cost required by the convolutional layer with dense multiplication is reduced. Meanwhile, collaborative optimization is carried out on a nonlinear activation function, a new activation function sensitivity and importance quantitative evaluation algorithm is designed, and an adaptive learnable polynomial function is introduced, so that the fitting precision of the activation function is improved, and the polynomial replacement loss is reduced; and the powerful prediction performance of the neural network model is reserved by using a multi-stage knowledge distillation method, and the calculation and communication efficiency of privacy protection neural network reasoning is effectively improved on the premise of not influencing the prediction precision.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence information security technology, and is a deep neural network reasoning method for privacy protection, and in particular to a neural network privacy reasoning method and device based on winograd sparse convolution and differentiable activation function approximation. Background Art

[0002] The new generation of artificial intelligence technology represented by technologies such as autonomous driving, intelligent question answering, and content generation is leading the third information technology revolution. Emerging technologies are closely integrated with applications such as intelligent medical care, financial forecasting, and risk assessment. The data sources are wider, the formats are more diverse, and the content is richer, which rapidly promotes changes in the field and leapfrog development. According to statistics, the amount of data generated globally will increase from 33ZB in 2018 to 175ZB in 2025. Large-scale data has become the main driving force for the development of deep technology. The efficient training and accurate reasoning of the new generation of high-performance neural network models are highly dependent on massive amounts of high-quality data and powerful computing resources, both of which are mainly concentrated in large technology companies (such as Google, Meta, Microsoft, etc. abroad, and Alibaba, Tencent, Baidu, etc. in China). Therefore, deep learning models are usually regarded as digital assets with intellectual property rights, and technology companies hope to make profits from them. On the other hand, ordinary users tend to outsource their data to service providers for heavy model reasoning calculations to solve problems such as limited local computing resources, and use high-performance deep learning models to explore the potential value of data. The interactive data between users and service providers often contains sensitive information. How to achieve efficient and secure data sharing and collaborative computing between data privacy and data value has become a difficult problem that needs to be solved urgently. Deep neural network privacy reasoning, as a technology that can perform reasoning calculations in an encrypted or protected state, provides a feasible solution to this problem.

[0003] The privacy reasoning method of deep neural network mainly relies on homomorphic encryption (HE) and secure multi-party computation (MPC) technology to ensure the security and confidentiality of data, but it also greatly increases the computation and communication costs in the privacy reasoning state, resulting in limited application scope and low computational efficiency in practical scenarios. To this end, researchers have proposed a series of optimization methods to reduce the computation and communication costs of privacy reasoning of deep neural networks, mainly including optimization frameworks based on privacy protocols and model architectures. The privacy reasoning optimization method based on privacy protocols refers to designing lightweight privacy computing interaction protocols, reducing the computational complexity of both parties or reducing the amount of communication data through ciphertext packaging or vector encoding. The privacy reasoning optimization method based on model architecture refers to designing privacy computing-friendly replacement or approximation schemes for the computing modules in the neural network model to improve the privacy reasoning efficiency of the neural network model.

[0004] Compared with the optimization method based on privacy protocol, the optimization method based on model architecture can better utilize the parameters and architecture redundancy of the neural network model, significantly reduce the computing delay and improve the communication efficiency under the premise of ensuring high-precision privacy reasoning, and can be combined with the former to achieve better optimization effect. In view of the problem that homomorphic encryption and secure multi-party computing cannot perform efficient nonlinear calculations, the existing model architecture-based methods mainly optimize the nonlinear activation function of the neural network, and believe that the multiplication efficiency in the convolution layer in the online reasoning stage is as efficient as the plaintext calculation and does not need to be optimized. However, with the continuous development of privacy computing technologies such as oblivious transfer (OT) and secret sharing (SS), the efficiency of calculating ReLU (Rectified linear units) nonlinear functions has been greatly improved. In the scenario of parallel privacy reasoning, the communication cost brought by convolution multiplication operations gradually increases in the entire privacy reasoning process. The existing optimization methods have low computational and communication efficiency. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation.

[0006] The technical solution adopted by the present invention is as follows:

[0007] In the first aspect, a neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation is provided, which is applied to a deep neural network privacy reasoning system, including a server and a client, wherein the server holds a training data set for generating a pre-trained model based on the data set training; the client holds a number of privacy reasoning service request data, and the client is used to send the request data to the server for privacy reasoning, and the privacy reasoning method includes:

[0008] S1: Perform parameter transformation and sparse processing on the pre-trained deep learning model to obtain a neural network prediction model after linear convolution layer sparsification;

[0009] S2: Retrain the model output by S1, wherein the accuracy of the neural network prediction model after the linear convolution layer is sparse is evaluated in each training iteration round. If the model accuracy loss is less than the preset threshold, the linear convolution layer is sparse, and the retrained winograd sparse deep learning model is obtained;

[0010] S3: Perform activation function approximation and replacement processing on the model output by S2 to obtain a neural network prediction model after nonlinear activation function approximation;

[0011] S4: Retrain the model output by S3, where the model accuracy is tested after each training iteration. If the model accuracy loss is within an acceptable range, the nonlinear activation function is approximated to obtain the final optimized model.

[0012] S5: Perform privacy reasoning based on the model output by S4.

[0013] In one embodiment, S1 includes:

[0014] S1.1: The pre-trained deep learning model is input into the convolutional layer sparsification module, and the importance of the convolution kernel is evaluated through the Winograd algorithm transformation. The redundant convolution kernels are removed from the model according to the importance evaluation results.

[0015] S1.2: Perform weight transposition and remove redundant weights in vector units to obtain the neural network prediction model after linear convolution layer sparsification.

[0016] In one embodiment, S1.1 includes:

[0017] S1.1.1: Multiply the convolution kernel of the input pre-trained deep learning model with the sparse transformation tensor of the winograd convolution algorithm, and calculate the winograd convolution parameters using the following formula:

[0018] U=ΣR·F

[0019] Among them, F is the convolution kernel of the pre-trained deep learning model, R is the sparse transformation tensor of the winograd convolution algorithm, and U is the winograd convolution parameter;

[0020] S1.1.2: Calculate the convolution kernel importance evaluation score based on the convolution kernel of the pre-trained deep learning model and the sparse transformation tensor of the winograd convolution algorithm:

[0021] I=∑||R·F||1

[0022] Among them, I is the convolution kernel importance evaluation score;

[0023] S1.1.3: Sort the calculated convolution kernel importance evaluation scores in descending order, and delete the p% convolution kernels with the smallest importance evaluation scores as redundant convolution kernels, where p is a value between 0 and 100.

[0024] In one embodiment, S1.2 includes:

[0025] S1.2.1: For an input channel number c o , the number of input channels is c i , the lth convolution layer with a convolution kernel size of n×n Transpose the winograd convolution parameter U, where k, c, ξ are weight index subscripts, 1≤k≤c o , 1≤c≤c i , 1≤ξ≤n 2 :

[0026]

[0027] S1.2.2: According to the transposed winograd convolution parameter U T The positional relationship of the winograd convolution parameters U is grouped and packaged. Specifically, the weights across all input channels at the (i, j) position in each output channel are packaged as a vector to obtain a weight vector set. Where k is the output channel index and 1≤k≤c o , i,j are the weight index subscripts of each two-dimensional convolution kernel, and 1≤i,j≤n;

[0028] S1.2.3: Input each weight vector in the obtained weight vector set g into the following formula to calculate the weight vector importance evaluation score:

[0029] H=||U k,:,i,j ||2

[0030] Among them, H is the importance evaluation score of the weight vector;

[0031] S1.2.4: Sort the calculated importance evaluation scores of the weight vectors in descending order, and delete q% of the weight vectors with the smallest importance evaluation scores, where q is a value between 0 and 100;

[0032] In one embodiment, S3 includes:

[0033] Input the Winograd sparse deep learning model retrained in step S2 into the activation function parameterized approximation module, and generate a corresponding mask tensor according to the learnable parameter sensitivity assigned to each activation function, where the position where the mask tensor value is 1 corresponds to an important activation function, and the position where the tensor value is 0 corresponds to a redundant activation function;

[0034] The generated mask tensor is further initialized with a polynomial and iteratively fitted to obtain a neural network prediction model after approximation of a nonlinear activation function.

[0035] In one embodiment, S3 includes:

[0036] S3.1: Each activation function unit is parameterized according to the following formula:

[0037] σ α (x) = α·σ(x) + (1-α)·x

[0038] Where σ(x) is the neural network activation function, x is the activation function input, and σ α (x) is the parameterized activation function, and α is the additional parameter of the activation function;

[0039] S3.2: Calculate the activation function importance evaluation score based on the activation function additional parameters and parameter gradients:

[0040]

[0041] in, is the parameter gradient, S is the importance evaluation score of the activation function;

[0042] S3.3: Initialize the all-one mask tensor with the same dimension as the activation function input feature map, calculate the activation function importance evaluation score and sort it in descending order, and update M according to the following formula, where is the indicator function and θ is the threshold parameter:

[0043]

[0044] Among them, M is the mask tensor, is the indicator function, θ is the threshold parameter, and M′ is the updated mask tensor;

[0045] S3.4: The server initializes the activation function unit of the deep neural network according to the updated mask tensor using the following formula to obtain the polynomial activation function version of the deep learning model as the neural network prediction model after the nonlinear activation function approximation:

[0046] σ poly (x)=M′·ReLU(X)+(1-M′)·(ρ1·x 2 +ρ2·x+ρ3)

[0047] where σ poly (·) is the polynomial activation function, ρ1=0, ρ2=1, ρ3=0 are the initialized polynomial coefficients.

[0048] In one embodiment, S5 includes:

[0049] The server receives the privacy reasoning service request data from the client. Based on the privacy interactive computing protocol, the server uses the model output by S4 to perform privacy reasoning and finally obtains the model prediction output and returns it to the client.

[0050] Based on the same inventive concept, the second aspect of the present invention provides a neural network privacy reasoning device based on winograd sparse convolution and differentiable activation function approximation, which is applied to a deep neural network privacy reasoning system, including a server and a client, wherein the server holds a training data set for generating a pre-trained model based on the data set training; the client holds a number of privacy reasoning service request data, and the client is used to send the request data to the server for privacy reasoning, and the privacy reasoning device includes:

[0051] The convolution layer sparsification module is used to perform parameter transformation and sparsification on the pre-trained deep learning model to obtain a neural network prediction model after linear convolution layer sparsification;

[0052] The first retraining module is used to retrain the model output by the convolutional layer sparsification module, wherein the accuracy of the neural network prediction model after the linear convolutional layer sparsification is evaluated in each training iteration round. If the model accuracy loss is less than a preset threshold, the linear convolutional layer sparsification is completed to obtain a retrained winograd sparsification deep learning model;

[0053] An activation function parameterization approximation module is used to perform activation function approximation and replacement processing on the model output by the first retraining module to obtain a neural network prediction model after nonlinear activation function approximation;

[0054] The second retraining module is used to retrain the model output by the activation function parameterization approximation module, wherein the model accuracy is tested after each training iteration. If the model accuracy loss is within an acceptable range, the nonlinear activation function approximation is completed to obtain the final optimized model;

[0055] The privacy inference module is used to perform privacy inference based on the model output by the second training module.

[0056] Based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation described in the first aspect.

[0057] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation described in the first aspect is implemented.

[0058] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0059] This paper mainly proposes a neural network privacy reasoning and optimization method based on the winograd convolution algorithm and activation function polynomial fitting technology. This method fully considers the redundancy of the linear convolution layer of the neural network, reduces the communication volume required for convolution multiplication operations by introducing multi-granularity sparsity, and performs sensitivity analysis and adaptive polynomial function approximation on the nonlinear activation function layer, further reducing the latency and communication cost of neural network privacy reasoning, and improving the efficiency of privacy computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 An overall flow chart of a neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation provided in an embodiment of the present invention;

[0062] Figure 2An example diagram of a neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation provided in an embodiment of the present invention;

[0063] Figure 3 A specific flow chart of a neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation provided in an embodiment of the present invention;

[0064] Figure 4 A module diagram of a neural network privacy reasoning device based on winograd sparse convolution and differentiable activation function approximation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The present invention aims to solve the problems of high computational latency and high communication cost in existing privacy-preserving neural network reasoning. Based on the multiplication-efficient convolution characteristics of the Winograd algorithm and combined with a multi-granularity weight pruning algorithm, the present invention maximizes the sparsity of the linear convolution layer and reduces the communication cost required for the multiplication-intensive convolution layer. At the same time, the nonlinear activation function is collaboratively optimized. By designing a new activation function sensitivity and importance quantitative evaluation algorithm, an adaptive learnable polynomial function is introduced to improve the fitting accuracy of the activation function, reduce the polynomial substitution loss, and use a multi-stage knowledge distillation method to retain the powerful prediction performance of the neural network model. Without affecting the prediction accuracy, the computational and communication efficiency of privacy-preserving neural network reasoning is effectively improved.

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Embodiment 1

[0068] The present invention discloses a privacy reasoning method for a neural network based on Winograd sparse convolution and differentiable activation function approximation, which is applied to a deep neural network privacy reasoning system, including a server and a client, wherein the server holds a training data set for generating a pre-trained model based on the training of the data set; the client holds a number of privacy reasoning service request data, and the client is used to send the request data to the server for privacy reasoning, see Figure 1 , the privacy reasoning method includes:

[0069] S1: Perform parameter transformation and sparse processing on the pre-trained deep learning model to obtain a neural network prediction model after linear convolution layer sparsification;

[0070] S2: Retrain the model output by S1, wherein the accuracy of the neural network prediction model after the linear convolution layer is sparse is evaluated in each training iteration round. If the model accuracy loss is less than the preset threshold, the linear convolution layer is sparse, and the retrained winograd sparse deep learning model is obtained;

[0071] S3: Perform activation function approximation and replacement processing on the model output by S2 to obtain a neural network prediction model after nonlinear activation function approximation;

[0072] S4: Retrain the model output by S3, where the model accuracy is tested after each training iteration. If the model accuracy loss is within an acceptable range, the nonlinear activation function is approximated to obtain the final optimized model.

[0073] S5: Perform privacy reasoning based on the model output by S4.

[0074] Specifically, the server holds a training dataset And generate a pre-trained model based on this dataset training The client holds several privacy inference service request data The request data is sent to the server for privacy reasoning. The server is based on the given data set. According to the input x n and its output label y n The mapping relationship between them generates a pre-trained neural network model Where n represents the nth input-label combination, the depth of the model (pre-trained neural network model) is L, and the model weight is l represents the lth layer of the model, the activation function is σ(·), and the entire neural network model contains K activation function units. When step 1 is executed for the first time, the model parameters Use pre-trained completion values;

[0075] S1 is to perform parameter transformation and sparsification processing on the pre-trained deep learning model. S2 is to retrain the model output by step S1 according to the training data set. The number of training rounds depends on the accuracy recovery of the model to obtain the retrained Winograd sparse deep learning model. S3 is to perform activation function approximation and replacement processing on the model output by S2. S4 is to retrain the model output by S3 to obtain the final optimized model. S5 is to perform privacy reasoning based on the final optimized model.

[0076] In one embodiment, S1 includes:

[0077] S1.1: The pre-trained deep learning model is input into the convolutional layer sparsification module, and the importance of the convolution kernel is evaluated through the Winograd algorithm transformation. The redundant convolution kernels are removed from the model according to the importance evaluation results.

[0078] S1.2: Perform weight transposition and remove redundant weights in vector units to obtain the neural network prediction model after linear convolution layer sparsification.

[0079] Specifically, S1 performs winograd-aware convolution kernel pruning and winograd-aware weight vector pruning through the convolution layer sparsification module to delete redundant convolution kernels and redundant weights.

[0080] In one embodiment, S1.1 includes:

[0081] S1.1.1: Multiply the convolution kernel of the input pre-trained deep learning model with the sparse transformation tensor of the winograd convolution algorithm, and calculate the winograd convolution parameters using the following formula:

[0082] U=ΣR·F

[0083] Among them, F is the convolution kernel of the pre-trained deep learning model, R is the sparse transformation tensor of the winograd convolution algorithm, and U is the winograd convolution parameter;

[0084] S1.1.2: Calculate the convolution kernel importance evaluation score based on the convolution kernel of the pre-trained deep learning model and the sparse transformation tensor of the winograd convolution algorithm:

[0085] I=Σ||R·F||1

[0086] Among them, I is the convolution kernel importance evaluation score;

[0087] S1.1.3: Sort the calculated convolution kernel importance evaluation scores in descending order, and delete the p% convolution kernels with the smallest importance evaluation scores as redundant convolution kernels, where p is a value between 0 and 100.

[0088] During the specific implementation, the size of p can be selected according to the actual situation to delete redundant convolution kernels.

[0089] In one embodiment, S1.2 includes:

[0090] S1.2.1: For an input channel number c o , the number of input channels is c i , the lth convolution layer with a convolution kernel size of n×n Transpose the winograd convolution parameter U, where k, c, ξ are weight index subscripts, 1≤k≤c o , 1≤c≤c i , 1≤ξ≤n 2 :

[0091]

[0092] S1.2.2: According to the transposed winograd convolution parameter U T The positional relationship of the winograd convolution parameters U is grouped and packaged. Specifically, the weights across all input channels at the (i, j) position in each output channel are packaged as a vector to obtain a weight vector set. Where k is the output channel index and 1≤k≤c o , i,j are the weight index subscripts of each two-dimensional convolution kernel, and 1≤i,j≤n;

[0093] S1.2.3: The obtained weight vector set Each weight vector in is input into the following formula to calculate the weight vector importance evaluation score:

[0094] H=||U k,:,i,j ||2

[0095] Among them, H is the importance evaluation score of the weight vector;

[0096] S1.2.4: Sort the calculated importance evaluation scores of the weight vectors in descending order, and delete q% of the weight vectors with the smallest importance evaluation scores, where q is a value between 0 and 100.

[0097] During the specific implementation, the size of q can be selected according to the actual situation to delete redundant weight vectors.

[0098] In one embodiment, S3 includes:

[0099] Input the Winograd sparse deep learning model retrained in step S2 into the activation function parameterized approximation module, and generate a corresponding mask tensor according to the learnable parameter sensitivity assigned to each activation function, where the position where the mask tensor value is 1 corresponds to an important activation function, and the position where the tensor value is 0 corresponds to a redundant activation function;

[0100] The generated mask tensor is further initialized with a polynomial and iteratively fitted to obtain a neural network prediction model after approximation of a nonlinear activation function.

[0101] Specifically, the activation function parameterization approximation module mainly performs activation function mask generation based on sensitivity analysis and activation parameterization approximation processing based on distillation. By generating a mask tensor and further performing polynomial initialization and iterative fitting replacement, a neural network prediction model after nonlinear activation function approximation is obtained.

[0102] In one embodiment, S3 includes:

[0103] S3.1: Each activation function unit is parameterized according to the following formula:

[0104] σ α (x) = α·σ(x) + (1-α)·x

[0105] Where σ(x) is the neural network activation function, x is the activation function input, and σ α (x) is the parameterized activation function, and α is the additional parameter of the activation function;

[0106] S3.2: Calculate the activation function importance evaluation score based on the activation function additional parameters and parameter gradients:

[0107]

[0108] in, is the parameter gradient, S is the importance evaluation score of the activation function;

[0109] S3.3: Initialize the all-1 mask tensor with the same dimension as the input feature map of the activation function, calculate the activation function importance evaluation score and sort it in descending order, and update the mask tensor according to the following formula, where is the indicator function and θ is the threshold parameter:

[0110]

[0111] Among them, M is the mask tensor, is the indicator function, θ is the threshold parameter, and M′ is the updated mask tensor;

[0112] S3.4: The server initializes the activation function unit of the deep neural network according to the updated mask tensor using the following formula to obtain the polynomial activation function version of the deep learning model as the neural network prediction model after the nonlinear activation function approximation:

[0113] σ poly (x)=M′·ReLU(X)+(1-M′)·(ρ1·x 2 +ρ2·x+ρ3)

[0114] where σ poly(·) is the polynomial activation function, ρ1=0, ρ2=1, ρ3=0 are the initialized polynomial coefficients.

[0115] After obtaining the neural network prediction model after the nonlinear activation function approximation, the method further includes: the server side calculates the prediction model according to the training data set The model output from step S3.4 is retrained, and the polynomial coefficients ρ1, ρ2, ρ3 of each activation function unit are updated. The number of training rounds depends on the recovery of the model accuracy to obtain the final optimized model.

[0116] In one embodiment, S5 includes:

[0117] The server receives the privacy reasoning service request data from the client. Based on the privacy interactive computing protocol, the server uses the model output by S4 to perform privacy reasoning and finally obtains the model prediction output and returns it to the client.

[0118] The method proposed by the present invention is described below through specific embodiments.

[0119] The method of the present invention is to solve the problems of high latency and high communication cost in existing neural network privacy inference. Figure 2 The neural network convolution layer sparsification and activation function layer parameterization algorithms in illustrate the optimization method for privacy reasoning.

[0120] Bundle Figure 2 The prediction model obtained by the server based on the given data set pre-training is imported as the input neural network into the neural network convolution layer sparsification module. First, it is input into the winograd-aware convolution kernel pruning algorithm. According to the importance scores of different convolution kernels after winograd transformation, some convolution kernels with lower scores are removed. Then it is input into the winograd-aware weight vector pruning algorithm. After the convolution kernel weights are transposed, they are grouped in vector form and the importance scores are calculated. After sorting, some weight vectors with lower scores are removed. After that, the neural network is retrained and then input into the activation function parameterization module. After the activation function mask generation based on sensitivity analysis and the activation function parameterization approximation algorithm based on knowledge distillation, the activation function units with lower sensitivity in the neural network are replaced with parameter-adaptive polynomial functions. After multi-step knowledge distillation training, the final optimized neural network model is obtained and handed over to the server and client for efficient and lightweight privacy reasoning interaction. Figure 3 It is a flowchart of the neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation in the present invention, and the detailed description process is as follows:

[0121] Step 1: Figure 2The pre-trained neural network in is used as the input prediction model, and the model is input into the convolutional layer sparse module. The Winograd algorithm is transformed to calculate and sort the convolution kernel importance scores, and some convolution kernels with lower scores are removed from the model. Then, the weights are transposed, and redundant weights are removed in units of vectors to obtain the neural network prediction model after linear convolution layer sparseness.

[0122] Step 2: Retrain the model output from step 1. The accuracy of the model is evaluated in each training iteration. If the model accuracy loss is less than the preset threshold, the linear convolution layer is sparsified.

[0123] Step 3: Input the neural network prediction model retrained in step 2 into the activation function parameterized approximation module, and generate the corresponding mask tensor according to the sensitivity of the learnable parameters assigned to each activation function. The position with a tensor value of 1 corresponds to a more important activation function and needs to be retained, while the position with a tensor value of 0 corresponds to a redundant activation function, which is further polynomially initialized and iteratively fitted to obtain the neural network prediction model after nonlinear activation function approximation;

[0124] Step 4: Retrain the model output in step 3, test the model accuracy after each training iteration, and complete the nonlinear activation function approximation if the model accuracy loss is within an acceptable range;

[0125] Step 5: Output the neural network prediction model after optimization in step 4 and conduct privacy reasoning interaction between the two parties.

[0126] From the above description, it can be seen that the present invention is based on the multiplication efficient convolution characteristics of the winograd algorithm, combined with a multi-granularity weight pruning algorithm, to maximize the sparsity of the linear convolution layer and reduce the communication cost required for the multiplication dense convolution layer. At the same time, the nonlinear activation function is collaboratively optimized, and a new activation function sensitivity and importance quantitative evaluation algorithm is designed, and an adaptive learnable polynomial function is introduced to improve the fitting accuracy of the activation function, reduce the polynomial replacement loss, and use a multi-stage knowledge distillation method to retain the powerful prediction performance of the neural network model, effectively improving the computational and communication efficiency of privacy protection neural network reasoning without affecting the prediction accuracy.

[0127] Embodiment 2

[0128] Based on the same inventive concept, this embodiment discloses a neural network privacy reasoning device based on winograd sparse convolution and differentiable activation function approximation, which is applied to a deep neural network privacy reasoning system, including a server and a client, wherein the server holds a training data set for generating a pre-trained model based on the data set training; the client holds a number of privacy reasoning service request data, and the client is used to send the request data to the server for privacy reasoning, see Figure 4 , the privacy inference device comprises:

[0129] The convolution layer sparsification module 401 is used to perform parameter transformation and sparsification processing on the pre-trained deep learning model to obtain a neural network prediction model after linear convolution layer sparsification;

[0130] The first retraining module 402 is used to retrain the model output by the convolutional layer sparsification module, wherein the accuracy of the neural network prediction model after the linear convolutional layer sparsification is evaluated in each training iteration round, and if the model accuracy loss is less than a preset threshold, the linear convolutional layer sparsification is completed to obtain a retrained Winograd sparsification deep learning model;

[0131] An activation function parameterization approximation module 403 is used to perform activation function approximation and replacement processing on the model output by the first retraining module to obtain a neural network prediction model after nonlinear activation function approximation;

[0132] A second retraining module 404 is used to retrain the model output by the activation function parameterization approximation module, wherein the model accuracy is tested after each training iteration round, and if the model accuracy loss is within an acceptable range, the nonlinear activation function approximation is completed;

[0133] The privacy reasoning module 405 is used to perform privacy reasoning based on the model output by the second training module.

[0134] Since the device introduced in the second embodiment of the present invention is a device used to implement the privacy inference method of the neural network based on winograd sparse convolution and differentiable activation function approximation in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the device, so it is not repeated here. All devices used in the method in the first embodiment of the present invention belong to the scope of protection of the present invention.

[0135] Embodiment 3

[0136] Based on the same inventive concept, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the first embodiment is implemented.

[0137] Since the computer-readable storage medium introduced in the third embodiment of the present invention is a computer-readable storage medium used to implement the privacy inference method of the neural network based on winograd sparse convolution and differentiable activation function approximation in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer-readable storage medium, so it is not repeated here. All computer-readable storage media used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0138] Embodiment 4

[0139] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in Embodiment 1 when executing the program.

[0140] Since the computer device introduced in the fourth embodiment of the present invention is a computer device used to implement the privacy inference method of the neural network based on Winograd sparse convolution and differentiable activation function approximation in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer device, so it is not repeated here. All computer devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0141] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic creative concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation, characterized in that: The system is applied to a privacy-preserving inference system of a deep neural network, and includes a server and a client. The server holds a training data set, and is used to generate a pre-trained model based on the training data set. The client holds a number of privacy-preserving inference service request data, and the client is used to send the request data to the server for privacy-preserving inference. The privacy-preserving inference method includes: S1: Perform parameter transformation and sparse processing on the pre-trained deep learning model to obtain a neural network prediction model after linear convolution layer sparsification; S2: Retrain the model output by S1, wherein the accuracy of the neural network prediction model after the linear convolution layer is sparse is evaluated in each training iteration round. If the model accuracy loss is less than the preset threshold, the linear convolution layer is sparse, and the retrained winograd sparse deep learning model is obtained; S3: Perform activation function approximation and replacement processing on the model output by S2 to obtain a neural network prediction model after nonlinear activation function approximation; S4: Retrain the model output by S3, where the model accuracy is tested after each training iteration. If the model accuracy loss is within an acceptable range, the nonlinear activation function is approximated to obtain the final optimized model. S5: Perform privacy reasoning based on the model output by S4.

2. The neural network privacy inference method based on winograd sparse convolution and differentiable activation function approximation as claimed in claim 1, characterized in that: S1 includes: S1.1: The pre-trained deep learning model is input into the convolutional layer sparsification module, and the importance of the convolution kernel is evaluated through the Winograd algorithm transformation. The redundant convolution kernels are removed from the model according to the importance evaluation results. S1.2: Perform weight transposition and remove redundant weights in vector units to obtain the neural network prediction model after linear convolution layer sparsification.

3. The neural network privacy inference method based on winograd sparse convolution and differentiable activation function approximation as claimed in claim 2, characterized in that: S1.1 includes: S1.1.1: Multiply the convolution kernel of the input pre-trained deep learning model with the sparse transformation tensor of the winograd convolution algorithm, and calculate the winograd convolution parameters using the following formula: U=ΣR·F Among them, F is the convolution kernel of the pre-trained deep learning model, R is the sparse transformation tensor of the winograd convolution algorithm, and U is the winograd convolution parameter; S1.1.2: Calculate the convolution kernel importance evaluation score based on the convolution kernel of the pre-trained deep learning model and the sparse transformation tensor of the winograd convolution algorithm: I=Σ‖R·F‖1 Among them, I is the convolution kernel importance evaluation score; S1.1.3: Sort the calculated convolution kernel importance evaluation scores in descending order, and delete the p% convolution kernels with the smallest importance evaluation scores as redundant convolution kernels, where p is a value between 0 and 100.

4. The neural network privacy inference method based on winograd sparse convolution and differentiable activation function approximation as claimed in claim 3, characterized in that: S1.2 includes: S1.2.1: For an input channel number c o , the number of input channels is c i , the lth convolution layer with a convolution kernel size of n×n Transpose the winograd convolution parameter U, where k, c, ξ are weight index subscripts, 1≤k≤c o , 1≤c≤c i , 1≤ξ≤n 2 : S1.2.2: According to the transposed winograd convolution parameter U T The positional relationship of the winograd convolution parameters U is grouped and packaged. Specifically, the weights across all input channels at the (i, j) position in each output channel are packaged as a vector to obtain a weight vector set. Where k is the output channel index and 1≤k≤c o , i,j are the weight index subscripts of each two-dimensional convolution kernel, and 1≤i,j≤n; S1.2.3: The obtained weight vector set Each weight vector in is input into the following formula to calculate the weight vector importance evaluation score: H=||U k,:,i,j ||2 Among them, H is the importance evaluation score of the weight vector; S1.2.4: Sort the calculated importance evaluation scores of the weight vectors in descending order, and delete q% of the weight vectors with the smallest importance evaluation scores, where q is a value between 0 and 100.

5. The neural network privacy inference method based on winograd sparse convolution and differentiable activation function approximation as claimed in claim 1, characterized in that S3 include: Input the Winograd sparse deep learning model retrained in step S2 into the activation function parameterized approximation module, and generate a corresponding mask tensor according to the learnable parameter sensitivity assigned to each activation function, where the position where the mask tensor value is 1 corresponds to an important activation function, and the position where the tensor value is 0 corresponds to a redundant activation function; The generated mask tensor is further initialized with a polynomial and iteratively fitted to obtain a neural network prediction model after approximation of a nonlinear activation function.

6. The neural network privacy inference method based on winograd sparse convolution and differentiable activation function approximation as claimed in claim 5, characterized in that S3 include: S3.1: Each activation function unit is parameterized according to the following formula: s α (x)=α·σ(x)+(1-α)·x Where σ(x) is the neural network activation function, x is the activation function input, and σ α (x) is the parameterized activation function, and α is the additional parameter of the activation function; S3.2: Calculate the activation function importance evaluation score based on the activation function additional parameters and parameter gradients: in, is the parameter gradient, S is the importance evaluation score of the activation function; S3.3: Initialize the all-one mask tensor with the same dimension as the activation function input feature map, calculate the activation function importance evaluation score and sort it in descending order, and update M according to the following formula, where is the indicator function and θ is the threshold parameter: Among them, M is the mask tensor, is the indicator function, θ is the threshold parameter, and M′ is the updated mask tensor; S3.4: The server initializes the activation function unit of the deep neural network according to the updated mask tensor using the following formula to obtain the polynomial activation function version of the deep learning model as the neural network prediction model after the nonlinear activation function approximation: s poly (x)=M′·ReLU(X)+(1-M′)·(ρ1·x 2 +ρ2·x+ρ3) where σ poly (·) is the polynomial activation function, ρ1=0, ρ2=1, ρ3=0 are the initialized polynomial coefficients.

7. The neural network privacy inference method based on winograd sparse convolution and differentiable activation function approximation as claimed in claim 1, characterized in that S5 include: The server receives the privacy reasoning service request data from the client. Based on the privacy interactive computing protocol, the server uses the model output by S4 to perform privacy reasoning and finally obtains the model prediction output and returns it to the client.

8. A neural network privacy inference device based on Winograd sparse convolution and differentiable activation function approximation, characterized in that: The system is applied to a privacy-preserving inference system of a deep neural network, and includes a server and a client. The server holds a training data set, and is used to generate a pre-trained model based on the training data set. The client holds a number of privacy-preserving inference service request data, and the client is used to send the request data to the server for privacy-preserving inference. The privacy-preserving inference device includes: The convolution layer sparsification module is used to perform parameter transformation and sparsification on the pre-trained deep learning model to obtain a neural network prediction model after linear convolution layer sparsification; The first retraining module is used to retrain the model output by the convolutional layer sparsification module, wherein the accuracy of the neural network prediction model after the linear convolutional layer sparsification is evaluated in each training iteration round. If the model accuracy loss is less than a preset threshold, the linear convolutional layer sparsification is completed to obtain a retrained winograd sparsification deep learning model; An activation function parameterization approximation module is used to perform activation function approximation and replacement processing on the model output by the first retraining module to obtain a neural network prediction model after nonlinear activation function approximation; The second retraining module is used to retrain the model output by the activation function parameterization approximation module, wherein the model accuracy is tested after each training iteration. If the model accuracy loss is within an acceptable range, the nonlinear activation function approximation is completed to obtain the final optimized model; The privacy reasoning module is used to perform privacy reasoning based on the model output by the second training module.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation is implemented as described in any one of claims 1 to 7.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation as described in any one of claims 1 to 7.

Images