Neural network privacy inference method and device based on Winograd sparse convolution and differentiable activation function approximation

Abstract

The present invention belongs to the field of artificial intelligence information security technology and specifically discloses a neural network privacy inference method and device based on Winograd sparse convolution and differentiable activation function approximation. To address the high computational latency and high communication cost of existing neural network privacy inference, the Winograd algorithm's efficient multiplication convolution characteristics, combined with a multi-granularity weight pruning algorithm, maximizes the sparsity of linear convolution layers and reduces the communication cost required by multiplication-intensive convolution layers. Simultaneously, nonlinear activation functions are collaboratively optimized. By designing a new activation function sensitivity and importance quantitative evaluation algorithm and introducing an adaptive learnable polynomial function, the fitting accuracy of the activation function is improved, the polynomial substitution loss is reduced, and a multi-stage knowledge distillation method is used to retain the strong predictive performance of the neural network model. This effectively improves the computational and communication efficiency of privacy-preserving neural network inference without affecting prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence information security technology, and is a deep neural network reasoning method for privacy protection, and in particular to a neural network privacy reasoning method and device based on Winograd sparse convolution and differentiable activation function approximation. Background Art

[0002] A new generation of artificial intelligence (AI), represented by autonomous driving, intelligent question-answering, and content generation, is leading the third information technology revolution. These emerging technologies are closely integrated with applications such as smart healthcare, financial forecasting, and risk assessment. Data from a wider range of sources, in more diverse formats, and with richer content are rapidly driving transformation and leapfrogging in these fields. According to statistics, the amount of data generated globally will grow from 33 zettabytes (ZB) in 2018 to 175 ZB in 2025. Large-scale data is becoming a key driver of deep learning technology development. The efficient training and accurate inference of the next generation of high-performance neural network models rely heavily on massive amounts of high-quality data and powerful computing resources, both of which are primarily concentrated in large technology companies (e.g., Google, Meta, and Microsoft abroad, and Alibaba, Tencent, and Baidu in China). Consequently, deep learning models are often viewed as digital assets with intellectual property rights, from which tech companies seek to profit. Meanwhile, ordinary users tend to outsource their data to service providers for the heavy lifting of model inference computations, addressing challenges such as limited local computing resources and leveraging high-performance deep learning models to unlock the potential value of their data. The interactive data between users and service providers often contains sensitive information. Striking a balance between data privacy and value, achieving efficient and secure data sharing and collaborative computing has become a pressing challenge. Deep neural network privacy inference, a technology that can perform inference calculations while data is encrypted or protected, offers a viable solution to this challenge.

[0003] Deep neural network privacy inference methods primarily rely on homomorphic encryption (HE) and secure multi-party computation (MPC) technologies to ensure data security and confidentiality. However, this also significantly increases the computational and communication costs of privacy inference, resulting in limited application and low computational efficiency in practical scenarios. To this end, researchers have proposed a series of optimization methods to reduce the computational and communication costs of privacy inference in deep neural networks, primarily including optimization frameworks based on privacy protocols and model architectures. Privacy inference optimization methods based on privacy protocols involve designing lightweight privacy computing interaction protocols, reducing the computational complexity of both parties or reducing the amount of communication data through methods such as ciphertext packing or vector encoding. Model architecture-based privacy inference optimization methods involve designing privacy-friendly replacement or approximation schemes for the computational modules in neural network models to improve the privacy inference efficiency of neural network models.

[0004] Compared to privacy-preserving protocol-based optimization methods, model-architecture-based optimization methods can better leverage the parameter and architectural redundancy of neural network models, significantly reducing computational latency and improving communication efficiency while ensuring high-precision privacy-preserving inference. These methods can also be combined with the former to achieve even better optimization results. To address the challenges of homomorphic encryption and secure multi-party computation (SSL / TLS) in efficiently performing nonlinear computations, existing model-architecture-based methods primarily optimize the nonlinear activation functions of neural networks, assuming that multiplication operations in convolutional layers during online inference are as efficient as plaintext computations and require no optimization. However, with the continuous advancement of privacy-preserving computing technologies such as oblivious transfer (OT) and secret sharing (SS), the efficiency of computing ReLU (Rectified linear units) nonlinear functions has significantly improved. However, in parallel privacy-preserving inference scenarios, the communication cost of convolutional multiplication operations gradually increases as a percentage of the overall privacy-preserving inference process. Existing optimization methods suffer from low computational and communication efficiency. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a neural network privacy reasoning method based on Winograd sparse convolution and differentiable activation function approximation.

[0006] The technical solution adopted in the present invention is as follows:

[0007] In a first aspect, a neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation is provided. The method is applied to a deep neural network privacy inference system, including a server and a client. The server holds a training dataset and is used to generate a pre-trained model based on the dataset training. The client holds a number of privacy inference service request data and is used to send the request data to the server for privacy inference. The privacy inference method includes:

[0008] S1: Perform parameter transformation and sparsification on the pre-trained deep learning model to obtain a neural network prediction model after the linear convolution layer is sparsified;

[0009] S2: Retrain the model output by S1. In each training iteration, the accuracy of the neural network prediction model after the linear convolution layer is sparsified is evaluated. If the model accuracy loss is less than a preset threshold, the linear convolution layer sparsification is completed, and a retrained Winograd sparsified deep learning model is obtained.

[0010] S3: Perform activation function approximation and replacement processing on the model output by S2 to obtain a neural network prediction model after nonlinear activation function approximation;

[0011] S4: Retrain the model output by S3. After each training iteration, the model accuracy is tested. If the model accuracy loss is within an acceptable range, the nonlinear activation function is approximated to obtain the final optimized model.

[0012] S5: Perform privacy reasoning based on the model output by S4.

[0013] In one embodiment, S1 includes:

[0014] S1.1: The pre-trained deep learning model is input into the convolutional layer sparsification module, and the convolution kernel importance is evaluated through the Winograd algorithm. Redundant convolution kernels are removed from the model based on the importance evaluation results.

[0015] S1.2: Perform weight transposition and remove redundant weights in vector units to obtain the neural network prediction model after linear convolution layer sparsification.

[0016] In one embodiment, S1.1 includes:

[0017] S1.1.1: Multiply the convolution kernel of the input pre-trained deep learning model with the sparse transformation tensor of the Winograd convolution algorithm and calculate the Winograd convolution parameters using the following formula:

[0018] U=ΣR·F

[0019] Among them, F is the convolution kernel of the pre-trained deep learning model, R is the sparse transformation tensor of the winograd convolution algorithm, and U is the winograd convolution parameter;

[0020] S1.1.2: Calculate the convolution kernel importance evaluation score based on the convolution kernel of the pre-trained deep learning model and the sparse transformation tensor of the Winograd convolution algorithm:

[0021] I=∑||R·F||1

[0022] Among them, I is the convolution kernel importance evaluation score;

[0023] S1.1.3: Sort the calculated convolution kernel importance evaluation scores in descending order, and delete the p% convolution kernels with the smallest importance evaluation scores as redundant convolution kernels, where p is a value between 0 and 100.

[0024] In one embodiment, S1.2 includes:

[0025] S1.2.1: For an input channel number c o , the number of input channels is c i , the lth convolution layer with a convolution kernel size of n×n Transpose the winograd convolution parameter U, where k, c, ξ are weight index subscripts, 1≤k≤c o , 1≤c≤c i , 1≤ξ≤n 2 :

[0026]

[0027] S1.2.2: According to the transposed winograd convolution parameter U T The positional relationship is used to group and pack the winograd convolution parameters U. Specifically, the weights across all input channels at the (i, j) position in each output channel are packed as a vector to obtain a weight vector set. Where k is the output channel index and 1≤k≤c o , i, j are the weight index subscripts of each two-dimensional convolution kernel, and 1≤i, j≤n;

[0028] S1.2.3: Input each weight vector in the obtained weight vector set g into the following formula to calculate the weight vector importance evaluation score:

[0029] H=||U k,:,i,j ||2

[0030] Among them, H is the importance evaluation score of the weight vector;

[0031] S1.2.4: Sort the calculated weight vector importance evaluation scores in descending order, and delete the q% weight vectors with the smallest importance evaluation scores, where q is a value between 0 and 100;

[0032] In one embodiment, S3 includes:

[0033] Input the Winograd sparse deep learning model retrained in step S2 into the activation function parameterization approximation module, and generate a corresponding mask tensor based on the sensitivity of the learnable parameters assigned to each activation function, where the positions where the mask tensor value is 1 correspond to important activation functions, and the positions where the tensor value is 0 correspond to redundant activation functions;

[0034] The generated mask tensor is further initialized with polynomials and iteratively fitted to obtain a neural network prediction model after approximation of the nonlinear activation function.

[0035] In one embodiment, S3 includes:

[0036] S3.1: Parameterize each activation function unit according to the following formula:

[0037] σ α (x) = α·σ(x) + (1-α)·x

[0038] Where σ(x) is the neural network activation function, x is the activation function input, σ α (x) is the parameterized activation function, and α is the additional parameter of the activation function;

[0039] S3.2: Calculate the activation function importance evaluation score based on the activation function additional parameters and parameter gradients:

[0040]

[0041] in, is the parameter gradient, S is the importance evaluation score of the activation function;

[0042] S3.3: Initialize an all-one mask tensor with the same dimension as the activation function input feature map, calculate the activation function importance evaluation score and sort it in descending order, and update M according to the following formula, where is the indicator function and θ is the threshold parameter:

[0043]

[0044] Among them, M is the mask tensor, is the indicator function, θ is the threshold parameter, and M′ is the updated mask tensor;

[0045] S3.4: The server initializes the activation function unit of the deep neural network based on the updated mask tensor using the following formula to obtain a polynomial activation function version of the deep learning model, which serves as the neural network prediction model after nonlinear activation function approximation:

[0046] σ poly (x)=M′·ReLU(X)+(1-M′)·(ρ1·x 2 +ρ2·x+ρ3)

[0047] where σ poly (·) is the polynomial activation function, ρ1=0, ρ2=1, ρ3=0 are the initialized polynomial coefficients.

[0048] In one embodiment, S5 includes:

[0049] The server receives the privacy reasoning service request data from the client. Based on the privacy interactive computing protocol, the server uses the model output by S4 to perform privacy reasoning and finally obtains the model prediction output and returns it to the client.

[0050] Based on the same inventive concept, the second aspect of the present invention provides a neural network privacy reasoning device based on Winograd sparse convolution and differentiable activation function approximation, which is applied to a deep neural network privacy reasoning system. The device includes a server and a client. The server holds a training dataset and is used to generate a pre-trained model based on the dataset training. The client holds a number of privacy reasoning service request data and is used to send the request data to the server for privacy reasoning. The privacy reasoning device includes:

[0051] The convolution layer sparsification module is used to perform parameter transformation and sparsification on the pre-trained deep learning model to obtain a neural network prediction model after linear convolution layer sparsification.

[0052] The first retraining module is used to retrain the model output by the convolutional layer sparsification module. The accuracy of the neural network prediction model after linear convolutional layer sparsification is evaluated in each training iteration. If the model accuracy loss is less than a preset threshold, the linear convolutional layer sparsification is completed to obtain a retrained Winograd sparsified deep learning model.

[0053] An activation function parameterization approximation module is used to perform activation function approximation and replacement processing on the model output by the first retraining module to obtain a neural network prediction model after nonlinear activation function approximation;

[0054] The second retraining module is used to retrain the model output by the activation function parameterization approximation module. The model accuracy is tested after each training iteration. If the model accuracy loss is within an acceptable range, the nonlinear activation function approximation is completed to obtain the final optimized model.

[0055] Privacy reasoning module, used for privacy reasoning based on the model output by the second training module

[0056] Based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation described in the first aspect.

[0057] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements the neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation described in the first aspect.

[0058] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0059] This paper proposes a neural network privacy inference and optimization method based on the Winograd convolution algorithm and activation function polynomial fitting technology. This method fully considers the redundancy of the linear convolution layer of the neural network. By introducing multi-granularity sparsity, it reduces the communication required for convolution multiplication operations. Furthermore, it performs sensitivity analysis and adaptive polynomial function approximation on the nonlinear activation function layer, further reducing the latency and communication cost of neural network privacy inference and improving the efficiency of privacy computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 An overall flowchart of a neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation provided by an embodiment of the present invention;

[0062] Figure 2An example diagram of a neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation provided by an embodiment of the present invention;

[0063] Figure 3 A specific flowchart of the neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation provided in an embodiment of the present invention;

[0064] Figure 4 A block diagram of a neural network privacy inference device based on Winograd sparse convolution and differentiable activation function approximation, provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] This invention addresses the challenges of high computational latency and high communication costs in existing privacy-preserving neural network inference. Based on the multiplication-efficient convolution characteristics of the Winograd algorithm and combined with a multi-granularity weight pruning algorithm, it maximizes the sparsity of linear convolutional layers and reduces the communication costs required for multiplication-intensive convolutional layers. It also collaboratively optimizes nonlinear activation functions. By designing a new activation function sensitivity and importance quantitative evaluation algorithm, introducing an adaptive learnable polynomial function, improving the fitting accuracy of the activation function and reducing polynomial substitution loss, and using a multi-stage knowledge distillation method to retain the powerful predictive performance of the neural network model, this invention effectively improves the computational and communication efficiency of privacy-preserving neural network inference without affecting prediction accuracy.

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0067] Example 1

[0068] This invention discloses a neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation, which is applied to a deep neural network privacy inference system. The method includes a server and a client. The server holds a training dataset and is used to generate a pre-trained model based on the dataset training. The client holds a number of privacy inference service request data and is used to send the request data to the server for privacy inference. Figure 1 , the privacy reasoning method includes:

[0069] S1: Perform parameter transformation and sparsification on the pre-trained deep learning model to obtain a neural network prediction model after the linear convolution layer is sparsified;

[0070] S2: Retrain the model output by S1. In each training iteration, the accuracy of the neural network prediction model after the linear convolution layer is sparsified is evaluated. If the model accuracy loss is less than a preset threshold, the linear convolution layer sparsification is completed, and a retrained Winograd sparsified deep learning model is obtained.

[0071] S3: Perform activation function approximation and replacement processing on the model output by S2 to obtain a neural network prediction model after nonlinear activation function approximation;

[0072] S4: Retrain the model output by S3. After each training iteration, the model accuracy is tested. If the model accuracy loss is within an acceptable range, the nonlinear activation function is approximated to obtain the final optimized model.

[0073] S5: Perform privacy reasoning based on the model output by S4.

[0074] Specifically, the server holds a training dataset And generate a pre-trained model based on this dataset training The client holds several privacy inference service request data The request data is sent to the server for privacy reasoning. The server is based on the given data set. According to the input x n and its output label y n The mapping relationship between them generates a pre-trained neural network model Where n represents the nth input-label combination, the depth of the model (pre-trained neural network model) is L, and the model weight is l represents the lth layer of the model, the activation function is σ(·), and the entire neural network model contains K activation function units. When step 1 is executed for the first time, the model parameters Use pre-trained completion values;

[0075] S1 performs parameter transformation and sparsification on the pre-trained deep learning model. S2 retrains the model output by step S1 based on the training dataset. The number of training rounds depends on the accuracy recovery of the model, and a retrained Winograd sparsified deep learning model is obtained. S3 performs activation function approximation and replacement on the model output by S2. S4 retrains the model output by S3 to obtain the final optimized model. S5 performs privacy reasoning based on the final optimized model.

[0076] In one embodiment, S1 includes:

[0077] S1.1: The pre-trained deep learning model is input into the convolutional layer sparsification module, and the convolution kernel importance is evaluated through the Winograd algorithm. Redundant convolution kernels are removed from the model based on the importance evaluation results.

[0078] S1.2: Perform weight transposition and remove redundant weights in vector units to obtain the neural network prediction model after linear convolution layer sparsification.

[0079] Specifically, S1 performs Winograd-aware convolution kernel pruning and Winograd-aware weight vector pruning through the convolution layer sparsification module to remove redundant convolution kernels and redundant weights.

[0080] In one embodiment, S1.1 includes:

[0081] S1.1.1: Multiply the convolution kernel of the input pre-trained deep learning model with the sparse transformation tensor of the Winograd convolution algorithm and calculate the Winograd convolution parameters using the following formula:

[0082] U=ΣR·F

[0083] Among them, F is the convolution kernel of the pre-trained deep learning model, R is the sparse transformation tensor of the winograd convolution algorithm, and U is the winograd convolution parameter;

[0084] S1.1.2: Calculate the convolution kernel importance evaluation score based on the convolution kernel of the pre-trained deep learning model and the sparse transformation tensor of the Winograd convolution algorithm:

[0085] I=Σ||R·F||1

[0086] Among them, I is the convolution kernel importance evaluation score;

[0087] S1.1.3: Sort the calculated convolution kernel importance evaluation scores in descending order, and delete the p% convolution kernels with the smallest importance evaluation scores as redundant convolution kernels, where p is a value between 0 and 100.

[0088] During the specific implementation process, the size of p can be selected according to the actual situation to delete redundant convolution kernels.

[0089] In one embodiment, S1.2 includes:

[0090] S1.2.1: For an input channel number c o , the number of input channels is c i , the lth convolution layer with a convolution kernel size of n×n Transpose the winograd convolution parameter U, where k, c, ξ are weight index subscripts, 1≤k≤c o , 1≤c≤c i , 1≤ξ≤n 2 :

[0091]

[0092] S1.2.2: According to the transposed winograd convolution parameter U T The positional relationship is used to group and pack the winograd convolution parameters U. Specifically, the weights across all input channels at the (i, j) position in each output channel are packed as a vector to obtain a weight vector set. Where k is the output channel index and 1≤k≤c o , i, j are the weight index subscripts of each two-dimensional convolution kernel, and 1≤i, j≤n;

[0093] S1.2.3: The obtained weight vector set Each weight vector in is input into the following formula to calculate the weight vector importance evaluation score:

[0094] H=||U k,:,i,j ||2

[0095] Among them, H is the importance evaluation score of the weight vector;

[0096] S1.2.4: Sort the calculated importance evaluation scores of the weight vectors in descending order, and delete q% of the weight vectors with the smallest importance evaluation scores, where q is a value between 0 and 100.

[0097] During the specific implementation process, the size of q can be selected according to the actual situation to delete redundant weight vectors.

[0098] In one embodiment, S3 includes:

[0099] Input the Winograd sparse deep learning model retrained in step S2 into the activation function parameterization approximation module, and generate a corresponding mask tensor based on the sensitivity of the learnable parameters assigned to each activation function, where the positions where the mask tensor value is 1 correspond to important activation functions, and the positions where the tensor value is 0 correspond to redundant activation functions;

[0100] The generated mask tensor is further initialized with polynomials and iteratively fitted to obtain a neural network prediction model after approximation of the nonlinear activation function.

[0101] Specifically, the activation function parameterization approximation module mainly performs activation function mask generation based on sensitivity analysis and activation parameterization approximation processing based on distillation. By generating a mask tensor and further performing polynomial initialization and iterative fitting replacement, a neural network prediction model after nonlinear activation function approximation is obtained.

[0102] In one embodiment, S3 includes:

[0103] S3.1: Parameterize each activation function unit according to the following formula:

[0104] σ α (x) = α·σ(x) + (1-α)·x

[0105] Where σ(x) is the neural network activation function, x is the activation function input, σ α (x) is the parameterized activation function, and α is the additional parameter of the activation function;

[0106] S3.2: Calculate the activation function importance evaluation score based on the activation function additional parameters and parameter gradients:

[0107]

[0108] in, is the parameter gradient, S is the importance evaluation score of the activation function;

[0109] S3.3: Initialize an all-one mask tensor with the same dimension as the activation function input feature map, calculate the activation function importance evaluation score and sort it in descending order, and update the mask tensor according to the following formula, where is the indicator function and θ is the threshold parameter:

[0110]

[0111] Among them, M is the mask tensor, is the indicator function, θ is the threshold parameter, and M′ is the updated mask tensor;

[0112] S3.4: The server initializes the activation function unit of the deep neural network based on the updated mask tensor using the following formula to obtain a polynomial activation function version of the deep learning model, which serves as the neural network prediction model after nonlinear activation function approximation:

[0113] σ poly (x)=M′·ReLU(X)+(1-M′)·(ρ1·x 2 +ρ2·x+ρ3)

[0114] where σ poly(·) is the polynomial activation function, ρ1=0, ρ2=1, ρ3=0 are the initialized polynomial coefficients.

[0115] After obtaining the neural network prediction model after the nonlinear activation function approximation, the method further includes: the server side calculates the prediction model based on the training data set. The model output from step S3.4 is retrained, and the polynomial coefficients ρ1, ρ2, ρ3 of each activation function unit are updated. The number of training rounds depends on the recovery of the model accuracy to obtain the final optimized model.

[0116] In one embodiment, S5 includes:

[0117] The server receives the privacy reasoning service request data from the client. Based on the privacy interactive computing protocol, the server uses the model output by S4 to perform privacy reasoning and finally obtains the model prediction output and returns it to the client.

[0118] The method proposed in the present invention is described below through specific examples.

[0119] The method of the present invention is to solve the problems of high computational delay and high communication cost in existing neural network privacy inference. Figure 2 The neural network convolution layer sparsification and activation function layer parameterization algorithms in illustrate the optimization method for privacy reasoning.

[0120] Bundle Figure 2 The server-side prediction model, pre-trained on a given dataset, is imported as the input neural network into the convolutional layer sparsification module. It is first fed into a Winograd-aware convolutional kernel pruning algorithm, which removes kernels with low importance scores based on their Winograd-transformed importance scores. It is then fed into a Winograd-aware weight vector pruning algorithm, which transposes the kernel weights and groups them into vectors. The importance scores are then calculated and sorted, and the weight vectors with low scores are removed. The neural network is then retrained and fed into the activation function parameterization module. After a sensitivity analysis-based activation function mask generation and a knowledge distillation-based activation function parameterization approximation algorithm, the less sensitive activation function units in the neural network are replaced with parameter-adaptive polynomial functions. After multi-step knowledge distillation training, the optimized neural network model is finally obtained and passed to the server and client for efficient and lightweight privacy-preserving reasoning interactions. Figure 3 This is a flowchart of the neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation in the present invention. The detailed description process is as follows:

[0121] Step 1, Figure 2The pre-trained neural network in is used as the input prediction model, which is input into the convolutional layer sparsification module. The Winograd algorithm is transformed to calculate and sort the convolution kernel importance scores. The convolution kernels with lower scores are removed from the model. Then, the weights are transposed and redundant weights are removed in vector units to obtain the neural network prediction model after linear convolution layer sparsification.

[0122] Step 2: Retrain the model output from step 1. The accuracy of the model is evaluated in each training iteration. If the model accuracy loss is less than the preset threshold, the linear convolution layer is sparsified.

[0123] Step 3: Input the neural network prediction model retrained in Step 2 into the activation function parameterized approximation module. Based on the sensitivity of the learnable parameters assigned to each activation function, a corresponding mask tensor is generated. Positions with a tensor value of 1 correspond to more important activation functions and need to be retained, while positions with a tensor value of 0 correspond to redundant activation functions. These are further polynomially initialized and iteratively fitted to obtain the neural network prediction model after nonlinear activation function approximation.

[0124] Step 4: Retrain the model output in step 3 and test the model accuracy after each training iteration. If the model accuracy loss is within an acceptable range, the nonlinear activation function approximation is completed.

[0125] Step 5: Output the neural network prediction model after optimization in step 4 and conduct privacy reasoning interaction between the two parties.

[0126] As can be seen from the above description, this invention maximizes the sparsity of linear convolution layers and reduces the communication cost of multiplication-intensive convolution layers based on the multiplication-efficient convolution characteristics of the Winograd algorithm, combined with a multi-granularity weight pruning algorithm. At the same time, it collaboratively optimizes nonlinear activation functions. By designing a new activation function sensitivity and importance quantitative evaluation algorithm, introducing an adaptive learnable polynomial function, improving the fitting accuracy of the activation function, reducing polynomial substitution loss, and using a multi-stage knowledge distillation method to retain the powerful predictive performance of the neural network model, it effectively improves the computational and communication efficiency of privacy-preserving neural network reasoning without affecting prediction accuracy.

[0127] Example 2

[0128] Based on the same inventive concept, this embodiment discloses a neural network privacy inference device based on Winograd sparse convolution and differentiable activation function approximation, which is applied to a deep neural network privacy inference system. The device includes a server and a client. The server holds a training dataset for generating a pre-trained model based on the dataset training; the client holds a number of privacy inference service request data, which the client sends to the server for privacy inference. Figure 4 , the privacy inference device includes:

[0129] The convolution layer sparsification module 401 is used to perform parameter transformation and sparsification processing on the pre-trained deep learning model to obtain a neural network prediction model after linear convolution layer sparsification;

[0130] A first retraining module 402 is configured to retrain the model output by the convolutional layer sparsification module, wherein the accuracy of the neural network prediction model after the linear convolutional layer sparsification is evaluated in each training iteration round. If the model accuracy loss is less than a preset threshold, the linear convolutional layer sparsification is completed to obtain a retrained Winograd sparsified deep learning model;

[0131] An activation function parameterization approximation module 403 is used to perform activation function approximation and replacement processing on the model output by the first retraining module to obtain a neural network prediction model after nonlinear activation function approximation;

[0132] A second retraining module 404 is configured to retrain the model output by the activation function parameterization approximation module, wherein the model accuracy is tested after each training iteration. If the loss of model accuracy is within an acceptable range, the nonlinear activation function approximation is completed;

[0133] The privacy reasoning module 405 is used to perform privacy reasoning based on the model output by the second training module.

[0134] Since the device described in Example 2 of the present invention is used to implement the neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation described in Example 1 of the present invention, the specific structure and variations of this device are readily understood by those skilled in the art based on the method described in Example 1 of the present invention, and thus will not be further described here. All devices used in the method described in Example 1 of the present invention fall within the scope of protection of the present invention.

[0135] Example 3

[0136] Based on the same inventive concept, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the first embodiment is implemented.

[0137] Since the computer-readable storage medium described in Example 3 of the present invention is used to implement the neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation described in Example 1 of the present invention, the specific structure and variations of this computer-readable storage medium are readily understood by those skilled in the art based on the method described in Example 1 of the present invention, and thus will not be further described here. All computer-readable storage media used in the method of Example 1 of the present invention fall within the scope of protection of the present invention.

[0138] Example 4

[0139] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first embodiment when executing the program.

[0140] Since the computer device described in Example 4 of the present invention is used to implement the neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation in Example 1 of the present invention, the specific structure and variations of this computer device are readily understood by those skilled in the art based on the method described in Example 1 of the present invention, and therefore will not be further described here. All computer devices used in the method of Example 1 of the present invention fall within the scope of protection of the present invention.

[0141] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, the present invention is intended to include such changes and modifications to the embodiments of the present invention if they fall within the scope of the claims and their equivalents.

Claims

1. A neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation, characterized by: The system is applied to a deep neural network privacy inference system, including a server and a client. The server holds a training dataset and is used to generate a pre-trained model based on the dataset training. The client holds a number of privacy inference service request data and is used to send the request data to the server for privacy inference. The privacy inference method includes: S1: Perform parameter transformation and sparsification on the pre-trained deep learning model to obtain a neural network prediction model after the linear convolution layer is sparsified; S2: Retrain the model output by S1. In each training iteration, the accuracy of the neural network prediction model after the linear convolution layer is sparsified is evaluated. If the model accuracy loss is less than a preset threshold, the linear convolution layer sparsification is completed, and a retrained Winograd sparsified deep learning model is obtained. S3: Perform activation function approximation and replacement processing on the model output by S2 to obtain a neural network prediction model after nonlinear activation function approximation; S4: Retrain the model output by S3. After each training iteration, the model accuracy is tested. If the model accuracy loss is within an acceptable range, the nonlinear activation function is approximated to obtain the final optimized model. S5: Perform privacy reasoning based on the model output by S4; Among them, S1 includes: S1.1: Input the pre-trained deep learning model into the convolutional layer sparsification module, perform convolution kernel importance evaluation through the Winograd algorithm, and remove redundant convolution kernels from the model based on the importance evaluation results; S1.2: Perform weight transposition and remove redundant weights in vector units to obtain the neural network prediction model after linear convolution layer sparsification. S3 includes: Input the Winograd sparse deep learning model retrained in step S2 into the activation function parameterization approximation module, and generate a corresponding mask tensor based on the sensitivity of the learnable parameters assigned to each activation function, where the positions where the mask tensor value is 1 correspond to important activation functions, and the positions where the tensor value is 0 correspond to redundant activation functions; The generated mask tensor is further initialized with polynomials and iteratively fitted to obtain a neural network prediction model after approximation of the nonlinear activation function.

2. The neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation as claimed in claim 1, characterized in that: S1.1 includes: S1.1.1: Multiply the convolution kernel of the input pre-trained deep learning model with the sparse transformation tensor of the Winograd convolution algorithm and calculate the Winograd convolution parameters using the following formula: in, is the convolution kernel of the pre-trained deep learning model, is the sparse transformation tensor of the winograd convolution algorithm, is the winograd convolution parameter; S1.1.2: Calculate the convolution kernel importance evaluation score based on the convolution kernel of the pre-trained deep learning model and the sparse transformation tensor of the Winograd convolution algorithm: in, Evaluate the importance score of the convolution kernel; S1.1.3: Sort the calculated convolution kernel importance evaluation scores in descending order, and place the one with the smallest importance evaluation score first. The convolution kernels are regarded as redundant convolution kernels and deleted. A value between 0 and 100.

3. The neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation as claimed in claim 2, characterized in that: S1.2 includes: S1.2.1: For an output channel number of , the number of input channels is , the convolution kernel size is No. Convolutional layer , the winograd convolution parameters Perform a transpose transformation, where is the weight index subscript, , , : S1.2.2: According to the transposed winograd convolution parameters Positional relationship, the winograd convolution parameters Grouping and packaging, specifically: for each output channel The weights across all input channels are packed as a vector to obtain a set of weight vectors ,in is the output channel index and , is the weight index subscript of each 2D convolution kernel, and ; S1.2.3: The obtained weight vector set Each weight vector in is input into the following formula to calculate the weight vector importance evaluation score: in, Evaluate the importance score of the weight vector; S1.2.4: Sort the calculated weight vector importance evaluation scores in descending order, and select the one with the smallest importance evaluation score. The weight vector is deleted. A value between 0 and 100.

4. The neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation as claimed in claim 1, characterized in that S3 include: S3.1: Parameterize each activation function unit according to the following formula: in is the neural network activation function, is the activation function input, is the parameterized activation function, Additional parameters for activation functions; S3.2: Calculate the activation function importance evaluation score based on the activation function additional parameters and parameter gradients: in, is the parameter gradient, Evaluation score for the importance of the activation function; S3.3: Initialize the full feature map with the same dimension as the activation function input feature map Mask tensor, calculate the activation function importance evaluation score and sort it in descending order, according to the following formula Update, where is the indicator function, is the threshold parameter: in, is the mask tensor, is the indicator function, is the threshold parameter, is the updated mask tensor; S3.4: The server initializes the activation function unit of the deep neural network based on the updated mask tensor using the following formula to obtain a polynomial activation function version of the deep learning model, which serves as the neural network prediction model after nonlinear activation function approximation: in is a polynomial activation function, are the polynomial coefficients. The values ​​of the polynomial coefficients after initialization are .

5. The neural network privacy inference method based on Winograd sparse convolution and differentiable activation function approximation as claimed in claim 1, characterized in that S5 include: The server receives the privacy reasoning service request data from the client. Based on the privacy interactive computing protocol, the server uses the model output by S4 to perform privacy reasoning and finally obtains the model prediction output and returns it to the client.

6. A neural network privacy inference device based on Winograd sparse convolution and differentiable activation function approximation, characterized in that: A privacy-preserving inference system for deep neural networks includes a server and a client. The server holds a training dataset and generates a pre-trained model based on the dataset. The client holds a number of privacy-preserving inference service request data and sends the request data to the server for privacy-preserving inference. The privacy-preserving inference device includes: The convolution layer sparsification module is used to perform parameter transformation and sparsification on the pre-trained deep learning model to obtain a neural network prediction model after linear convolution layer sparsification. The first retraining module is used to retrain the model output by the convolutional layer sparsification module. The accuracy of the neural network prediction model after linear convolutional layer sparsification is evaluated in each training iteration. If the model accuracy loss is less than a preset threshold, the linear convolutional layer sparsification is completed to obtain a retrained Winograd sparsified deep learning model. An activation function parameterization approximation module is used to perform activation function approximation and replacement processing on the model output by the first retraining module to obtain a neural network prediction model after nonlinear activation function approximation; The second retraining module is used to retrain the model output by the activation function parameterization approximation module. The model accuracy is tested after each training iteration. If the model accuracy loss is within an acceptable range, the nonlinear activation function approximation is completed to obtain the final optimized model. The privacy reasoning module is used to perform privacy reasoning based on the model output by the second training module; Among them, the convolutional layer sparsification module is specifically used for: The pre-trained deep learning model is input into the convolutional layer sparsification module, and the importance of the convolution kernel is evaluated through the Winograd algorithm. The redundant convolution kernels are removed from the model based on the importance evaluation results. Perform weight transposition and remove redundant weights in vector units to obtain the neural network prediction model after linear convolution layer sparsification; The activation function parameterized approximation module is specifically used to perform the following steps: The Winograd sparse deep learning model retrained in the first retraining module is input into the activation function parameterization approximation module. According to the sensitivity of the learnable parameters assigned to each activation function, a corresponding mask tensor is generated, where the positions where the mask tensor value is 1 correspond to important activation functions, and the positions where the tensor value is 0 correspond to redundant activation functions; The generated mask tensor is further initialized with polynomials and iteratively fitted to obtain a neural network prediction model after approximation of the nonlinear activation function.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the program implements the neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation as described in any one of claims 1 to 5.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the neural network privacy reasoning method based on winograd sparse convolution and differentiable activation function approximation as described in any one of claims 1 to 5.

Images