Baler learning model using trimming and replacement

Abstract

An exemplary system includes a processor to trim a machine learning model based on importance of neurons or weights. The processor is further configured to permutate and pack remaining neurons or weights of the trimmed machine learning model to reduce the amount of ciphertext computation under selected constraints.

Description

Background Art

[0001] The present technology relates to machine learning models. More specifically, the technology relates to the execution of machine learning models under homomorphic encryption.

[0002] Homomorphic encryption (HE) allows operations to be performed on encrypted data. For example, in a client-server scenario where a client wants a server to perform a function f(x), this cryptosystem can be used. The client can provide x, and the function f can be obtained from different sources. HE enables the server to homomorphically compute the function f(x) without knowing the specific value of the variable x. The client can then use the private key to decrypt the result encrypted using the corresponding public key. In some schemes, multiple clients can provide multiple keys. For example, in a multi-key fully homomorphic encryption (FHE) scheme, each client can have its own private key and provide the associated public key to the server for encrypting the result.

[0003] HE operations can be performed using a single instruction multiple data (SIMD) paradigm, in which a message is divided into an array of values ​​called slots. A single HE operation is applied to all these slots at once. Specifically, a single ciphertext encrypts a fixed-size vector, and the homomorphic operation on the ciphertext is performed on the elements of the plaintext vector by slot. For example, in the CKKS HE scheme, up to thousands of encrypted values ​​are stored in a single encrypted message and processed simultaneously. In order to take advantage of SIMD features, more than one input element can be packed and encrypted in each ciphertext. Therefore, the packing method can significantly affect latency, throughput, communication costs, and memory requirements. Therefore, methods for packing or grouping these values ​​into encrypted messages can be used to improve performance. For example, the original way to pack a plaintext matrix can be to pack it in row-major order (in tandem) until all slots of a given ciphertext are "full", then create a new ciphertext and repeat. HELayers is an example software development kit (SDK) that automates the packing process for data scientists. Specifically, HELLayer uses a special packing technique called a tile tensor. A slice tensor is a data structure that packs tensors in fixed-size blocks called tiles. For example, a tensor can be a vector or a matrix. With a fixed size, the slice tensor data structure naturally matches HE because each slice can be encrypted into a single ciphertext, where different elements of each slice are mapped to different slots in its ciphertext. In addition, tensors can also be used to implement different layers of a neural network. For example, one solution uses a 5-dimensional tensor represented as C, X, Y, F, B, where C is the channel dimension that encodes the channel of the input, X, Y is the width and height dimensions of the image; F is the filter dimension that encodes the different filters of each layer, and B is the batch dimension that encodes different images in order to classify them. In addition, the same tensor can be covered by slices of different shapes of the same size. For example, a matrix can be originally covered by a column vector or a row vector, but the matrix can also be covered by a two-dimensional slice as long as the number of elements in the slice matches the number of slots in the ciphertext. In addition, slice tensors allow other manipulations, such as copying elements along one or more dimensions. Some frameworks allow easy switching between one patch shape to another, and also allow easy setting of the amount of replication along each dimension. In the following, we use patch shape to include this replication amount along each dimension. Different patch shapes can lead to different performance. For example, one patch shape may require more memory but be optimal in run time, while another shape may be optimal in memory but takes more time to run. In order to find the best shape that their system supports for a given goal, some methods use an optimizer that scans the shape configuration space and reports the best detected shape.In the context of pruning, packing the neurons and weights of a neural network into slices poses a problem: pruning can only be done at the resolution of the entire slice (i.e., ciphertext), not individual neurons or weights.

[0004] FHE operations can be significantly more expensive than their plaintext counterparts. For example, the computational intensity of FHE operations can be three to five orders of magnitude higher than that of operations performed on plaintext. One optimization used in the plaintext neural network domain is the use of pruning through operation elimination. Pruning improves latency by reducing the number of operations that must be performed, and also suppresses overfitting, and thus improves the accuracy of the deployed network. Pruning the network introduces zeros in weights and / or activations so that calculations involving these values ​​can be skipped. Thus, pruning reduces latency and energy for reasoning execution. For example, a simple weight pruning scheme removes all weights with values ​​less than a certain threshold. As a result, this reduces the number of operations that need to be formed during reasoning. Although pruning complex models generally leads to improved test accuracy by reducing variance, oversimplification can lead to underfitting and therefore a decrease in accuracy. Some pruning techniques can allow most weights to be removed with a small accuracy degradation. In addition, retraining the network with pruning in place can mitigate accuracy loss because training reduces the error output at each neuron.

[0005] However, one challenge with pruning in the context of HE-enabled inference is that the latency or energy savings due to the reduction in the number of operations is not necessarily proportional to the pruning, nor is it necessarily proportional to the degree of pruning. For example, latency or energy savings may not necessarily scale with the number of weights removed. Instead, the actual reduction in operations may also depend on the packing method. This may be because the zeros introduced during pruning may be packed into the same ciphertext message with other non-zero values, in which case the entire ciphertext must be preserved as is and the number of operations performed on this ciphertext remains unchanged. Therefore, all packed values ​​in the ciphertext message may have to be zero in order to prune the entire message and obtain latency and energy benefits. One solution is to prune in groups that match the shape of the slices encoded into the ciphertext message. However, this pruning method may result in the removal of important weights, which may therefore cause a significant decrease in the accuracy of inference. In addition, these pruning methods may not guarantee that optimality constraints involving latency, energy, and accuracy are met. Summary of the invention

[0006] According to embodiments described herein, a system may include a processor for pruning a machine learning model based on the importance of neurons or weights. The processor may further permute and pack the remaining neurons or weights of the pruned machine learning model to reduce the amount of ciphertext computation under selected constraints. Thus, the processor may implement pruning-aware packing of the machine learning model, which improves the performance of inference. Preferably, the processor prunes and packs sequentially. In this embodiment, pruning can better improve the efficiency of packing. Optionally, the importance is based on the criticality of the neurons. In this embodiment, neurons that are not important to the accuracy of the model at inference can be pruned to improve efficiency. Optionally, the importance is based on the value of the weight. In this embodiment, weights that are not important to the accuracy of the model can be marked and can therefore be ignored during inference. Optionally, the selected constraints include inference accuracy constraints. In this embodiment, a specific accuracy can be ensured at inference. Optionally, the selected constraints include memory constraints. In this embodiment, a specific memory usage can be ensured at inference. Optionally, the selected constraints include delay constraints. In this embodiment, a specific delay can be ensured during inference. Preferably, pruning the machine learning model includes eliminating operations from the machine learning model. In this embodiment, the efficiency of the machine learning model at the time of reasoning can be improved. Preferably, the ciphertext computation includes performing homomorphic encrypted reasoning of the pruned, permuted, and packed machine learning model. In this embodiment, the homomorphic encrypted reasoning can have improved accuracy and performance.

[0007] According to another embodiment described herein, a method may include pruning a machine learning model based on the importance of neurons or weights via a processor. The method may also include permuting and packing the remaining neurons or weights of the pruned machine learning model via a processor to reduce the amount of ciphertext calculations under selected constraints. Therefore, the method can achieve pruning-aware packing of the machine learning model, which improves the performance of reasoning. Optionally, the method may also include performing homomorphic encrypted reasoning using the pruned, permuted and packed machine learning model. In this embodiment, homomorphic encrypted reasoning can have improved accuracy and performance. Optionally, pruning the machine learning model includes pruning the weights of the machine learning model by setting weights having values ​​not exceeding a threshold to zero. In this embodiment, weights can be marked and ignored during reasoning. Optionally, pruning the machine learning model includes pruning neurons of the machine learning model. In this embodiment, neurons can be removed and not used during training and reasoning. Optionally, permuting the machine learning model includes using balanced clustering. In this embodiment, the maximum number of zero slices can be found more efficiently. Optionally, permuting the machine learning model includes alternating between permuting rows and columns of a weight matrix corresponding to weights between layers of the machine learning model until convergence is detected. In this embodiment, a maximum number of zero slices can be found. Optionally, the method includes extending the pruned and permuted machine learning model to not prune zero values ​​within a pruned packed shape of partial zero values. In this embodiment, accuracy lost during pruning can be regained. Optionally, the method includes: simulating the pruned and packed machine learning model to obtain latency scores and memory scores associated with multiple packing shapes and pruning thresholds, wherein the pruned, permuted and packed machine learning model includes pruning thresholds and packing shapes that minimize an objective function based on the selected constraints. In this embodiment, any selected constraints can be used to ensure that constraints are met during inference.

[0008] According to another embodiment described herein, a computer program product for pruning and packing machine learning models may include a computer-readable storage medium having program code embodied therewith. The program code may be executed by a processor to cause the processor to prune the machine learning model based on the importance of neurons or weights. The program code may also cause the processor to permute and pack the remaining neurons or weights of the pruned machine learning model to reduce the amount of ciphertext calculations under selected constraints. Therefore, the program code may implement pruning-aware packing for machine learning models, which improves the performance of reasoning. Optionally, the program code may also cause the processor to set weights with values ​​not exceeding a threshold to zero. In this embodiment, zero weights may be marked and not considered during training and reasoning. Optionally, the program code may also cause the processor to use a heuristic to permute the machine learning model. In this embodiment, the zero pieces to be pruned may be increased more efficiently. Optionally, the program code may also cause the processor to also permute the machine learning model using balanced clustering. In this embodiment, zero pieces may be found more efficiently. Optionally, the program code may also cause the processor to alternate between permuting rows and columns of a weight matrix corresponding to weights between layers of the machine learning model until convergence is detected. In this embodiment, the maximum number of zero pieces may be found. Optionally, the program code may also cause the processor to retrain the pruned machine learning model and execute the pruned machine learning model to obtain an accuracy score of the pruned machine learning model associated with a specific pruning threshold. In this embodiment, the accuracy score may be used to select the best combination of pruning, permutation, and packing. Optionally, the program code may also cause the processor to simulate the pruned and packed machine learning model to obtain a delay score and a memory score associated with a plurality of packing shapes and pruning thresholds, wherein the pruned, permuted, and packed machine learning model includes a pruning threshold and a packing shape that minimizes the objective function based on the selected constraints. In this embodiment, the objective function calculator may use the delay score and the memory score to select the best combination of pruning, permutation, and packing. Optionally, the program code may also cause the processor to perform homomorphic encryption reasoning using the pruned, permuted, and packed machine learning model. In this embodiment, homomorphic encryption reasoning may be performed more efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of an example system for pruning, permuting, and packing machine learning models;

[0010] Figure 2 is a block diagram of an example system for generating encrypted results based on encrypted data using a machine learning model that uses pruning and permutation packing;

[0011] Figure 3Ais a process flow diagram of an example process that may select a combination of network packing, pruning, and permutation based on an objective function;

[0012] Figure 3B is a process flow diagram of an example process that may select a combination of network packing, pruning, permutation, and expansion based on an objective function;

[0013] Figure 4 is a diagram showing an example process of weight matrix pruning, permutation, and packing;

[0014] Figure 5 is a diagram illustrating an example process of permutation using a balanced variant of k-means;

[0015] Figure 6 is a diagram illustrating an example process of weight permutation for a multi-layer neural network;

[0016] Figure 7 is a diagram illustrating an example process of neuron pruning using permutation;

[0017] Figure 8 is a diagram illustrating an example process of weight pruning using permutation;

[0018] Fig. 9 is a process flow diagram of an example method that can pack, prune, and permute machine learning models under selected constraints;

[0019] Fig.10 is a process flow diagram of an example method that can select a combination of network packing, pruning, and permutation based on an objective function;

[0020] Fig.11 is a process flow diagram of an example method that can generate encrypted results based on encrypted data using a machine learning model that packs using pruning and permutation;

[0021] Fig.12 is a block diagram of an example computing device that can pack, prune, and permute machine learning models under selected constraints;

[0022] Fig.13 is a diagram of an example cloud computing environment according to embodiments described herein;

[0023] Fig.14 is a diagram of example abstract model layers according to embodiments described herein;

[0024] Fig.15 is an example tangible non-transitory computer-readable medium that can package, prune, and permute machine learning models under selected constraints;

[0025] Fig.16is a diagram illustrating an example process of weight pruning and permutation with example extensions; and

[0026] Fig.17 is a diagram illustrating an example set of different combinations of pruning, permuting, expanding, and packing according to embodiments described herein. DETAILED DESCRIPTION

[0027] According to an embodiment of the present disclosure, a system includes a processor for pruning a machine learning model based on the importance and weights of neurons. The processor will further permute and pack the remaining neurons or weights of the pruned machine learning model to reduce the amount of ciphertext calculations under the selected constraints. Thus, an embodiment of the present disclosure provides a pruning-aware packing method for machine learning model inference under homomorphic encryption (HE), which obtains the maximum performance advantage from the pruning step without causing minimal degradation in accuracy. In particular, when experiments were conducted on autoencoder neural networks, significant improvements in efficiency were noted, especially for larger tiles. Specifically, for tile sizes of 4×4, 8×8, and 16×16, the example iterative k-means permutation algorithm increased the number of tiles with only zero elements from 40% to 50%, 20% to 40%, and 8% to 40%.

[0028] See now Figure 1 , a block diagram illustrates an example system for pruning, permuting, and packing machine learning models. The example system is generally referred to by the reference numeral 100. Figure 1 A computing device 102 is included. For example, the computing device 102 can be a server. In some examples, the computing device 102 can be a node of a cloud computing service. The computing device 102 includes a network pruner 104, a network permutator 106, a network packager 108, and an objective function evaluator 110. The computing device 102 is shown as receiving a machine learning model 112. For example, the machine learning model 112 can be any suitable machine learning model trained to perform HE operations. In different examples, the machine learning model 112 can be a neural network. For example, the machine learning model 112 can be a convolutional neural network (CNN), an autoencoder, or any other suitable machine learning model. In different examples, the machine learning model 112 can be encrypted or unencrypted. The system 100 also includes a selected constraint 114 shown as received by the computing device 102. For example, the selected constraint 114 can be any suitable constraint, such as an inference accuracy constraint, a memory constraint, a delay constraint, an amortized delay, a power constraint, an energy constraint, or any combination thereof. The system also includes a pruned, permuted, and packaged machine learning model 116 shown as output by the computing device 102.

[0029] exist Figure 1In an example of , a computing device 102 receives a machine learning model 112 and selected constraints 114, and outputs a pruned, permuted, and packed machine learning model 116 that satisfies the selected constraints 114. In different examples, the machine learning model 112 may be encrypted or unencrypted. For example, the machine learning model 112 may be encrypted after being trained on proprietary information. Thus, the weights of the machine learning model 112 may be deployed to the untrusted computing device 102 in an encrypted format. The computing device 102 may thus learn about the shape of the machine learning model 112, such as the number of layers and the number of parameters for each layer, but may not know about the values ​​of any parameters. In some examples, activation inputs from the client may also be encrypted under HE. In addition, if some operations are eliminated, the computing device 102 may also be allowed to learn which operations are eliminated. In this way, by keeping the model confidential, the underlying proprietary information can be kept confidential.

[0030] In some examples, the machine learning model 112 may be unencrypted. For example, the machine learning model 112 may have been trained on publicly available data without any restrictions and therefore may not need to be kept confidential. In other examples, the machine learning model 112 may be encrypted. For example, the machine learning model 112 may have been trained on data that is subject to accessibility restrictions.

[0031] Still see Figure 1, the network pruner 104 of the computing device 102 can prune the machine learning model 112. For example, the network pruner 104 can use any number or type of suitable pruning thresholds or parameters to prune the machine learning model 112. In different examples, the threshold can be the value of the weight having a larger L1 norm than a fixed percentage of other weights, referred to herein as L1-based pruning. In some examples, any other suitable pruning parameters may be received. For example, the pruning parameters may include whether to prune weights or neurons, and whether to use random pruning, global pruning, or local pruning. For example, random pruning can randomly prune neurons or randomly set model weights to zero. In global pruning, all layers are pruned at the same time. In local, local pruning, each layer is pruned according to other parameters. When random pruning is used, these parameters may not have an impact. However, when considering, for example, L1-based pruning, the parameters may have a strong effect. For example, if the processor prunes 50% of the network, only the initial layer can be pruned. In various examples, any of six different pruning configurations from a combination of these parameters {W, L}×{R, L1}×{W, N}, where G / R / {W, N} is the same as L / R / {W, N}, and W refers to the pruning weights, N refers to the pruned neurons, G refers to the global pruning method, L refers to the local pruning method, and L1 refers to L1-based pruning. In some examples, the network pruner 104 may use a packing-based pruning configuration, which is also referred to herein as pruning packing. For example, the network pruner 104 may first select a packing shape size. In the example of a slice tensor, the packing shape size may be the slice size. In different examples, the slice size may be 2x2, 4x8, or 8x8. The network pruner 104 may then split each matrix into slices. For each slice, the network pruner 104 may calculate the minimum, maximum, or average of its values, and prune the slice with the lowest result.

[0032] The network permutator 106 may permute the machine learning model 112 after the machine learning model is pruned. For example, the network permutator 106 may permute the machine learning model 112 using any suitable heuristic, such as a balanced clustering heuristic. In some examples, the network permutator 106 may use a k-means clustering heuristic, such as Figure 5 In some examples, such as for Figure 6 Described in more detail, the network permutator 106 may permute rows and columns of a weight matrix corresponding to weights between layers of a machine learning model in a permuted manner.

[0033] The network packager 108 may package the pruned and permuted machine learning models using any suitable package shape or size. For example, the network packager 108 may package the pruned and permuted machine learning models using a variety of different package shapes and sizes.

[0034] In some examples, the network pruner 104, the network permutator 106, and the network packer 108 may generate a plurality of pruned, permuted, and packed machine learning models. In various examples, the objective function evaluator 110 may evaluate each combination of different pruning, permutation, and packing based on the objective function and one or more selected constraints 114. An example algorithm for calculating an example objective function is described with respect to FIG.

[0035] In various examples, the resulting pruned, permuted, and packed machine learning model 116 can be output and used for inference in a HE environment. Figure 2 An exemplary pruned, permuted, and packed machine learning model 116 used in this manner is described.

[0036] As a specific technical example, the HeLayers packaging solution can be used with a CKKS SEAL implementation targeting 128-bit security. For training, a cluster of server-class machines can be equipped with a GPU. Training can use PyTorch version 1.11.0 accelerated with CUDA version 11.6. The network architecture used can be an autoencoder network architecture in which each fully connected (FC) layer is followed by a square activation layer. For example, square activations can be used instead of rectified linear unit (ReLU) activations to support non-interactive solutions that require HE-friendly networks. In different examples, more sophisticated activations (such as higher degree activations or trainable activations) can be used additionally or alternatively to achieve better results. As an example, an autoencoder network can include a FC with 32 neurons or 64 neurons. In some examples, an autoencoder network may include multiple FC layers, such as three FC layers with 64, 32, and 64 neuron sizes. An autoencoder network can be trained on the MNIST dataset (first published in 1998), which has 60,000 images of 28x28x1=768 pixels. Thus, the input size of the autoencoder and the output size of the decoder may be 768. In some examples, the decoder may be fused to the encoder as an additional FC layer of related size and trained together. The number of training and retraining time periods may be set to {20, 10}, {30.20}, or any other suitable value. Batches of ten samples may be used, and the learning rate of the Adam optimizer may be set to 1e-3. In different examples, the loss function used may be any suitable loss function, such as the mean square error (MSE) between the input image and the reconstructed image. In this example, HeLayer uses data structures called Ctile tensors and Ptile tensors to hold slice tensors of encrypted and unencrypted data, respectively. These have an API called encoding to encode (pack) the data before encrypting it. Therefore, in some examples, HeLayer may be adapted to automatically identify zero slices by modifying different encoding functions to test whether all its elements are zero for each slice. In the case where the slice contains only zeros, the processor may not allocate it, but instead include a new flag to indicate that this is a zero slice. In a different example, when considering binary addition and multiplication operations that receive two inputs, if only one of the inputs has a set flag, the addition function can be modified to return another object, and the multiplication function can be modified to return a new empty slice with this flag set. In the case where both inputs are zero, the returned element can be an empty slice.

[0037] It should be understood that Figure 1 The block diagram is not intended to indicate that system 100 will include Figure 1 Instead, system 100 may include Figure 1Fewer or additional components not shown in the example (e.g., additional computing devices or additional machine learning models, pruned, permuted, and packed machine learning models, or additional processing (such as expansion), etc.). For example, the system 100 may further include a model expander to reduce neurons or weights that include zero values. For example, the model expander may perform an operation that reverses the pruning operation. Specifically, the model expander may search for slices that do not only hold zero values ​​and de-prune the zero elements inside them. The unpruned weight elements may then be trained to improve model accuracy. In this way, the model expander may regain some of the accuracy of the model that was lost due to the initial pruning. For example, if a slice is not reduced because it has non-zero elements, the system in reasoning cannot ignore the slice. Therefore, the model expander may instead make full use of its elements to increase the performance of the model at reasoning time.

[0038] Figure 2 is a block diagram illustrating an example system for generating an encryption result based on encrypted data using a machine learning model that is packed using pruning and permutation. The example system 200 includes Figure 2 For example, the system 200 includes a pruned, permuted, and packed machine learning model 116. The pruned, permuted, and packed machine learning model 116 of the system 200 is shown as receiving encrypted data 202 and outputting an encrypted result 204. For example, the encrypted data 202 can be any information to be classified, such as an image. In different examples, the encrypted result 204 can include a classification of the input encrypted data 202. In some examples, the encrypted result 204 can also include a confidence score for the classification.

[0039] As previously mentioned, one practical application of HE is for encrypted inference on neural networks running on the cloud. For example, system 200 can be used to diagnose COVID-19 by classifying X-ray images of patients in a hospital environment, for which encrypted X-ray images are securely transmitted to the cloud. In this example, computing device 102 can be a server running a machine learning model that is trained on a different system using proprietary data that is not available to the public, and therefore the parameters of the network can be encrypted to hide them from the server. These parameters can include weights or biases. In this example, the client can obtain an encrypted classification result 204 from the server without the server learning anything about the image or network parameter values. The client can then decrypt the encrypted result 204 using a key. For example, the key can correspond to the key used to encrypt the encrypted X-ray image.

[0040] It should be understood that Figure 2 The block diagram is not intended to indicate that system 200 includes Figure 2 Instead, system 200 may include Figure 2 , or fewer or additional components not shown in the example (e.g., additional data or additional results, etc.). For example, the pruned, permuted, and packed machine learning models may alternatively be pruned, permuted, expanded, and packed machine learning models, or as described below. Fig.17 The product of any combination of the operations described in .

[0041] FIG3 is a process flow diagram of an example process that can select a combination of network packing, pruning, and permutation based on an objective function. Process 300 can be implemented using any suitable computing device, such as Fig.12 The computing device 1200 or Figure 1 For example, the method described below may be performed by Figure 1 , 12 and 15 are implemented by the computing device 102, the processor 1202 or the processor 1502.

[0042] Figure 3A The process of pruning, permuting, and packing a machine learning model for inference on a network using a two-dimensional slice tensor and a batch size of 1 is shown. In a different example, blocks 302-336 can be part of a pre-deployment process and performed on a system that has access to the plaintext network parameters, since the pruning algorithm typically takes the value of each parameter as input. In addition, Figure 3A Consider an example of pruning individual weights based on a threshold. Figure 3A The general goal in the example of can be to find a pruning threshold (PRUNE_THRES) and slice tensor shape that optimizes the machine learning model for a given objective function. In different examples, the objective function can be based on any combination of accuracy, latency, memory requirements, energy consumption, and any other suitable selected constraints.

[0043] At block 302, process 300 begins. In various examples, a processor may receive a trained model, a set of pruning thresholds, and a set of different patch shapes. For example, the pruning threshold may be as follows, among other suitable values: Figure 4 and Figure 5 In the example of , the value is "1". In a different example, the pruning threshold may be a pruning threshold based on L1.

[0044] At decision diamond 304, the processor determines whether each pruning threshold PRUNE_THRES in the received pruning threshold set THRES_ALL has been processed. If all pruning thresholds THRES_ALL have been processed, the process can continue at decision diamond 318. If all pruning thresholds THRES_ALL have not been processed, the process can continue at box 306.

[0045] At block 306, the processor sets the received trained model as the model to be processed. In some examples, the processor may process multiple models and may therefore retrieve one of a set of models to be pruned, permuted, and packed.

[0046] At decision diamond 308, the processor determines whether all layers in the model have been processed. If all layers in the model have been processed, the process can continue at block 312. If all layers in the model have not been processed, the process can continue at block 310.

[0047] At box 310, the processor prunes the selected layer of the model based on the selected pruning threshold PRUN_THRES. For example, the processor can prune the machine learning model based on the threshold. In different examples, the processor can use weight pruning, neuron pruning, or a combination thereof.

[0048] At block 312, the processor retrains the pruned model. For example, the processor may retrain the pruned network to recover some of the resulting accuracy loss from pruning.

[0049] At block 314, the processor executes the trained pruned model to obtain an accuracy score for the trained pruned model. For example, an updated accuracy score may be obtained by running inference on the test set on the pruned and retrained machine learning model.

[0050] At block 316, the processor appends the combination of the pruning threshold PRUNE_THRES, the model, and the associated accuracy score of the model pruned using the pruning threshold PRUNE_THRES to the set of model records MODEL_RECS. For example, the set of model records MODEL_RECS may be stored in a file.

[0051] At decision diamond 318, the processor determines whether each combination of pruning threshold PRUNE_THRES, model, and associated accuracy score of the model pruned using pruning threshold PRUNE_THRES has been processed using permutations. If not, the process can continue at decision diamond 320. If yes, the process can continue at decision diamond 330.

[0052] At decision diamond 320, the processor determines whether all different tile sizes and tile shapes received in TILE_SHAPES have been processed for a particular combination of pruning threshold PRUNE_THRES, model, and associated accuracy score for the model pruned using the pruning threshold PRUNE_THRES. If so, the process can continue with additional combinations at decision diamond 318. If not, the process can continue with additional permutations at box 322. For example, the set TILE_SHAPES can be an independent set of integer tuples. As an example, for a set of two 2D tiles of length=2, width=2 and length=3, width=3, the values ​​of TILE_SHAPES can be {(2,2),(3,3)}, respectively.

[0053] At block 322, the processor permutes the model using a combination of the patch shapes T1 and T2. For example, the processor may permutate the model using any suitable heuristic, such as an iterative clustering algorithm. In some examples, the heuristic may be a balanced clustering heuristic. In some examples, the processor may use a balanced k-means clustering technique, such as in Figure 5 Described in .

[0054] At block 324, the processor packs the permuted model. For example, the processor may pack the weight and activation tensors into T1xT2 slices. In a different example, the processor may also discard zero slices.

[0055] At block 326, the processor simulates the packed permutation model to generate associated latency and memory values ​​for the packed permutation model. For example, these latency and memory scores may be calculated in response to receiving selected latency and memory constraints from the client. Thus, the processor may simulate the network to obtain estimates of latency and memory requirements for the packed permutation model.

[0056] At block 328, the processor appends to the log file a combination of the pruning threshold PRUNE_THRES, the model, and the associated accuracy score of the model pruned using the pruning threshold PRUNE_THRES, and the latency and memory scores of the model when packed and permuted with the slice shapes T1 and T2. The log file may thus include rows corresponding to all combinations of different pruning thresholds, permutations, and slice tensor shapes.

[0057] At decision diamond 330, the processor determines whether each record in the record file has been processed to generate the objective function score. If not, the process can continue at block 332 to process additional records. If it has been processed, the process can continue at block 334.

[0058] At block 332, the processor calculates an objective function score for each record in the record file. For example, the objective function score may depend on the objective function and different selected optimization constraints. In the example of FIG. 3, these optimization constraints include accuracy, latency, and memory constraints.

[0059] At block 334, the processor selects the record row from the record file associated with the lowest objective function score calculated at block 332. For example, the best record row may be chosen depending on the objective function and the optimization constraints.

[0060] At block 336 , the process ends. In some examples, the processor may output the model pruned, permuted, and packed using the selected record rows of block 334 .

[0061] Figure 3A The process flow diagram of 300A is not intended to indicate that the operations of process 300A will be performed in any particular order, or that all operations of process 300A will be included in every case. For example, although shown using slice tensor grouping for illustration, any other suitable grouping may alternatively be used. Additionally, Figure 3A Process 300A considers pruning individual weights based on thresholds. However, other pruning methods may be used, such as pruning groups of weights that will be packaged together into the same encrypted message, as well as techniques in the prior art, such as pruning based on activation criticality, dynamic pruning and splicing of weights, and other suitable pruning techniques. In addition, process 300A may include any suitable number of additional operations. In some examples, although process 300A uses an exhaustive search strategy to find the optimal point, a local search strategy may be used instead. For example, an exhaustive search to find a permutation of a matrix with the maximum number of zero slices may cost O(M!N!), which may be prohibitive even for moderately sized weights. Therefore, in some embodiments, process 300A may alternatively permute rows and columns based on heuristics to make the problem tractable. For example, method 300A may use Figure 5 The example k-means heuristic described in . In some examples, such as Figure 3B As described in , method 300A may further include expanding the slice of partial zero values ​​to further increase accuracy and efficiency.

[0062] Figure 3B is a process flow diagram of an example process that can select a combination of network packing, pruning, permutation, and expansion based on an objective function. Process 300B can be implemented using any suitable computing device, such as Fig.12 The computing device 1200 or Figure 1 For example, the method described below may be performed by Figure 1 , 12 and 15 are implemented by the computing device 102, the processor 1202 or the processor 1502.

[0063] Figure 3B The course 300B includes Figure 3A . In addition, at decision diamond 338, the processor determines whether all layers in the model have been further processed. If not, the process continues at box 340. If yes, then process 300B can continue at box 342.

[0064] At block 340, the processor performs an expansion operation. For example, the expansion operation may unprune any partial slices in a layer of the machine learning model.

[0065] At block 342, the processor retrains the model. For example, the machine learning model can be retrained with unpruned values ​​to improve the accuracy of the resulting retrained model.

[0066] At block 344, the processor executes the machine learning model on the test set of data to generate an updated accuracy score for the retrained model. For example, the updated accuracy score may be higher due to the additional weights available during training. In a different example, when calculating the objective function at block 332, the updated accuracy score may replace the accuracy score in the log file and be used in place of the previous accuracy score.

[0067] Figure 3B The process flow diagram of FIG. 300B is not intended to indicate that the operations of process 300B will be performed in any particular order, or that all operations of process 300B will be included in every case. For example, although Figure 3B The example shows Fig.17 The example P3E, but in some examples, Figure 3B This may include Fig.17 Pruning-based packaging in P4E or Fig.17 Pruning-based semi-packaging in P5E.

[0068] Figure 4 is a diagram illustrating an example process of weight matrix pruning, permutation, and packing. Example process 400 may be performed by any suitable processor, such as Figure 1 , 12 and a processor of the computing device 102 of 15 , the processor 1202 , or the processor 1502 .

[0069] Figure 4 The process 400 includes an initial weight matrix 402. As an example, the numbered rows of the weight matrix 402 correspond to a first layer of the neural network and the numbered columns of the weight matrix correspond to a second layer of the neural network. Figure 4 A simple example of a 4x8 weight matrix and a packed shape of 2x2 slices is shown. Figure 4As shown, the values ​​of the weight matrix range from 0.1 to 1.9 and correspond to the weights between neurons of the two layers. Process 400 includes a pruned weight matrix 404, wherein values ​​less than 1.0 have been pruned from weight matrix 402.

[0070] Process 400 further illustrates a first pruned and packed weight matrix 406, wherein one of the eight slices is a zero slice containing all zero values. Figure 4 , the zero-value slice tensor that can be discarded is shown with a bold solid outline. Other non-zero slices are indicated using a dashed outline. Process 400 further includes a pruned and permuted weight matrix 408, on which the best permutation has been applied. For example, any number of different permutations may have been performed, and the best permutation is selected and applied based on the maximization of zero slices. In some examples, the best permutation can be selected using an objective function as described herein. Process 400 also includes a pruned, permuted, and packed weight matrix 410, where four of the eight slices are zero slices, indicated by a bold outline. Pruning 412 of weight matrix 402 is indicated by a first arrow. Packing 414 of pruned weight matrix 404 is indicated by a second arrow. Permutation 416 of pruned weight matrix 404 is indicated by a third arrow. Packing 418 of pruned and permuted weight matrix 408 is indicated by a fourth arrow. Figure 4 Also shown are a neural network 420 corresponding to weight matrix 402, a pruned neural network 422 having pruned weights corresponding to zeros of pruned weight matrix 402, and a pruned and permuted neural network 424 having an ordering of rows and columns corresponding to pruned and permuted weight matrix 408.

[0071] exist Figure 4 In the example of , the example pruning threshold applied has a value of 1, and therefore weights with values ​​< 1.0 have been zeroed in the pruned weight matrix 404. Figure 4 As shown in box 406 of , if the weight tensor is packed as is at the stage shown in box 404, only one of the eight slice tensors contains all zeros and can therefore be discarded by the processor. The remaining seven slice tensors in box 406 contain a mixture of non-zeros in addition to zero. Therefore, these slice tensors cannot be discarded. In order to improve the number of zero tensors that can be discarded, the processor may therefore permute the rows and columns of the tensor before packing the permuted slice tensors. For example, the processor may rearrange the rows and columns so that zero values ​​are grouped together as much as possible. In different examples, the processor may use a permutation process to perform this regrouping, thereby producing an optimal permutation 408. For example, any suitable permutation process may be used. In some examples, the permutation process used may be Figure 5 The permutation process of permuting the rows and columns of the permutation weight matrix is ​​described in .

[0072] The processor has increased the number of zero slices to a maximum of four by permuting the rows and columns according to a balanced k-means permutation algorithm, as shown in block 410. For example, the processor may have used the following Figure 5 The balanced k-means permutation described in . This increase in zero slices directly translates into a reduction in the execution time of the network when performing inference. In addition, the permutation of the rows and columns of the weight matrix 404 is equivalent to shuffling the neurons within one or more layers of the weight matrix 404, and therefore does not affect the functionality of the entire neural network.

[0073] It should be understood that Figure 4 The block diagram is not intended to indicate that process 400 includes Figure 4 Instead, process 400 may include Figure 4 , or fewer or additional components not shown in (e.g., additional layers, neurons, weights, slice shapes, sizes, or additional permutations, etc.). In different examples, slice tensors of higher dimensions may be used alternatively, such as 2x2x256 slice tensors. In some examples, a batch dimension may also be used. For example, the batch dimension may include using a subset of the original weight matrix for pruning purposes. For example, given a ciphertext that encrypts a vector of 1024 elements, box 410 may have to prune all 1024 packed elements, which may reduce accuracy. Alternatively, the processor may instead assume an inference system that performs inference on a batch of 256 samples at a time. In this case, the slice tensor will allocate 2x2 slots per sample in each ciphertext. Therefore, the processor may only need to prune 2X2 slices from the weight matrix, which may be more feasible.

[0074] Figure 5 is a diagram illustrating an example process for permutation using a balanced variant of k-means. The example process 500 may be performed by any suitable processor, such as Figure 1 , 12 and a processor of the computing device 102 of 15 , the processor 1202 , or the processor 1502 .

[0075] In various examples, the example heuristics used for permutation are based on the k-means clustering technique. More specifically, Figure 5 The example of shows the use of balanced k-means. Figure 5 The process 500 of FIG. 500 includes a first weight matrix 502. For example, the first weight matrix 502 can be a pruned weight matrix having a T1xT2 patch tensor. The set of numbers for the rows and the set of numbers for the columns are used to indicate the initial ordering of the rows and columns, respectively. At iteration 0 504, the initial weight matrix 502 includes only a patch tensor of one zero patch indicated by a bold line, where the values ​​of the patch tensor are all zero.

[0076] In various examples, the rows of the pruned weight matrix 502 can be considered as vectors, and a first iteration of k-means 506 can be applied to generate a new weight matrix 508 in which the rows are permuted to increase the number of zero slices. Specifically, the new weight matrix 508 includes two zero slices indicated by the bold lines. In addition, the new order of the rows is indicated by the bold numbers. Specifically, row 0 has been moved down two positions to be placed between row 2 and row 3.

[0077] exist Figure 5 In the example process 500 of , the new matrix 508 is then transposed 510 to generate a transposed matrix 512. At arrow 514, the processor may then apply a second iteration of k-means to the transposed matrix 512 to generate a second new matrix 516. The second new matrix 516 shows a new ordering of the original columns as indicated by the bold numbers.

[0078] At arrow 518, the processor may transpose the second new matrix 516 to generate a transposed second new matrix 520. The transposed second new matrix 520 includes four zero tiles, as indicated by the bold outline.

[0079] In various examples, process 500 is repeated until convergence. For example, convergence may be achieved when row and column permutations do not result in any additional zero slices. As an example, if the processor prunes 400 elements and the slice size is 2X2=4 elements, and the processor detects 100 zero slices, convergence may be assumed. However, alternatively, the processor may stop process 500 after a given threshold. For example, the processor may stop process 500 after 80% of the elements form zero slices. In some examples, the distance function used may be a Hamming distance. For example, a non-zero cell may be considered to have a value of "1". In various examples, the number of clusters used by the processor for k-means is equal to the number of slices along rows or columns, depending on the iteration being performed. For example, given an MxN matrix and t1xt2 slices, the number of clusters for the i-th iteration may be equal to ceil(M / t1) [if i is even] and ceil(N / t2) [if i is odd]. In this example, for an 8x16 matrix with 4x2 slices, the number of clusters would thus be 2, 8, 2, 8, ..., etc.

[0080] It should be understood that Figure 5 The block diagram is not intended to indicate that system 500 will include Figure 5 Instead, system 500 may include Figure 5Fewer or additional components not shown in (e.g., additional layers, neurons, weights, patch shapes, or additional types or permutations of iterations, etc.). In various examples, process 500 may alternatively use a higher dimensional patch tensor. For example, the process may use a 2x2x256 patch tensor. In various examples, the k-means clustering technique of process 500 may alternatively be replaced with any other suitable balanced clustering technique. For example, alternative balanced clustering techniques may include agglomerative clustering or graph segmentation techniques, such as the normalized cut (Ncut) technique first described in 1997, which measures the total dissimilarity between different groups and the total similarity within a group when posing image segmentation as a graph segmentation problem. Other clustering techniques with balanced variants that can be used include Gaussian mixture models (GMM) and density-based spatial clustering of applications with noise (DBSCAN).

[0081] Figure 6 is a diagram illustrating an example process for weight permutation of a multi-layer neural network. Example process 600 may be performed using any suitable computing device, such as Fig.12 The computing device 1200 or Figure 1 For example, process 600 may be implemented by Figure 1 , 12 and 15 , the computing device 102 , the processor 1202 , or the processor 1502 .

[0082] The example process 600 includes a first permutation 602 and a second permutation 604. As indicated by two arrows 606, the processor may repeat the permutations 602 and 604 until convergence.

[0083] In a different example, in the case of a single weight matrix for a 2-layer network, the processor can permute the rows and columns independently. However, for deeper networks, such as Figure 6 In the neural network shown in , when the rows or columns of a given layer are permuted, the weights of adjacent layers may also be affected. Figure 6 In the example of FIG. 1 , the example neural network includes five layers labeled A, B, C, D, and E. The weight sets depicted as lines between and connecting the respective layers A, B, C, D, and E are labeled W, respectively. AB ,W BC ,W CD and W DE .exist Figure 6 In the example, W BC and W DE The transpose of is marked as and

[0084] In this example, shuffling the neurons in layer B is converted into transposing the weight matrix However, this also permutes the rows of the previous weight matrix W AB Therefore, in different examples, the processor can sequentially permute the weight matrix W AB and transpose the weight matrix In the case of neuron permutations in layer D, the processor can similarly permute the weight matrix W CD and transpose the weight matrix In this way, the processor can replace the layer set in block 602.

[0085] At block 604, the processor may similarly permute the remaining set of layers along the columns. For example, the processor may use the weight matrix W AB , W CD and transpose the weight matrix and The columns of layers A, C, and E are permuted. At block 604, the shuffling of neurons in layer C is converted into a processor sequentially permuting the weight matrix W. CD and the transposed weight matrix The processor can also similarly separate and simultaneously permute the weight matrix W AB and transpose the weight matrix Columns.

[0086] At block 606, the process is repeated. For example, the processor may iterate over blocks 602 and 604 until convergence is reached. In some examples, convergence may be detected based on a permutation that does not result in any additional zero slices. In a different example, convergence may be based on a preset maximum iteration count that resolves the oscillating behavior. For example, if the algorithm oscillates between permutations with 41 zero slices and 42 zero slices, convergence may be detected after a preset number of oscillations. In some examples, the processor may detect convergence in response to determining that the zero tile count obtained in the last N iterations has not changed.

[0087] It should be understood that Figure 6 The schematic diagram is not intended to indicate that process 600 will include Figure 6 Instead, process 600 may include Figure 6 Fewer or additional components not shown (e.g., additional layers, neurons, weights, patch shapes, or additional types or permutations of iterations, etc.).

[0088] Figure 7 is a diagram illustrating an example process for neuron pruning using permutation. The example process 700 may be performed using any suitable computing device, such as Fig.12 The computing device 1200 or Figure 1 For example, process 700 may be implemented by Figure 1, 12 and 15 are implemented by the computing device 102, the processor 1202 or the processor 1502.

[0089] For a 4-layer neural network, it is shown Figure 7 Example process 700 of neuron pruning for a layer A, B, C, D having 6, 4, 8, and 4 neurons in each of the layers A, B, C, and D, respectively. The neurons of the layers A, B, C, and D can be described as vectors X A , X B , X C and X D . Figure 7 Also includes the associated weight matrix W AB , W CD And the transposed weight matrix A collection of. Figure 7 As shown, process 700 finds a rearranged network so that the packing method can discard the maximum number of encrypted messages containing only zeros, thereby improving inference latency on this pruned network without affecting the functionality and, therefore, the accuracy of the pruned network at inference time. Figure 7 In the neuron-only pruning example shown in , the processor can simply remove the last k empty columns and I empty rows of each of the pruned and permuted weight matrices.

[0090] At block 702, the original 4-layer neural network includes all of its original weights. At block 704, after pruning 706 indicated by the arrow, the valid portion of the original weights has been removed, as indicated by the gray box. For example, pruning 706 can be performed using any suitable pruning technique, such as by pruning thresholds. Figure 7 In the example of , the pruning threshold can be a neuron criticality threshold. However, as indicated by the bold box, only a total of four 2x2 packs contain all zeros and are therefore considered to be zero slices corresponding to neurons that can be discarded.

[0091] At block 708, after permutation 710, the number of zero pieces has increased to a total of 11 zero pieces that can be discarded. By discarding 11 instead of two encrypted messages, the inference latency of the resulting pruning, permutation, and packing network can be significantly improved.

[0092] It should be understood that Figure 7 The schematic diagram is not intended to indicate that process 700 will include Figure 7 Instead, process 700 may include Figure 7 Fewer or additional components not shown (e.g., additional layers, neurons, weights, patch shapes, or additional types or permutations of iterations, etc.).

[0093] Figure 8is a diagram illustrating an example process for weight pruning with permutation. The example process 800 may be performed using any suitable computing device, such as Fig.12 The computing device 1200 or Figure 1 For example, process 800 may be implemented by Figure 1 , 12 and 15 , the computing device 102 , the processor 1202 , or the processor 1502 .

[0094] Figure 8 The example process 800 for weight pruning is similarly shown for a 4-layer neural network with 6, 4, 8, and 4 neurons in each of layers A, B, C, and D, respectively. Figure 8 Also includes the associated weight matrix W AB , W CD And the transposed weight matrix A collection of. Figure 8 As shown, process 800 similarly finds a rearranged network such that the packing method can discard a maximum number of encrypted messages containing only zeros, thereby improving inference latency on this pruned network without affecting functionality and, therefore, accuracy of the pruned network at inference time.

[0095] exist Figure 8 In the example of , at block 804, weights corresponding to zero slices are discarded via pruning 806, but the neurons themselves are retained. For example, pruning entire neurons may be too aggressive for some networks. Figure 8 As shown in , the processor can alternatively be more conservative and prune only the weights. However, in the example of weight pruning, the processor cannot simply discard the last few rows and columns, as Figure 7 In contrast, Figure 8 In the weight pruning example of , the processor marks each zero slice in block 808 after permutation 810 with a tag requiring the server to skip any computations using that slice.

[0096] Fig. 9 is a process flow diagram of an example method for packing, pruning, and permuting machine learning models under selected constraints. Method 900 may be implemented using any suitable computing device, such as Fig.12 The computing device 1200 or Figure 1 For example, the method described below may be performed by Figure 1 , 12 and 15 are implemented by the computing device 102, the processor 1202 or the processor 1502.

[0097] At block 902, the processor receives a trained machine learning model and selected constraints. For example, the trained machine learning model may be encrypted or unencrypted. In some examples, the machine learning model may be a neural network, such as a convolutional neural network. In various examples, the selected constraints may include inference accuracy constraints, memory constraints, latency constraints, or any combination thereof.

[0098] At block 904, the processor prunes the trained machine learning model based on the importance of the neurons and weights. For example, the processor may set weights having values ​​not exceeding a threshold to zero. In some examples, the processor prunes the weights of the machine learning model. For example, the processor may prune weights by setting weights having values ​​not exceeding a threshold to zero and marking certain packed shapes (such as slices of weights) as zero slices to be ignored.

[0099] At box 906, the processor permutes and packs the remaining neurons and weights of the pruned machine learning model to reduce the amount of ciphertext computation under the selected constraints. In different examples, the processor may use a heuristic to permutate the machine learning model. For example, the heuristic may be a balanced clustering heuristic. In some examples, the processor may permutate the machine learning model by alternating between rows and columns of a weight matrix corresponding to weights between layers of the machine learning model until convergence is detected. In some examples, the processor may retrain the pruned machine learning model and execute the pruned machine learning model to obtain an accuracy score of the pruned machine learning model associated with a specific pruning threshold. In some examples, the processor may simulate the pruned and packed machine learning model to obtain latency scores and memory scores associated with multiple packing shapes and pruning thresholds. For example, the pruned, permuted, and packed machine learning model may have a pruning threshold and a packing shape that minimizes the objective function based on the selected constraints. In different examples, the processor may prune and pack the machine learning model sequentially. For example, the processor may calculate a specific combination of pruning, permutation, and packing, and apply the combination on the machine learning model so that the maximum number of neurons or weights are pruned from the machine learning model.

[0100] Fig. 9The processing flow chart of method 900 is not intended to indicate that the operations of method 900 will be performed in any particular order, or that all operations of method 900 will be included in every case. In addition, method 900 may include any appropriate number of additional operations. For example, method 900 may further include performing homomorphic encrypted reasoning using pruned, permuted, and packed machine learning models. In some examples, the processor prunes neurons of the machine learning model. For example, the processor may remove the last few empty columns and empty rows of each weight matrix in the pruned and permuted weight matrices. In different examples, method 900 may further include extending the pruned, packed, and permuted machine learning models to utilize zero values ​​within the packed shapes.

[0101] Fig.10 is a process flow diagram of an example method for selecting a combination of network packing, pruning, and permutation based on an objective function. Method 1000 may be implemented using any suitable computing device, such as Fig.12 The computing device 1200 or Figure 1 For example, the method described below may be performed by Figure 1 , 12 and 15 are implemented by the computing device 102, the processor 1202 or the processor 1502.

[0102] At block 1002, the processor receives a trained machine learning model and an objective function. For example, the trained machine learning model may be encrypted or unencrypted. The objective function may include different constraints.

[0103] At block 1004, for each of a plurality of selected pruning techniques and parameters, the processor prunes the layers of the machine learning model using the selected pruning technique, retrains the pruned machine learning model, and runs the retrained machine learning model on the test set to generate an updated accuracy score. For example, each of the pruning techniques and parameters may be associated with a different updated accuracy score.

[0104] At block 1006, for each combination of a plurality of selected packing configurations and pruning techniques, the processor permutes the pruned machine learning model to increase the amount of zero value packing; packs the permuted, pruned machine learning model; discards zero value packing; and simulates the pruned packed machine learning model to estimate a metric of interest. For example, the metric of interest may include latency, memory usage, and other potential metrics of interest.

[0105] At block 1008, the processor computes an objective function for each pruned and packed machine learning model corresponding to the particular combination of the selected packing configuration and pruning technique based on the corresponding updated accuracy score and the metric of interest.

[0106] At block 1010, the processor outputs the pruned and packed machine learning model with the lowest objective function. For example, a pruned and packed machine learning model that minimizes the objective function given a specific set of constraints may be output.

[0107] Fig.10 The process flow diagram of method 1000 is not intended to indicate that the operations of method 1000 will be performed in any particular order, or that all operations of method 1000 will be included in every case. In addition, method 1000 may include any suitable number of additional operations. For example, method 1000 may further include performing homomorphic encrypted reasoning using the pruned, permuted, and packed machine learning model. In some examples, method 1000 may include extending the pruned, permuted, and packed machine learning model to unprune slices that do not have all zero values, and retraining these slices to improve the reasoning accuracy of the network. For example, each of the unpruned slices may have a full set value instead of having zero values ​​from pruning.

[0108] Fig.11 is a process flow diagram of an example method that can generate an encryption result based on encrypted data using a machine learning model that is packed using pruning and permutation. The method 1100 can be implemented using any suitable computing device, for example, Fig.12 The computing device 1200 or Figure 2 For example, the method described below can be performed by Figure 2 This is achieved by using a pruned, permuted, and packed machine learning model 116.

[0109] At block 1102, the processor sends the encrypted data to the pruned, permuted, and packed machine learning model. For example, the encrypted data may include an encrypted image, or any other type of data to be classified. In various examples, the pruned, permuted, and packed machine learning model may have been trained using the techniques described herein (such as via the above Fig. 9 and 10 Method 900 or 1000) pruning, permutation, and packing.

[0110] At block 1104, the processor receives an encrypted result from the pruned, permuted, and packed machine learning model. For example, the encrypted result may be a classification or image or other data.

[0111] Fig.11The process flow diagram of method 1100 is not intended to indicate that the operations of method 1100 are to be performed in any particular order or that all operations of method 1100 are to be included in every case. Moreover, method 1100 may include any suitable number of additional operations. For example, method 1100 may include decrypting the encrypted result using a key corresponding to a key used to encrypt the encrypted data, the encrypted data being sent to the pruned, permuted, and packed machine learning model.

[0112] It should be understood that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings described herein is not limited to cloud computing environments. Instead, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0113] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0114] Features are as follows:

[0115] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities, such as server time and network storage, as needed without requiring manual interaction with the service provider.

[0116] Wide Area Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0117] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is location independence in the sense that consumers typically do not control or know the exact location of the provided resources, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0118] Rapid elasticity: In some cases, the ability to scale out quickly and in quickly can be provisioned quickly and elastically. To the consumer, the capacity available for provisioning often appears to be unlimited and can be purchased in any quantity at any time.

[0119] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the utilized services.

[0120] The service model is as follows:

[0121] Software as a Service (SaaS): The capability provided to the consumer is to use the provider's applications running on the cloud infrastructure. The applications are accessible from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0122] Platform as a Service (PaaS): The capability provided to consumers is to deploy consumer-created or acquired applications onto cloud infrastructure, where the applications are created using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possible configuration of the application hosting environment.

[0123] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0124] The deployment model is as follows:

[0125] Private Cloud: The cloud infrastructure is operated only for the organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0126] Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0127] Public cloud: Cloud infrastructure is available to the general public or large industrial groups and is owned by an organization that sells cloud services.

[0128] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0129] The cloud computing environment is service-oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure consisting of a network of interconnected nodes.

[0130] Fig.12 1 is a block diagram of an example computing device that can package, prune, and permute machine learning models under selected constraints. Computing device 1200 can be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smart phone. In some examples, computing device 1200 can be a cloud computing node. Computing device 1200 can be described in the general context of computer system executable instructions, such as program modules executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computing device 1200 can be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located in local and remote computer system storage media including memory storage devices.

[0131] The computing device 1200 may include a processor 1202 for executing stored instructions, and a memory device 1204 for providing temporary memory space for the operation of the instructions during operation. The processor may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 1204 may include random access memory (RAM), read-only memory, flash memory, or any other suitable memory system.

[0132] Processor 1202 may be connected via system interconnect 1206 (e.g., The computing device 1200 is connected to an input / output (I / O) device interface 1208 suitable for connecting the computing device 1200 to one or more I / O devices 1210. The I / O devices 1210 may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touch pad or a touch screen, etc. The I / O devices 1210 may be built-in components of the computing device 1200, or may be devices externally connected to the computing device 1200.

[0133] The processor 1202 may also be linked to a display interface 1212 adapted to connect the computing device 1200 to a display device 1214 via a system interconnect 1206. The display device 1214 may include a display screen that is a built-in component of the computing device 1200. The display device 1214 may also include a computer monitor, a television, or a projector, etc., that is externally connected to the computing device 1200. In addition, a network interface controller (NIC) 1216 may be adapted to connect the computing device 1200 to a network 1218 via the system interconnect 1206. In some embodiments, the NIC 1216 may use any suitable interface or protocol to transmit data, such as an Internet Small Computer System Interface, etc. The network 1218 may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, etc. An external computing device 1220 may be connected to the computing device 1200 via the network 1218. In some examples, the external computing device 1220 may be an external network server 1220. In some examples, the external computing device 1220 may be a cloud computing node.

[0134] The processor 1202 may also be linked to a storage device 1222 via a system interconnect 1206, which may include a hard drive, an optical drive, a USB flash drive, a drive array, or any combination thereof. In some examples, the storage device may include a model pruner module 1224, a model permutator module 1226, and a model packager module 1228. The model pruner module 1224 may receive a machine learning model and one or more selected constraints. For example, the selected constraints may include an inference accuracy constraint, a memory constraint, a delay constraint, a delay of amortization, a power constraint, an energy constraint, or any combination thereof. The model pruner module 1224 may prune the machine learning model based on the importance of neurons and weights. For example, the importance may be based on the criticality of the neuron. The criticality of the neuron may be a measure of the accuracy loss caused by removing a particular neuron. In some examples, the importance may be based on the value of the weight. For example, a pruning threshold may be used to set weights having values ​​not exceeding a threshold to zero. The model pruner module 1224 may eliminate operations from the machine learning model. In some examples, an operation may be associated with one or more neurons. The model permutator module 1226 and the model packer module 1228 can permute and pack the remaining neurons and weights of the pruned machine learning model to reduce the amount of ciphertext calculations under the selected constraints. In various examples, the model permutator module 1226 can use any suitable heuristics, such as balanced clustering heuristics, to permute the machine learning model. For example, the balanced clustering heuristic can be balanced k-means clustering. In some examples, the model permutator 1226 can permute the machine learning model using alternating permutations of rows and columns. The model packer module 1228 can pack the machine learning model using any suitable packing method. The model packer module 1228 can use a packing method that reduces ciphertext calculations by maximizing multiple zero-value packing shapes. In some examples, the model pruner module 1224 and the model packer module 1228 can prune and pack sequentially. For example, pruning and packing can be based on a combination of pruning, packing, and permutation determined using an objective function. The objective function evaluator 1230 can calculate the objective function of each of any number of combinations of packing methods, permutation techniques, and pruning thresholds or parameters.

[0135] It should be understood that Fig.12 The block diagram is not intended to indicate that the computing device 1200 is to include Fig.12 , but computing device 1200 may include fewer or Fig.121200). For example, computing device 1200 may also include a model expander for expanding the pruned, packed, and permuted model so that slices that do not have all zero values ​​are not pruned, and retraining these slices is used to improve the inference accuracy of the network. In some examples, computing device 1200 may also include an execution module for executing homomorphic encrypted inference of pruned, permuted, and packaged machine learning models. In addition, any of the functions of model pruner 1224, model permutator module 1226, and model packager module 1228 may be partially or completely implemented in hardware and / or processor 1202. For example, the functions may be implemented using a dedicated integrated circuit, logic implemented in an embedded controller, or logic implemented in processor 1202, etc., etc. In some embodiments, the functionality of the model pruner module 1224, the model permutator module 1226, and the model packager module 1228 may be implemented using logic, where, as mentioned herein, the logic may include any suitable hardware (e.g., a processor, etc.), software (e.g., an application, etc.), firmware, or any suitable combination of hardware, software, and firmware.

[0136] Reference now Fig.13 , depicts an illustrative cloud computing environment 1300. As shown, the cloud computing environment 1300 includes one or more cloud computing nodes 1302 with which a local computing device used by a cloud consumer can communicate, such as a personal digital assistant (PDA) or cellular phone 1304A, a desktop computer 1304B, a laptop computer 1304C, and / or an automobile computer system 1304N. The nodes 1302 can communicate with each other. They can be grouped physically or virtually in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above (not shown). This allows the cloud computing environment 1300 to provide infrastructure, platform, and / or software as a service for which the cloud consumer does not need to maintain resources on a local computing device. It should be understood that Fig.13 The types of computing devices 1304A-N shown in are intended for illustration only, and computing node 1302 and cloud computing environment 1300 may communicate with any type of computing device (eg, using a web browser) over any type of network and / or network-addressable connection.

[0137] Reference now Fig.14 , showing a cloud computing environment 1300 ( Fig.13 ) provides a set of functional abstraction layers. It should be understood in advance that Fig.14 The components, layers, and functions shown in are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0138] The hardware and software layer 1400 includes hardware and software components. Examples of hardware components include: mainframe 1401; server 1402 based on RISC (Reduced Instruction Set Computer) architecture; server 1403; blade server 1404; storage device 1405; and network and network components 1406. In some embodiments, software components include network application server software 1407 and database software 1408.

[0139] Virtualization layer 1410 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 1411 ; virtual storage 1412 ; virtual networks 1413 , including virtual private networks; virtual applications and operating systems 1414 ; and virtual clients 1415 .

[0140] In one example, the management layer 1420 may provide the functionality described below. Resource provisioning 1421 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within a cloud computing environment. Metering and pricing 1422 provides cost tracking when resources are utilized in a cloud computing environment, as well as bills or invoices for consuming these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 1423 provides access to the cloud computing environment for consumers and system administrators. Service level management 1424 provides cloud computing resource allocation and management so that the required service levels are met. Service level agreement (SLA) planning and fulfillment 1425 provides pre-scheduling and procurement of cloud computing resources, where future demand is anticipated based on the SLA.

[0141] The workload layer 1430 provides examples of functions that can utilize the cloud computing environment. Examples of workloads and functions that can be provided from this layer include: mapping and navigation 1431; software development and lifecycle management 1432; virtual classroom education delivery 1433; data analysis processing 1434; transaction processing 1435; and machine learning model optimization 1436.

[0142] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, and the computer-readable program instructions are used to cause a processor to perform various aspects of the present invention.

[0143] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0144] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0145] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or codes or object codes written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and traditional process programming languages ​​such as "C" programming language or similar programming languages. The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, in order to perform various aspects of the present invention, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0146] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present technology. It will be understood that each block of the flowcharts and / or block diagrams and combinations of blocks in the flowcharts and / or block diagrams can be implemented by computer-readable program instructions.

[0147] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0148] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0149] Reference now Fig.15 , depicts a block diagram of an example tangible non-transitory computer-readable medium 1500 that can package, prune, and permute machine learning models under selected constraints. The tangible non-transitory computer-readable medium 1500 can be accessed by a processor 1502 via a computer interconnect 1504. In addition, the tangible non-transitory computer-readable medium 1500 can include instructions for directing the processor 1502 to perform Fig. 9 and 10 The operation codes of methods 900 and 1000 are as follows.

[0150] like Fig.15 As shown, the various software components discussed herein may be stored on a tangible, non-transitory computer-readable medium 1500, for example, the model pruner module 1506 includes code for pruning a machine learning model based on the importance of neurons and weights. The model pruner module 1506 also includes code for setting weights having values ​​not exceeding a threshold to zero. In some examples, the model pruner module 1506 includes code for performing the following operations. In some examples, the model pruner module 1506 includes code for performing the following operations. The model permutator module 1508 includes code and weights for permuting the remaining neurons of the pruned machine learning model to reduce the amount of computation of the ciphertext under the selected constraints. The model permutator module 1508 also includes code for permuting the machine learning model using a heuristic. For example, the model permutator module 1508 may include code for permuting the machine learning model using balanced clustering. In some examples, the model permutator module 1508 may include code for alternating between rows and columns of a weight matrix corresponding to weights between layers of the machine learning model until convergence is detected. The model packager module 1510 includes code for packaging neurons and weights of a pruned machine learning model to reduce the amount of ciphertext computation under selected constraints. The model packager module 1510 also includes code for performing the following operations. The objective function evaluator module 1512 includes code for simulating pruned and packaged machine learning models to obtain latency scores and memory scores associated with several packing shapes and pruning thresholds. For example, the objective function evaluator module 1512 includes code for detecting a pruning threshold and a packing shape that minimizes an objective function based on selected constraints. In some examples, the objective function evaluator module 1512 includes code for retraining a pruned machine learning model and executing the pruned machine learning model to obtain an accuracy score for the pruned machine learning model associated with a specific pruning threshold.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of the architecture, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the boxes may not occur in the order mentioned in the figure. For example, two boxes shown in succession may actually be executed substantially simultaneously, or the boxes may sometimes be executed in the opposite order, depending on the functions involved. It will also be noted that each box illustrated in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart illustration, may be implemented by a dedicated hardware-based system that performs a specified function or action or executes a combination of dedicated hardware and computer instructions. It should be understood that, depending on the specific application, Fig.15 Any number of additional software components not shown in the figure may be included in the tangible, non-transitory computer-readable medium 1500. For example, the computer-readable medium 1500 may also include code for performing homomorphic encryption reasoning using the pruned, permuted, and packed machine learning model. In some examples, the computer-readable medium 1500 may also include code for extending the pruned and permuted machine learning model to not prune zero values ​​within the pruned packed shape of the partial zero values.

[0152] Fig.16 is a diagram illustrating an exemplary process of weight pruning and permutation using an exemplary extension. The exemplary process 1600 may be implemented using any suitable computing device, such as Fig.12 The computing device 1200 or the optional model extender is added Figure 1 For example, the process 700 may be performed by Figure 1 , 12 and 15 , the computing device 102 , the processor 1202 , or the processor 1502 .

[0153] At block 1602, the processor receives a data stream having layers A, B, C, D and weights W. AB , W BC , W CD The trained neural network of BC The transposed matrix of is labeled Before pruning, the weight matrix W AB , W CD Does not contain any zero values.

[0154] At block 1604, several weight values ​​have been pruned to zero by pruning 1606, resulting in two zero slices containing only zero values. As shown, at block 1604, the accuracy of the resulting neural network may be reduced, but the efficiency is increased.

[0155] At block 1608, the order of the weight matrix has been permuted via a permutation operation 1610 to increase the number of zero slices to a total of seven zero slices. As shown in block 1608, accuracy is not affected, but efficiency is improved.

[0156] At block 1612, the accuracy of the neural network has been increased by the expansion operation 1614. Specifically, the zero values ​​of the partially zero slices have been utilized by the expansion operation 1614 to increase the accuracy of the neural network. Specifically, the expansion operation 1614 can de-prune any zero values ​​in the partially zero slices so that these values ​​can be used for training. Thus, block 1612 can recover most of the accuracy loss of block 1604.

[0157] It should be understood that Fig.16 The diagram is not intended to indicate that process 1600 will include Fig.16 In contrast, process 1600 may include fewer or Fig.16 Additional components not shown (e.g., additional layers, neurons, weights, patch shapes, or additional types or iterations of permutations, or may include final packing, etc.).

[0158] Fig.17 1 is a diagram illustrating an example set of different combinations of pruning, permutation, expansion, and packing according to embodiments described herein. The example combinations can be implemented by system 100 to generate pruned, permuted, and packed learning models or pruned, permuted, expanded, and packed machine learning models.

[0159] Fig.17 Various combinations of training, pruning, permuting, expanding, retraining, and packing according to the techniques described herein are shown. These different combinations are represented by the acronyms P2, P2T, P3, P3E, P4, P4E, P5E, and P6. Fig.17 As shown, each combination starts by training a machine learning model. For example, the machine learning model can be a neural network. In various examples, once the trained machine learning model is ready, the processor can prune the neurons or weights of the trained machine learning model based on a certain criterion. All strategies except P2T first pass the above Figure 1Pruning is performed using one of the six pruning configurations discussed in . In the example of P2T, the initial pruning is based on packing. Because P2T performs packing-based pruning, and because we prune complete slices, the processor does not need to perform other steps such as permutation or expansion in P2T. In contrast, when non-packing-aware pruning is performed, the pruned weights or neurons may not necessarily be organized in a good way that will lead to extensive elimination of slice operations. Therefore, the processor can apply additional operations, such as permutation or expansion, to improve the effective use of slices. As described above, the permutation operation can include permuting the rows and columns of the weight matrix after the pruning operation to bring the zero elements together. The expansion operation is opposite to the pruning operation. For example, the expansion operation can include searching for slices that do not only hold only zero values, and removing the pruning of the zero elements within these slices. Therefore, in the embodiments of P3, P3E, P4, P4E, P5E and P6, permutation is then also performed.

[0160] In the example of P4, instead of expanding the model as in P3, the processor can perform a second pruning-aware-packing step to reduce all incomplete zero pieces. In the examples of P5 and P6, after the first permutation step, the processor can perform a semi-packing-aware pruning method Prune sim-pack , which locates partially zeroed slices and prunes some more elements inside them instead of all. Subsequently, the processor can reapply the permutation algorithm. In this way, the processor can help the permutation heuristic while maintaining the pruning configuration originally applied. After the second permutation step, the processor can determine whether to expand or pack-awarely prune the slice based on the number of zeros inside the slice. Finally, for all combinations, the processor can perform retraining of the machine learning model to increase the accuracy and final packing of the retrained machine learning model. In various examples, these integrated combinations of permutation, expansion, pruning, and packing provide various trade-offs between accuracy, performance, and memory consumption, and therefore provide options for various use cases.

[0161] It should be understood that Fig.17 The diagram is not intended to indicate that the set 1700 is to include Fig.17 In contrast, process 1700 may include fewer or Fig.17 Additional components not shown in (e.g., additional training, pruning, permutation, expansion, retraining or packaging, etc.). Fig.17 It is not intended to be an exhaustive list of combinations of the various operations described herein.

[0162] The description of various embodiments of the present technology has been presented for the purpose of illustration, but it is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements existing in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A system comprising a processor configured to: Pruning machine learning models based on the importance of neurons or weights; and Permute and pack the remaining neurons or weights of the pruned machine learning model to reduce the amount of ciphertext computation under the chosen constraints.

2. The system according to claim 1, wherein: The processor is used to prune and pack sequentially.

3. The system according to claim 1, wherein: The importance is based on the criticality of the neuron.

4. The system according to claim 1, wherein: The importance is based on the value of the weight.

5. The system according to claim 1, wherein: The selected constraints include inference accuracy constraints.

6. The system according to claim 1, wherein: The selected constraints include memory constraints.

7. The system according to claim 1, wherein: The selected constraints include delay constraints.

8. The system according to claim 1, wherein: Pruning the machine learning model includes eliminating operations from the machine learning model.

9. The system according to claim 1, wherein: The ciphertext computation includes performing homomorphic encrypted inference of the pruned, permuted, and packed machine learning models.

10. A computer-implemented method comprising: Pruning a machine learning model based on the importance of neurons or weights via a processor; as well as The remaining neurons or weights of the pruned machine learning model are permuted and packed via the processor to reduce the amount of ciphertext computation under the selected constraints.

11. The computer-implemented method of claim 10, further comprising: Perform homomorphic encrypted inference using pruned, permuted, and packed machine learning models.

12. The computer-implemented method of claim 10, wherein: Pruning the machine learning model includes: pruning weights of the machine learning model by setting weights having values ​​not exceeding a threshold to zero.

13. The computer-implemented method of claim 10, wherein: Pruning a machine learning model includes: pruning neurons of the machine learning model.

14. The computer-implemented method of claim 10, wherein: Permuting the machine learning model includes using balanced clustering.

15. The computer-implemented method of claim 10, wherein: Permuting the machine learning model includes alternating between permuting rows and columns of a weight matrix corresponding to weights between layers of the machine learning model until convergence is detected.

16. The computer-implemented method of claim 10, further comprising: The pruned and permuted machine learning model is extended to not prune zero values ​​within a pruned packed shape of partial zero values.

17. The computer-implemented method of claim 10, further comprising: A pruned and packed machine learning model is simulated to obtain latency scores and memory scores associated with multiple packing shapes and pruning thresholds, wherein the pruned, permuted, and packed machine learning model includes pruning thresholds and packing shapes that minimize an objective function based on the selected constraints.

18. A computer program product for pruning and packaging a machine learning model, the computer program product comprising a computer-readable storage medium having program code embodied therewith, the program code being executable by a processor to cause the processor to: Pruning machine learning models based on the importance of neurons or weights; and Permute and pack the remaining neurons or weights of the pruned machine learning model to reduce the amount of ciphertext computation under the chosen constraints.

19. The computer program product of claim 18, further comprising: Program code executable by the processor to set weights having values ​​not exceeding a threshold to zero.

20. The computer program product of claim 18, further comprising: Program code executable by the processor to permute the machine learning model using a heuristic.

21. The computer program product of claim 18, further comprising: Program code executable by the processor to permute the machine learning model using balanced clustering.

22. The computer program product of claim 18, further comprising: Program code executable by the processor to alternate between permuting rows and columns of a weight matrix corresponding to weights between layers of the machine learning model until convergence is detected.

23. The computer program product of claim 18, further comprising: Program code executable by the processor to retrain the pruned machine learning model and execute the pruned machine learning model to obtain an accuracy score for the pruned machine learning model associated with a particular pruning threshold.

24. The computer program product of claim 18, further comprising: Program code executable by the processor to simulate a pruned and packed machine learning model to obtain latency scores and memory scores associated with multiple packing shapes and pruning thresholds, wherein the pruned, permuted, and packed machine learning model includes pruning thresholds and packing shapes that minimize an objective function based on the selected constraints.

25. The computer program product of claim 18, further comprising: Program code executable by the processor to perform homomorphically encrypted inference using the pruned, permuted, and packed machine learning models.

Images