Bayesian Optimization of Sparsity Rate in Model Compression

Abstract

The present invention discloses Bayesian optimization of sparsity rate in model compression. One embodiment of the method includes: determining, by a Bayesian optimizer, a first sparsity rate associated with a limit on the loss of accuracy due to compressing a machine learning model. The method also includes selecting, by the Bayesian optimizer, a second sparsity rate that optimizes a predefined objective function for the machine learning model within a search space defined by the first sparsity rate. The method also includes generating a compressed version of the machine learning model having the second sparsity rate.

Description

Background Art

[0001] Fields of various embodiments

[0002] Embodiments of the present disclosure relate generally to compression of machine learning models, and more specifically, to Bayesian optimization of sparsity ratios in model compression.

[0003] Description of Related Technology

[0004] Deep neural networks typically have a large number of computationally heavy layers, such as convolutional layers and / or fully connected layers. Such neural networks are also often trained and deployed using full-precision arithmetic. As a result, executing DNNs can involve a lot of complexity, latency, and memory consumption.

[0005] A variety of techniques can be employed to reduce the memory footprint and inference time of a deep neural network. For example, the parameters and / or activations of the neural network can be quantized to reduce the bit width of the parameters (e.g., from 32-bit floating point to 8-bit floating point) and / or certain matrix multiplication operations to compute activations can be converted to logical operations. In addition, weights in the neural network can be pruned (i.e., removed) when the weights have an absolute value and / or an impact on the training error of the neural network below a threshold.

[0006] Existing techniques for compressing neural networks require an explicit specification of a target sparsity ratio, which represents the proportion of parameters in the pruned model that have zero values ​​compared to the original model. Alternatively, existing techniques perform manual exploration of the search space to prune or compress the neural network. Manual exploration and / or selection of the amount of compression is computationally intensive and inconvenient, since the optimal sparsity ratio depends on the neural network, the target platform, and the optimization goal, and each trial in the exploration requires retraining the compressed model to converge.

[0007] As mentioned previously, there is a need in the art for more efficient techniques to determine the optimal sparsity ratio for compressing machine learning models. Summary of the invention

[0008] One embodiment of the present invention describes a technique for compressing a machine learning model. The technique includes determining, by a Bayesian optimizer, a first sparsity rate associated with a limit on accuracy loss caused by compressing the machine learning model. The technique also includes: selecting, by the Bayesian optimizer, a second sparsity rate that optimizes a predefined objective function for the machine learning model within a search space defined by the first sparsity rate. The technique also includes generating a compressed version of the machine learning model having the second sparsity rate.

[0009] One technical advantage of the disclosed technology is the ability to infer a sparsity rate that satisfies constraints and / or goals associated with compressing a machine learning model, rather than requiring a user to manually specify a sparsity rate to compress the machine learning model. Thus, the disclosed technology provides technical improvements in computer systems, applications, and / or techniques for compressing and / or executing machine learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order that the manner in which the above-described features of various embodiments may be understood in detail, a more particular description of the inventive concept briefly outlined above may be given by reference to various embodiments, some of which are shown in the accompanying drawings. It should be noted, however, that the drawings illustrate only typical embodiments of the inventive concept and are therefore not to be considered in any way limiting of the scope and that there are other equally effective embodiments.

[0011] Figure 1 A system configured to implement one or more aspects of the various embodiments is shown.

[0012] Figure 2 According to one or more aspects of various embodiments Figure 1 A more detailed description of the Bayesian optimizer and compression engine.

[0013] Figure 3 According to one or more aspects of various embodiments, Figure 2 An example graph of the iterations performed by the Bayesian optimizer.

[0014] Figure 4 is a flow chart of method steps for selecting a sparsity rate to compress a machine learning model according to one or more aspects of various embodiments.

[0015] Figure 5 is a flowchart of method steps for compressing a machine learning model according to one or more aspects of various embodiments. DETAILED DESCRIPTION

[0016] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the inventive concept may be practiced without one or more of these specific details.

[0017] System Overview

[0018] Figure 1A computing device 100 configured to implement one or more aspects of various embodiments is shown. In one embodiment, the computing device 100 may be a desktop computer, a laptop computer, a smart phone, a personal digital assistant (PDA), a tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images, and which is suitable for practicing one or more embodiments. The computing device 100 is configured to run a Bayesian optimizer 122 and a compression engine 124 resident in a memory 116. It should be noted that the computing device described herein is illustrative, and any other technically feasible configurations fall within the scope of the present disclosure. For example, multiple instances of the Bayesian optimizer 122 and the compression engine 124 may be executed on a set of nodes in a distributed system to implement the functionality of the computing device 100.

[0019] In one embodiment, the computing device 100 includes, but is not limited to, an interconnect (bus) 112 connecting one or more processing units 102, an input / output (I / O) device interface 104 coupled to one or more input / output (I / O) devices 108, a memory 116, a storage 114, and a network interface 106. The one or more processing units 102 may be any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), an artificial intelligence (AI) accelerator, any other type of processing unit, or a combination of different processing units (e.g., a CPU configured for use in conjunction with a GPU). In general, the one or more processing units 102 may be any technically feasible hardware unit capable of processing data and / or executing software applications. In addition, in the context of the present disclosure, the computing elements shown in the computing device 100 may correspond to a physical computing system (e.g., a system in a data center), or may be a virtual computing instance executed within a computing cloud.

[0020] In one embodiment, I / O devices 108 include devices capable of providing input (such as a keyboard, a mouse, a touch screen, etc.) and devices capable of providing output (such as a display device). In addition, I / O devices 108 may include devices capable of both receiving input and providing output (such as a touch screen, a universal serial bus (USB) port, etc.). I / O devices 108 may be configured to receive various types of input from an end-user (e.g., a designer) of computing device 100, and may also provide various types of output to the end-user of computing device 100, such as displayed digital images or digital video or text. In some embodiments, one or more I / O devices 108 are configured to couple computing device 100 to network 110.

[0021] In one embodiment, network 110 is any technically feasible communication network that allows data to be exchanged between computing device 100 and an external entity or device such as a network server or another networked computing device. For example, network 110 may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and / or the Internet, among others.

[0022] In one embodiment, storage 114 includes non-volatile memory for applications and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. Bayesian optimizer 122 and compression engine 124 may be stored in storage 114 and loaded into memory 116 when executed.

[0023] In one embodiment, the memory 116 includes a random access memory (RAM) module, a flash memory unit, or any other type of memory unit or combination thereof. The one or more processing units 102, the I / O device interface 104, and the network interface 106 are configured to read data from the memory 116 and write data to the memory 116. The memory 116 includes various software programs executable by the one or more processors 102 and application data associated with the software programs, including the Bayesian optimizer 122 and the compression engine 124.

[0024] In some embodiments, the Bayesian optimizer 122 and the compression engine 124 include functionality to improve the runtime performance of a machine learning model by compressing the machine learning model. For example, a machine learning model includes a deep neural network (DNN) with a large number of neurons, parameters, and / or layers. As a result, the reasoning of the DNN requires a large consumption of computing and memory resources. In order to improve reasoning time, memory usage, and / or other runtime properties of the machine learning model, the Bayesian optimizer 122 and / or the compression engine 124 creates a compressed version of the machine learning model by selectively pruning a subset of weights in the DNN and / or quantizing the weights. In turn, the compressed version generates much less resource overhead and latency than the uncompressed model without a significant decrease in accuracy.

[0025] More specifically, the Bayesian optimizer 122 and the compression engine 124 include functionality for optimizing the compression of a machine learning model based on predefined and / or adjustable objectives and / or constraints. Continuing with the above example, the Bayesian optimizer 122 receives from a user an objective to be optimized when compressing the machine learning model, a lower limit on the loss of accuracy associated with compressing the machine learning model, and / or a compression scheme. The Bayesian optimizer 122 and the compression engine 124 then iteratively sample different sparsity rates to compress the machine learning model using the compression scheme. Ultimately, the Bayesian optimizer 122 and the compression engine 124 produce a compressed version of the machine learning model with a sparsity rate that best optimizes the objective while satisfying the constraints represented by the limit on loss of accuracy. Refer below Figure 2 Various embodiments of the Bayesian optimizer 122 and the compression engine 124 are described in more detail.

[0026] Bayesian Optimization of Sparsity Rate in Model Compression

[0027] Figure 2 According to various embodiments Figure 1 1 and 12. A more detailed illustration of the Bayesian optimizer 122 and the compression engine 124 of FIG. 1 is provided. In the illustrated embodiment, the Bayesian optimizer 122 samples the sparsity rates 218-220 based on a predefined precision loss limit 206, an objective function 208, and a compression scheme 240 to generate a compressed machine learning model 226 from a reference machine learning model 222 (e.g., an uncompressed machine learning model and / or a compressed machine learning model to be further compressed). For example, the Bayesian optimizer 122 receives the precision loss limit 206, the objective function 208, and / or the compression scheme 240 from a user via a function call, a command line parameter, a graphical user interface (GUI), and / or other types of interfaces or communication mechanisms.

[0028] In some embodiments, the sparsity rate (e.g., sparsity rates 218-220) represents the proportion of parameters that have been pruned (e.g., replaced with zero values) in the compressed machine learning model 226 compared to the original reference machine learning model 222. The sparsity rate also (or alternatively) reflects the ratio of the size of the compressed machine learning model 226 to the size of the original reference machine learning model 222. For example, the sparsity rate can be obtained by subtracting the density ratio of the compressed machine learning model 226 from 1, where the density ratio is equal to the size of the compressed machine learning model 226 divided by the size of the reference machine learning model 222. In general, the sparsity rate represents a reduction in the size of the compressed machine learning model 226 when compared to the original reference machine learning model 222.

[0029] The accuracy loss limit 206 represents a lower limit on the reduction in accuracy caused by generating the compressed machine learning model 226 from the reference machine learning model 222. For example, the user represents the accuracy loss limit 206 as a percentage or ratio of the accuracy of the reference machine learning model 222.

[0030] In some embodiments, objective function 208 comprises a black-box function to be maximized or minimized. For example, a user specifies objective function 208 as inference throughput (e.g., the rate at which compressed machine learning model 226 performs evaluations), memory footprint, latency (e.g., the amount of time required for compressed machine learning model 226 to perform evaluations), a domain-specific goal (e.g., "smoothness" or frame rate of video output), and / or another value that can be measured or calculated from the execution or output of compressed machine learning model 226. The user additionally specifies minimization or maximization of objective function 208 as a goal associated with compressed machine learning model 226.

[0031] The compression scheme 240 includes the type of compression to be applied to produce the compressed machine learning model 226. For example, the user selects the compression scheme 240 from a list that includes, but is not limited to, quantization of parameters (e.g., from 32-bit floating point to 16-bit floating point), unstructured magnitude-based pruning of weights in the DNN, pruning of neurons in the DNN, pruning blocks of various dimensions in the DNN, and / or low-rank tensor decomposition. In another example, the user programmatically expresses the compression scheme 240 as a combination of multiple compression schemes with control flow, iteration, recursion, and / or other imperative statements. These statements allow for selective compression of layers and / or other portions of the reference machine learning model 222 based on properties of the reference machine learning model 222 and / or the target hardware on which the compressed machine learning model 226 is to be deployed.

[0032] In one or more embodiments, the Bayesian optimizer 122 and the compression engine 124 perform a number of iterations 202-204 that sample different sparsity rates 218 from the reference machine learning model 222 using a compression scheme 240 for producing a compressed machine learning model 226. During the iterations, the Bayesian optimizer 122 selects a sparsity rate for the compressed machine learning model 226, and the compression engine 124 generates the compressed machine learning model 226 from the reference machine learning model 222 using the compression scheme 240 in a manner that respects the selected sparsity rate. The compression engine 124 also measures accuracy 236, performance 238, and / or another value related to the execution or output of the compressed machine learning model 226 and provides the measurements to the Bayesian optimizer 122. The Bayesian optimizer uses the measurements from the compression engine 124 to update the probabilistic confidences about the output as a function of the sparsity rate in the compressed machine learning model 226, and uses an acquisition function to select the next sparsity rate to sample in the next iteration. At the end of iterations 202 - 204 , the Bayesian optimizer 122 and the compression engine 124 generate a compressed machine learning model 226 having a sparsity ratio that optimizes the objective function 208 while satisfying the constraints represented by the precision loss limit 206 .

[0033] During each iteration performed by the Bayesian optimizer 122, the compression engine 124 generates a compressed machine learning model 226 having a sparsity ratio selected by the Bayesian optimizer 122 for that iteration. As shown, such compression includes training the reference machine learning model 222 with a quadratic regularization term 232, and a learning step 228 of a compression step 230 and a compression step 230 of converting the trained reference machine learning model 222 from the learning step 228 into a compressed machine learning model 226 having parameters 234.

[0034] For example, compression of the reference machine learning model 222 by the compression engine 124 is formulated as a constrained optimization problem having the following representation:

[0035] min w,θ L(w)stw=D(θ)

[0036] In the above representation, L() represents the loss function used to train the reference machine learning model 222, and w∈R P Decompression mapping D: θ∈R represents the uncompressed parameters (e.g., neural network weights) of the reference machine learning model 222. P →w∈R n The compressed parameters 234θ of the compressed machine learning model 226 are mapped to uncompressed parameters.

[0037] The above formula supports a variety of compression schemes. For example, pruning is defined as w = D (θ) = θ, where w is a real number, and θ is constrained to have fewer non-zero values ​​by removing (zeroing) smaller magnitude weights. In another example, a low-precision approximation (i.e., quantization) defines each parameter constraint w i =θ i , where w i is a higher precision representation of the weights in the reference machine learning model 222, and θ i is a lower precision representation of the same weights in the compressed machine learning model 226.

[0038] The compression map behaves like the inverse of D:

[0039]

[0040] In the above expression, C:R n →R + represents a cost function for promoting sparsity in compression parameters 234, where C(0)=0 and C(w)>0 (if w=0). κ represents a constraint on the number or complexity of compression parameters 234, and the "≤" super-index refers to the constrained form of the compression mapping.

[0041] In turn, the constrained optimization problem can be expressed using the following alternative representation:

[0042] min w,θ L(w)stC(θ)≤κ,w=θ

[0043] The constrained optimization problem can then be optimized via a quadratic regularization term 232 and / or an augmented-Lagrangian (AL) method:

[0044]

[0045]

[0046] The first equation uses the value of the quadratic regularization term 232 to optimize Q on (w,θ) while driving μ→∞ so that the equality constraint is satisfied. The second equation uses the AL method to alternately optimize L on (w,θ) by driving μ→∞ while updating λ←λ-μ(w-θ) A .

[0047] To optimize the above equation for the variables (w, θ), the compression engine 124 performs alternating optimization using a learning step 228 and a compression step 230. During the learning step 228, the compression engine 124 performs the following optimization using the following equation:

[0048]

[0049] The second term in the above equation represents the quadratic regularization term 232, which is used to train the reference machine learning model 226 to pull certain parameters to zero and other parameters to non-zero values.

[0050] If the AL method is used in the optimization, the compression engine 124 performs the learning step 228 using the following formula:

[0051]

[0052] During the compression step 230, various costs C may be calculated, such as (but not limited to) l 0 , l 1 , and l 2 2 Each of these costs can be solved by projecting onto l p When w is in the ball, θ = w. Otherwise, l 0 will keep the first κ weights unchanged and prune the rest, l 1 The first k weights are shrunk on average (where k depends on w and κ) and the rest are pruned, and all weights are shrunk by normalizing w 2 2 .

[0053] In addition, the compression step 230 is performed by setting θ I The subsequent learning step 228 then uses the loss function and the labeled θ I Update value w i As the compression engine 124 iteratively alternates between the learning step 228 and the compression step 230, different sets of label weights are explored until a set of label weights is found for which w i →0 is μ→∞, which will cause these weights to be pruned. Therefore, the compression engine 124 generates a compressed machine learning model 226 with a given sparsity rate and a local optimal accuracy loss of the reference machine learning model 222.

[0054] In some embodiments, the Bayesian optimizer 122 performs a first set of iterations 202 that samples sparsity rates 218 to generate a compressed machine learning model 226 to identify the highest sparsity rate that falls within the precision loss limit 206. The Bayesian optimizer 122 then sets the identified sparsity rate as a boundary of a search space 224 for a second set of iterations 204. In the second set of iterations 202, the Bayesian optimizer 122 samples other sparsity rates 220 for generating a compressed machine learning model 226 within the search space 244 to identify the sparsity rate that produces the best performance 238 in optimizing the objective function 208.

[0055] In one embodiment, the Bayesian optimizer 122 uses a Gaussian process (GP) prior distribution for a given black box function f(x), which may include the accuracy 236 and / or the objective function 208. The GP includes the distribution of the function, which is specified by the mean function m:X→R and the covariance function K:X×X→R. As the Bayesian optimizer 122 and the compression engine 124 accumulate observations D 1:t ={x 1:t ,y 1:t}, the Bayesian optimizer 122 combines the prior distribution P(f) with the likelihood function P(D 1:t |f) to produce the posterior distribution P(f|D 1:t )∝P(D 1:t |f)P(f).

[0056] During each of iterations 202, the compression engine 124 generates a compressed machine learning model 226 at the sparsity rate (e.g., sparsity rate 218) sampled by the Bayesian optimizer 122 for that iteration. The compression engine 124 also determines a precision 236 of the compressed machine learning model 226, and the Bayesian optimizer 122 updates the GP prior using the calculated precision 236 before optimizing the function through the GP prior to determine the next sparsity rate to sample in a consecutive iteration.

[0057] At the end of iteration 202, the Bayesian optimizer 122 sets the boundary of the search space 224 for iteration 204 to a range from sparsity rate 0 to the sparsity rate identified by iteration 202 as having substantially the same precision 236 as the precision loss limit 206. During each iteration in iteration 204, the compression engine 124 generates a compressed machine learning model 226 at the sparsity rate (e.g., sparsity rate 220) sampled by the Bayesian optimizer 122 for that iteration. The compression engine 124 also determines a performance 238 of the compressed machine learning model 226 as the value of the objective function 208 of the compressed machine learning model 226, and the Bayesian optimizer 122 updates the GP prior using the performance 238 before obtaining the function by optimizing the GP prior to determine the next sparsity rate to sample in a subsequent iteration.

[0058] The Bayesian optimizer also uses a sampling criterion represented by one or more acquisition functions 214-216a: X→R to guide the selection of the next sparsity rate to sample. These acquisition functions 214-216 depend on the previous observations {x n ,y n} and GP hyperparameter ρ. Under the GP prior, each acquisition function 214 or 216 depends on the predicted average function μ(x; {x n ,y n},ρ) and the prediction variance function σ2 (x; {x n ,y n},ρ).

[0059] The Bayesian optimizer 122 additionally selects the sparsity rates 218-220 based on the acquisition functions 214-216 for each set of iterations 202-204 using a combination of random sampling 210-212 and a sampling criterion. For example, the Bayesian optimizer 122 begins iteration 202 with one or more boundaries of the random sampling 210 of the sparsity rate 218. The Bayesian optimizer 122 obtains the value of the precision 236 returned by the compression engine 124 for the randomly sampled sparsity rates 218 and updates the GP prior of the precision 236 using the value. The Bayesian optimizer 122 then performs an additional iteration 202 that samples the sparsity rate 218 according to the acquisition function 214 associated with the GP prior of the precision 236.

[0060] Continuing with the above example, the Bayesian optimizer 122 similarly begins iterations with one or more iteration boundaries of random sampling 212 of the sparsity rates 220. The Bayesian optimizer 122 obtains values ​​of the performance 238 returned by the compression engine 124 for the randomly sampled sparsity rates 220 and uses these values ​​to update the GP prior for the objective function 208. The Bayesian optimizer 122 then performs an additional iteration 204 that samples the sparsity rates 220 according to an acquisition function 216 associated with the GP prior for the objective function 208.

[0061] The operation of the Bayesian optimizer 122 is illustrated using the following example pseudocode:

[0062]

[0063] In the above pseudo code, the input of the Bayesian optimizer 122 includes a reference machine learning model 222w and a precision loss limit 206ε. The output of the Bayesian optimizer 122s* represents the sparsity rate of the compressed machine learning model 226, which has the best performance 238 in the optimization objective function 208, and the value of the precision 236 falls within the precision loss limit 206.

[0064] More specifically, the above pseudo code uses an implicit level set upper confidence bound (ILS-UCB) acquisition function 214 for iteration 202, where the level set L is defined by subtracting ε from the accuracy of the reference machine learning model 222. Conversely, iteration 202 preferentially samples a sparsity rate 218 close to the level set to locate the sparsity that satisfies the accuracy loss limit 206, reducing the optimal sparsity rate s to obtain the accuracy loss limit 206. acc The total number and time of measurements required.

[0065] For example, the ILS-UCB acquisition function 214 used to select the sparsity rate 218 during iteration 202 includes the following representation:

[0066]

[0067] In the above representation, γ represents a tuning parameter between 0 and 1 that balances the contribution of the predicted mean of the acquisition function 214 to the predicted variance of the samples at iteration t.

[0068] Next, the pseudocode uses the acc ) constrained search space and the acquisition function of GP-UCB as the acquisition function 216 of iteration 204. Then, the pseudo code is used to perform iteration 204 in a manner to maximize the objective function 208 to sample the sparsity rate 220. Alternatively, a different acquisition function 216 of the GP lower confidence bound (GP-LCB) can be used to sample the sparsity rate 220 in a manner to minimize the objective function 208. When the acquisition function 216 selects the same sparsity rate in two consecutive iterations 204, the iteration 204 is interrupted and the selected sparsity rate is returned as the optimal sparsity rate s*.

[0069] Figure 3 According to one or more aspects of various embodiments, Figure 2 20. A collection of exemplary plots 302-304 of iterations performed by the Bayesian optimizer 122 of FIG. More specifically, plot 302 shows a prediction 306 of the Bayesian optimizer 122 for the black-box function as a function of the sparsity rate of the compressed machine learning model 226, and plot 304 shows an acquisition function 322 used by the Bayesian optimizer 122 to select the next sample 320 of the black-box function to be evaluated.

[0070] Within graph 302, prediction 306 represents the posterior predicted mean of the GP prior conditioned on previous samples 312-318 of the black-box function. Graph 302 also includes a shaded area 310 around prediction 306, which represents a 95% confidence interval for the mean. Graph 302 additionally shows the target 308 value of the black-box function that will be predicted by the Bayesian optimizer 122 via samples 312-318.

[0071] In one embodiment, samples 312-318 include four observations of the accuracy 236 of the compressed machine learning model 226 at various sparsity rates 218 sampled by the Bayesian optimizer 122 over four iterations 202. For example, samples 312-314 represent two randomly sampled sparsity rates used to "seed" the GP prior, and samples 316-318 represent sparsity rates selected by the Bayesian optimizer 122 based on the acquisition function 322. After evaluating the black-box function for samples 316-318, the Bayesian optimizer 122 updates the GP prior to produce an updated prediction 306 that better approximates the black-box function. The Bayesian optimizer 122 then uses the updated prediction 306 to construct or update the ILS-UCB acquisition function 322, and selects the next sample 320 for the consecutive iteration as the maximum value of the acquisition function 322.

[0072] For example, the Bayesian optimizer 122 uses ILS-UCB to prioritize sampling in a region near a level set where the average of the GP priors is closest to a threshold 324 representing the precision loss limit 206 of the compressed machine learning model 226. After multiple iterations 202, the Bayesian optimizer 122 identifies the sample 318 as having a sparsity rate that produces a compressed machine learning model 226 with an accuracy 236 equal to the precision loss limit 206. The Bayesian optimizer 122 then performs additional iterations 204 of sampling the sparsity rate 220 in a search space 224 bounded by the sparsity rate of the sample 318 to select another sparsity rate for producing a compressed machine learning model 226 that optimizes the user-specified objective function 208 while satisfying the precision loss limit 206.

[0073] Figure 4 is a flowchart of method steps for compressing a machine learning model according to one or more aspects of various embodiments. Figure 1 and Figure 2 The method steps are described herein as a system configured to perform the method steps in any order, but those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

[0074] As shown, the Bayesian optimizer 122 receives 402 a predefined objective function, a limit on the loss of accuracy due to compressing the machine learning model, and / or a compression scheme from a user. For example, the objective function includes an evaluation rate of the machine learning model, memory usage, evaluation time of the machine learning model, a domain-specific goal (e.g., "smoothness" or a frame rate of a video output), and / or another value that can be measured or calculated from the execution or output of the machine learning model. The user additionally specifies minimization or maximization of the objective function as a goal associated with the compressed machine learning model. In another example, the user specifies the limit on the loss of accuracy as a percentage or proportion of the accuracy of the uncompressed machine learning model. In a third example, the user specifies the compression scheme as quantization, unstructured amplitude-based pruning of weights in the DNN, neuron pruning in the DNN, block pruning of individual dimensions in the DNN, and / or low-rank tensor decomposition.

[0075] Next, the Bayesian optimizer 122 determines 404 a first sparsity rate associated with a limit on the loss of precision resulting from compressing the machine learning model using the compression scheme 404. For example, the Bayesian optimizer 122 performs a first series of iterations using the ILS-UCB sampling criterion to sample a sparsity rate for compressing the machine learning model. At the end of the first series of iterations, the Bayesian optimizer 122 identifies the first sparsity rate as an amount of compression (e.g., zero-valued parameters, parameter quantization, etc.) in the machine learning model that produces a precision corresponding to the limit on the loss of precision.

[0076] The Bayesian optimizer 122 then selects 406 a second sparsity rate that optimizes the objective function of the machine learning model within the search space defined by the first sparsity rate. For example, the Bayesian optimizer performs a second series of iterations that uses the GP-UCB (or GP-LCB) sampling criterion to sample the sparsity rate that maximizes (or minimizes) the objective function, where the sampled sparsity rates range from 0 to the first sparsity rate. At the end of the second series of iterations, the Bayesian optimizer 122 identifies the second sparsity rate as the amount of compression in the machine learning model that optimizes the objective function within a range beyond the precision loss limit.

[0077] During each iteration of selecting the first sparsity rate and the second sparsity rate performed by the Bayesian optimizer 122, the compression engine 124 generates a different compressed version of the machine learning model, where each compressed version has the sparsity rate selected by the Bayesian optimizer 122 for that iteration. The compression engine 124 also determines the accuracy and / or value of the objective function of the sparsity rate, and the Bayesian optimizer 122 updates the GP prior for the accuracy and / or the objective function based on the accuracy and / or value of the objective function. The operation of the compression engine 124 is described below with reference to Figure 5 Described in further detail.

[0078] Finally, the Bayesian optimizer 122 and / or the compression engine 124 generates 408 a compressed version of the machine learning model having a second sparsity rate. For example, the Bayesian optimizer 122 samples the second sparsity rate by optimizing the acquisition function on the GP prior of the objective function. The compression engine 124 then compresses the machine learning model to have the second sparsity rate and verifies that the compressed machine learning model produces a better objective function value than the previously compressed version of the machine learning model.

[0079] Figure 5 is a flowchart of method steps for compressing a machine learning model according to one or more aspects of various embodiments. Figure 1 and Figure 2 The method steps are described herein as a system configured to perform the method steps in any order, but those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

[0080] As shown, the compression engine 124 receives 502 sparsity rates sampled during iterations performed by the Bayesian optimizer 122. For example, the sparsity rates include randomly sampled sparsity rates used to "seed" a GP prior for a black-box function to be optimized, and / or sparsity rates selected by the Bayesian optimizer 122 based on sampling criteria and / or an acquisition function associated with the GP prior.

[0081] Next, the compression engine 124 performs 504 a learning step that trains the machine learning model with a quadratic regularization term. For example, the compression engine 124 updates the parameters of the machine learning model using a loss function that includes a quadratic regularization term. In turn, the quadratic regularization term applies a penalty that pulls certain parameters to zero based on squares calculated from the parameter values.

[0082] After the learning step is completed, the compression engine 124 performs 506 a compression step that compresses the machine learning model to have a sparsity ratio provided by the Bayesian optimizer. For example, the compression engine 124 marks a subset of parameters in the machine learning model to be compressed.

[0083] The compression engine 124 additionally repeats operations 504-506 until a stopping criterion is met 508. For example, the stopping criterion includes convergence in the parameters of the compressed machine learning model. As a result, the compression engine 124 iteratively alternates between learning steps and compression steps, which allows different sets of labeled parameters to be explored until the training parameters produced by the learning step are equal to the labeled parameters in the compression step. In another example, the compression engine 124 repeats operations 504-506 until a fixed number of learning and compression steps have been performed.

[0084] The compression engine 124 then provides 510 a measure of the accuracy and / or performance of the compressed machine learning model to the Bayesian optimizer 122. For example, the compression engine 124 determines the accuracy of the compressed machine learning model during a first series of iterations performed by the Bayesian optimizer to find a first sparsity ratio that reflects a precision loss limit for the compressed machine learning model. The compression engine 124 also (or alternatively) determines to optimize the performance of the compressed machine learning model in a predefined objective function during a second series of iterations performed by the Bayesian optimizer 122 to find a second sparsity ratio that produces an optimal value for the objective function.

[0085] Operations 502-510 may be repeated for the remaining iterations 512 performed by the Bayesian optimizer 122. For example, the compression engine 124 receives a new sparsity rate at the beginning of each iteration performed by the Bayesian optimizer 122, creates a compressed machine learning model with the sparsity rate, and provides the accuracy and / or performance of the compressed machine learning model to the Bayesian optimizer 122. The Bayesian optimizer 122 then uses the measured accuracy and / or performance to update the GP prior for accuracy and / or performance, and uses the acquisition function and / or selection criteria of the GP prior to select the next sparsity rate to sample in consecutive iterations. As a result, the compression engine 124 repeatedly generates compressed versions of the machine learning model and determines the accuracy and / or performance of each compressed version until a compressed version with the best performance that still maintains an acceptable level of accuracy is found.

[0086] In summary, the disclosed embodiments perform sample efficient Bayesian optimization of sparsity rates in a compressed machine learning model. A first series of Bayesian optimization iterations are performed to identify a first sparsity rate that represents a lower bound on the loss of precision for the compressed machine learning model. A second series of Bayesian optimization iterations are then performed to identify a second sparsity rate that produces an optimal value of a predefined objective function within a search space defined by the first sparsity rate. The machine learning model is then compressed to have the second sparsity rate, which enables the compressed machine learning model to optimize the objective function while maintaining an accuracy above the lower bound on the loss of precision.

[0087] One technical advantage of the disclosed technology is the ability to infer a sparsity rate that satisfies constraints and / or goals associated with compressing a machine learning model, rather than requiring a user to manually specify a sparsity rate to compress the machine learning model. Subsequently, the compressed machine learning model incurs significantly less inference time and resource overhead than an uncompressed machine learning model, without a significant decrease in accuracy. Another technical advantage includes sample-efficient selection of sparsity rates via Bayesian optimization, which greatly reduces the number of iterations (and associated delays and resource consumption) required to identify an optimal sparsity rate for compressing a machine learning model using traditional techniques that involve manual and / or random searches for sparsity rates to compress a machine learning model. Therefore, the disclosed technology provides technical improvements in computer systems, applications, and / or techniques for compressing and / or executing machine learning models.

[0088] 1. In some embodiments, a computer-implemented method for compressing a machine learning model includes: determining a first sparsity rate by a Bayesian optimizer, the first sparsity rate being associated with a limit on the loss of accuracy due to compressing the machine learning model; selecting a second sparsity rate by the Bayesian optimizer, the second sparsity rate optimizing a predefined objective function for the machine learning model within a search space defined by the first sparsity rate; and generating a compressed version of the machine learning model having the second sparsity rate.

[0089] 2. The computer-implemented method of clause 1 further comprises: generating a different compressed version of the machine learning model for each iteration performed by the Bayesian optimizer to select a first sparsity rate and a second sparsity rate.

[0090] 3. A computer-implemented method according to any one of clauses 1-2, wherein generating the different compressed versions of the machine learning model includes: performing a learning step, wherein the learning step trains the machine learning model using a quadratic regularization term; and performing a compression step, wherein the compression step compresses the machine learning model to have a specified sparsity rate in a given iteration of the Bayesian optimizer.

[0091] 4. A computer-implemented method according to any of clauses 1-3, wherein the learning step and the compressing step are repeated until a stopping criterion is met.

[0092] 5. A computer-implemented method according to any of clauses 1-4, wherein the Bayesian optimizer performs one or more iterations that randomly samples the first sparsity rate and the second sparsity rate for the machine learning model.

[0093] 6. A computer-implemented method according to any of clauses 1-5, wherein a Bayesian optimizer performs one or more iterations, the one or more iterations selecting the first sparsity rate based on an implicit level set upper confidence bound (ILS-UCB) sampling criterion.

[0094] 7. A computer-implemented method according to any of clauses 1-6, wherein the Bayesian optimizer performs one or more iterations that select the second sparsity rate based on a confidence bound sampling criterion.

[0095] 8. The computer-implemented method of any of clauses 1-7, further comprising: receiving the predefined objective function from a user.

[0096] 9. A computer-implemented method according to any one of clauses 1-8, wherein the predefined objective function includes at least one of the following: an evaluation rate of the machine learning model, a memory occupancy, an evaluation time of the machine learning model, and a domain-specific objective.

[0097] 10. A computer-implemented method according to any of clauses 1-9, wherein a Bayesian optimizer selects the second sparsity rate based on a Gaussian process for the predefined objective function.

[0098] 11. A computer-implemented method according to any of clauses 1-10, wherein generating a compressed version of the machine learning model includes: quantizing the weights of the machine learning model and pruning at least one of a subset of the weights from the machine learning model.

[0099] 12. In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to: compress a machine learning model by performing steps in which a Bayesian optimizer determines a first sparsity rate associated with a limit on the loss of accuracy resulting from compressing the machine learning model; selects a second sparsity rate by the Bayesian optimizer that optimizes a predefined objective function for the machine learning model within a search space defined by the first sparsity rate; and generates a compressed version of the machine learning model having the second sparsity rate.

[0100] 13. A non-transitory computer-readable medium according to clause 12, wherein the step further includes: generating a different compressed version of the machine learning model for each iteration performed by the Bayesian optimizer to select a first sparsity rate and a second sparsity rate.

[0101] 14. A non-transitory computer-readable medium according to any of clauses 12-13, wherein generating different compressed versions of a machine learning model comprises: performing a learning step that trains the machine learning model using a quadratic regularization term; and performing a compression step that compresses the machine learning model to have a specified sparsity rate in a given iteration of the Bayesian optimizer.

[0102] 15. A non-transitory computer-readable medium according to any of clauses 12-14, wherein the Bayesian optimizer performs one or more iterations that randomly samples the first sparsity rate and the second sparsity rate for the machine learning model.

[0103] 16. A non-transitory computer-readable medium according to any of clauses 12-15, wherein a Bayesian optimizer performs a first series of iterations of selecting the first sparsity rate based on an implicit level set upper confidence bound (ILS-UCB) sampling criterion and performs a second series of iterations of selecting the second sparsity rate based on a confidence bound sampling criterion.

[0104] 17. The non-transitory computer-readable medium of any of clauses 12-16, wherein the steps further comprise: receiving a predefined objective function and a limit on the loss of precision from a user.

[0105] 18. A non-transitory computer-readable medium according to any of clauses 12-17, wherein the predetermined objective function comprises at least one of inference throughput, memory footprint, latency, and a domain-specific objective.

[0106] 19. A non-transitory computer-readable medium according to any of clauses 12-18, wherein generating a compressed version of the machine learning model includes: quantizing the weights of the machine learning model and pruning at least one of a subset of weights from the machine learning model.

[0107] 20. In some embodiments, a system includes: a memory storing instructions; a processor for executing the instructions, determining, by a Bayesian optimizer, a first sparsity rate associated with a limit on the loss of accuracy caused by compressing the machine learning model; selecting, by the Bayesian optimizer, a second sparsity rate that optimizes a predefined objective function for the machine learning model within a search space defined by the first sparsity rate; and generating a compressed version of the machine learning model having the second sparsity rate.

[0108] Any and all combinations of any claim elements recited in any claim and / or any elements described in this application, in any manner, are within the intended scope of the invention and protection.

[0109] The description of the various embodiments has been given for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0110] The various aspects of the present embodiment may be embodied as a system, method or computer program product. Therefore, aspects of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects, which are generally referred to herein as "module", "system" or "computer". In addition, any hardware and / or software technology, process, function, component, engine, module or system described in the present disclosure may be implemented as a circuit or circuit set. In addition, various aspects of the present invention may take the form of a computer program product, which is embodied in one or more computer-readable media, wherein there is a computer-readable program code.

[0111] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media will include the following: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any other suitable combination of the foregoing. In the context of this article, computer-readable storage media can be any tangible medium that can contain or store programs for use by or in conjunction with an instruction execution system, device or device.

[0112] Aspects of the present disclosure are described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It will be understood that each frame of the flowchart illustration and / or block diagram, and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. When the instruction is executed by the processor of a computer or other programmable data processing device, the function / action specified in the flowchart and / or block diagram frame or frame can be implemented. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor or a field programmable gate array.

[0113] The flow charts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram may represent a module, segment or part of a code, which includes one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative embodiments, the functions indicated in the box may not occur in the order indicated in the figure. For example, depending on the functions involved, the two boxes shown in succession can actually be executed substantially in parallel, or sometimes these boxes can be executed in reverse order. It should also be noted that each box of the block diagram and / or flow chart description and the combination of the boxes in the block diagram and / or flow chart description can be implemented by a dedicated hardware-based system that performs a specified function or action, or by a combination of dedicated hardware and computer instructions.

[0114] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof and the scope of the disclosure is determined by the claims that follow.

Claims

1. A computer-implemented method for a compressed machine learning model for image processing, the method comprising: determining, by a Bayesian optimizer, a first sparsity rate associated with a limit on the loss of accuracy resulting from compressing the machine learning model; selecting, by the Bayesian optimizer, a second sparsity rate that optimizes a predefined objective function for the machine learning model within a search space defined by the first sparsity rate; as well as generating a compressed version of the machine learning model having the second sparsity rate, The first sparsity rate and the second sparsity rate correspond to different compressed versions of the machine learning model respectively.

2. The computer-implemented method of claim 1 , further comprising: A different compressed version of the machine learning model is generated for each iteration performed by the Bayesian optimizer to select the first sparsity rate and the second sparsity rate.

3. The computer-implemented method of claim 2, wherein generating the different compressed versions of the machine learning model comprises: performing a learning step, wherein the learning step trains the machine learning model using a quadratic regularization term; as well as A compression step is performed, wherein the compression step compresses the machine learning model to have a specified sparsity rate in a given iteration of the Bayesian optimizer.

4. The computer-implemented method of claim 3, wherein the learning step and the compressing step are repeated until a stopping criterion is met.

5. The computer-implemented method of claim 2, wherein the Bayesian optimizer performs one or more iterations that randomly samples the first sparsity rate and the second sparsity rate for the machine learning model.

6. The computer-implemented method of claim 2, wherein the Bayesian optimizer performs one or more iterations that select the first sparsity rate based on an implicit level set upper confidence bound (ILS-UCB) sampling criterion.

7. The computer-implemented method of claim 2, wherein the Bayesian optimizer performs one or more iterations that select the second sparsity rate based on a confidence bound sampling criterion.

8. The computer-implemented method of claim 1 , further comprising: The predefined objective function is received from a user.

9. A computer-implemented method according to claim 1, wherein the predefined objective function includes at least one of the following: an evaluation rate of the machine learning model, a memory footprint, an evaluation time of the machine learning model, and a domain-specific goal.

10. The computer-implemented method of claim 1, wherein the Bayesian optimizer selects the second sparsity rate based on a Gaussian process prior for the predefined objective function.

11. The computer-implemented method of claim 1 , wherein generating the compressed version of the machine learning model comprises: At least one of quantizing weights of the machine learning model and pruning a subset of the weights from the machine learning model.

12. A non-transitory computer-readable medium for image processing storing instructions that, when executed by a processor, cause the processor to compress a machine learning model by performing the following steps: determining, by a Bayesian optimizer, a first sparsity rate associated with a limit on the loss of accuracy resulting from compressing the machine learning model; selecting, by the Bayesian optimizer, a second sparsity rate that optimizes a predefined objective function for the machine learning model within a search space defined by the first sparsity rate; as well as generating a compressed version of the machine learning model having the second sparsity rate; The first sparsity rate and the second sparsity rate correspond to different compressed versions of the machine learning model respectively.

13. The non-transitory computer readable medium of claim 12, wherein the steps further comprise: A different compressed version of the machine learning model is generated for each iteration performed by the Bayesian optimizer to select the first sparsity rate and the second sparsity rate.

14. The non-transitory computer-readable medium of claim 13, wherein generating the different compressed version of the machine learning model comprises: performing a learning step, wherein the learning step trains the machine learning model using a quadratic regularization term; as well as A compression step is performed, wherein the compression step compresses the machine learning model to have a specified sparsity rate in a given iteration of the Bayesian optimizer.

15. The non-transitory computer-readable medium of claim 13, wherein the Bayesian optimizer performs one or more iterations that randomly samples the first sparsity rate and the second sparsity rate for the machine learning model.

16. The non-transitory computer-readable medium of claim 13, wherein the Bayesian optimizer performs a first series of iterations of selecting the first sparsity rate based on an implicit level set upper confidence bound (ILS-UCB) sampling criterion and performs a second series of iterations of selecting the second sparsity rate based on a confidence bound sampling criterion.

17. The non-transitory computer readable medium of claim 12, wherein the steps further comprise: The predefined objective function and the limit on the loss of accuracy are received from a user.

18. The non-transitory computer-readable medium of claim 12, wherein the predefined objective function comprises at least one of inference throughput, memory footprint, latency, and a domain-specific objective.

19. The non-transitory computer-readable medium of claim 12, wherein generating the compressed version of the machine learning model comprises: At least one of quantizing weights of the machine learning model and pruning a subset of the weights from the machine learning model.

20. A system for image processing, comprising: a memory for storing instructions; as well as A processor that executes the following instructions: determining, by a Bayesian optimizer, a first sparsity rate associated with a limit on the loss of accuracy resulting from compressing the machine learning model; selecting, by the Bayesian optimizer, a second sparsity rate that optimizes a predefined objective function for the machine learning model within a search space defined by the first sparsity rate; as well as generating a compressed version of the machine learning model having the second sparsity rate; The first sparsity rate and the second sparsity rate correspond to different compressed versions of the machine learning model respectively.

Images