TECHNIQUES FOR COMPRESSING ARTIFICIAL NEURAL NETWORKS

Abstract

At least one of the various embodiments is directed to a computer-implemented method for generating trained artificial neural networks.The method includes, for each model layer included in a trained model, training one or more student model layers to mimic the model layer for a first target device included in a plurality of target devices, generating one or more candidate architectures based on a constrained optimization problem and the one or more trained student model layers, training the one or more candidate architectures on a set of calibration data, selecting a first candidate architecture included in the one or more candidate architectures that is associated with a lowest amount of error, and performing a plurality of fine-tuning training operations on the first candidate architecture to generate a first trained student model.

Description

BACKGROUND OF THE INVENTIONField of the InventionEmbodiments of the present disclosure generally relate to compression of large speech models, and more particularly to techniques for compressing artificial neural networks.Description of the Prior ArtA large language model (LLM) is a type of artificial neural network (ANN) that has demonstrated remarkable performance in a wide spectrum of natural language processing (NLP) tasks including text generation and classification. However, as the size and complexity of LLMs increases, the computational and storage costs and latencies associated with training and provisioning LLMs for various end-user applications also increase. These increasing costs and latencies may limit the overall effectiveness and usefulness of LLMs. Accordingly, various techniques have been developed to facilitate training and provisioning of LLMs.One approach to increasing the overall effectiveness of LLMs involves constructing a family of smaller LLMs rather than a single large LLM, with each smaller LLM tailored to execute on specified hardware and within specified time conditions. Neural architecture search (NAS) is a technique commonly used to construct groups of smaller LLMs that are subject to various model size, hardware, and memory conditions. A NAS algorithm is classified according to the three phases used to construct the model architecture: the search space, the search strategy, and the power estimation strategy. The search space defines the set of architectures that can be used to represent the various smaller LLMs, including the number of layers, the type of layers (e.g., multilayer perceptron, convolution, attention, etc.), and the number of parameters per layer (e.g., number of neurons). The search strategy is used to explore the search space, select a given architecture based on a variety of factors, and build the smaller LLMs based on the selected architecture. The power estimation strategy estimates how well the model architecture found in the search phase works with new data. A typical performance estimation strategy is to train the model found in the search phase with a training data set and to evaluate the performance with a validation data set.A disadvantage of using NAS to develop groups of smaller LLMs is the amount of time it takes the NAS algorithm to explore the search space and select an architecture for use for smaller LLMs, particularly when the search space is large. Reducing the search space may speed up execution, but if the search space is too small, the NAS algorithm is less likely to find an optimized architecture for use for the smaller LLMs. Another drawback of NAS is that the power estimation strategy normally requires that each smaller LLM be trained from reason to then evaluated. Training numerous LLMs, even smaller, from reason to reason can take quite a lot of time and consume large amounts of computing resources, which can make NAS impractical for many applications.Another approach to increasing the overall effectiveness of LLMs involves model compression, where a pre-trained LLM is compressed to produce a smaller LLM. Three common model compression techniques are pruning (cropping), quantization, and knowledge distillation. Pruning is the process of removing redundant parameters, such as neurons, from an existing model. Redundant parameters are typically considered parameters whose distance from a model minimally affects the output of the model. Pruning may be unstructured, removing individual parameters from a model, regardless of where these parameters are within the model, or structured, removing groups of parameters from particular locations within a model. In quantization, the various weights are represented within a model using a reduced number of bits. Using fewer bits for the weights reduces the amount of memory resources consumed by the model and also reduces the computational complexity of the operations performed using the model. Knowledge distillation uses a larger, pre-trained teacher model to train a smaller teacher model, and during training, the knowledge of the teacher model is transferred to the student model.A disadvantage of model compression is the compromise between the increase in model efficiency and the loss of model accuracy. While the above model compression techniques result in smaller models that can be executed more quickly, large amounts of compression can result in significant execution inaccuracies due to the amounts of information removed from the models. Moreover, model compression often requires that the hyperparameters of a model be tuned manually, which is a process that can be tedious, time consuming, and error prone.As the foregoing illustrates, more effective techniques are required in the prior art for compressing LLMs and other artificial neural networks.SUMMARYAt least one of the various embodiments is directed to a computer-implemented method for generating trained artificial neural networks. The method includes, for each model layer included in a trained model, training one or more student model layers to mimic the model layer for a first target device included in a plurality of target devices, generating one or more candidate architectures based on a constrained optimization problem and the one or more trained student model layers, training the one or more candidate architectures on a set of calibration data, selecting a first candidate architecture included in the one or more candidate architectures and associated with a least amount of error, and performing a plurality of fine tuning training operations on the first candidate architecture to generate a first trained student model.At least one technical advantage of the disclosed techniques over the prior art is that the disclosed techniques may significantly facilitate training and provision of LLMs across a number of different hardware implementations. In this regard, the disclosed techniques may be used to generate a different trained student LLM for each different hardware implementation based on a single trained LLM. Thus, with the disclosed techniques, multiple different smaller trained student LLMs can be generated for multiple different hardware implementations without having to fundamentally train each student LLM, thereby reducing the time and computational resources required to provide new trained models. Moreover, the student LLMs generated using the disclosed techniques may improve the latency and storage space requirements of the original trained LLM without substantial loss of accuracy. The disclosed techniques also implement constrained optimization problems to automate the design of the various student LLM architectures, thereby eliminating the need for manual hyperparameter fine tuning. These technical advantages provide one or more technological improvements over prior art approaches.BRIEF DESCRIPTION OF THE DRAWINGSIn order that the manner in which the above-mentioned features of the present invention are described may be understood in detail, a more detailed description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the accompanying drawings. It should be understood, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered as limiting the scope thereof, as the invention may permit other equally effective embodiments. FIG. 1 is a block diagram of a computer-based system configured to implement one or more aspects of the various embodiments; FIG. 2 is a more detailed illustration of the architecture of the trained LLM of FIG. 1, according to various embodiments; FIG. 3 is a block diagram of an LLM architecture optimizer of FIG. 1, according to various embodiments; FIG. 4 is a more detailed illustration of the candidate architecture generator of FIG. 3, according to various embodiments; FIG. 5 is a more detailed illustration of the student architecture selector of FIG. 3, in accordance with various embodiments; FIG. 6 is a flow diagram of method steps for generating candidate architectures according to various embodiments; and FIG. 7 is a flow diagram of method steps for generating student LLMs according to various embodiments.DETAILED DESCRIPTIONIn the following description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without one or more of these specific details.System OverviewFIG. 1 illustrates a block diagram of a computer-based system 100 configured to implement one or more aspects of the various embodiments. As shown, the system 100 includes, without limitation, a compression server 110, a data store 120, a network 130, and a computing device 140. The compression server 110 includes, without limitation, processor(s) 112 and system memory 114. System memory 114 includes, without limitation, an LLM architecture optimizer 116 and a trained LLM 118. Computing device 140 includes, without limitation, processor(s) 142 and memory 144. The memory 144 includes, without limitation, an application 145. The data store 120 includes, without limitation, a student LLM 122.The compression server 110 shown herein is for illustrative purposes only and variations and modifications are possible without departing from the scope of the present disclosure. For example, the number and types of processors 112, the number of GPUs and / or other processing unit types, the number and types of system memories 114, and / or the number of applications included in the system memory 114 may be modified as desired. Further, the connection topology between the various units in Figure 1 may be modified as desired. In some embodiments, any combination of processor(s) 112 and system memory 114 and / or GPU(s) may be included in and / or replaced by any type of virtual computing system, distributed computing system, and / or cloud computing environment, such as a public, private, or hybrid cloud system.Processor(s) 112 receive user input from input devices, such as a keyboard or a mouse. Processor(s) 112 may be any technically feasible form of processing device configured to process data and execute program code. For example, any of the processor(s) 112 could be a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and so forth. In various embodiments, any of the operations and / or functions described herein may be performed by processor(s) 112 or any combination of these various processors, such as a CPU operating in cooperation with one or more GPUs. In various embodiments, the one or more GPU(s) perform parallel processing tasks, such as matrix multiplication and / or the like in LLM model computations. Processor(s) 112 may also receive user input from input devices, such as a keyboard or mouse, and generate output on one or more displays.The system memory 114 of the compression server 110 stores content such as software applications and data for use by processor(s) 112. System memory 114 may be any type of memory capable of storing data and software applications, such as random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash ROM), or any suitable combination of the foregoing. In some embodiments, memory (not shown) may supplement or replace system memory 114. The memory may include any number and type of external memory accessible by processor(s) 112. For example, and without limitation, the memory may include a secure digital card, an external flash memory, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, and / or any suitable combination of the foregoing.The LLM architecture optimizer 116, stored in the system memory 114, is configured to generate student LLMs 122 by compressing and optimizing the architecture of the trained LLM 118. In particular, the LLM architecture optimizer 116 generates student LLMs 122 by replacing the layers of the trained LLM 118 with different compressed or optimized layers based on different target constraints, such as hardware and memory constraints or latency and number of parameters. The LLM architecture optimizer 116 then stores student LLMs 122 in the data store 120. The student LLMs 122 may then be used for inferencing operations in any suitable application, such as the application 145 executing on the computing device 140.The trained LLM 118 may be any type of technically feasible machine learning model. For example, in various embodiments, the trained LLM 118 may be a transformer-based LLM model, such as a generative pre-trained transformer (GPT)), with any suitable architecture. Likewise, the student LLMs 122 may be any type of technically feasible machine learning models. For example, in various embodiments, the student LLMs 122 may be transformer-based LLMs, such as a GPT, with any suitable architecture. The architecture of the trained LLM 118 will be described in more detail below in connection with FIG. 2. The operations performed by the LLM architecture optimizer 116 to generate the student LLMs 122 by compressing and optimizing the trained LLM 118 will be described in more detail below in connection with FIGS. 3-7.The data store 120 provides non-volatile storage for applications and data in the compression server 110 and the computing device 140. For example, and without limitation, training data, trained (or deployed) machine learning models, and / or application data, including the trained LLM 118 and the student LLM 122, may be stored in the data store 120. In some embodiments, the data store 120 may include hard or removable hard disk drives, flash memory devices and compact disc read-only memory (CD-ROM), DVD-ROM (digital versatile disc-ROM), Blu-ray, high definition DVD (HD-DVD), or other magnetic, optical, or solid state storage devices. The data store 120 may be a network attached storage (NAS) and / or a storage area network (SAN). Although shown coupled to the compression server 110 and the computing device 140 via the network 130, in various embodiments, the compression server 110 or the computing device 140 may include the data store 120.The network 130 includes any technically feasible type of communication network that enables the exchange of data between the compression server 110, the computing device 140, the data storage 120, and external entities or devices, such as a web server or other networked computing device. For example, the network 130 may include, but is not limited to, a wide area network (WAN), a local area network (LAN), a cellular network, a wireless network (WiFi), and / or the Internet.The computing device 140 shown herein is for illustrative purposes only and variations and modifications are possible without departing from the scope of the present disclosure. For example, the number and types of processors 142, the number and types of system memories 144, and / or the number of applications included in the system memory 144 may be modified as desired. Further, the connection topology between the various units in Figure 1 may be modified as desired. In some embodiments, any combination of processor(s) 142 and / or system memory 144 may be included in and / or replaced by any type of virtual computing system, distributed computing system, and / or cloud computing environment, such as a public, private, or hybrid cloud system.Processor(s) 142 receive user input from input devices, such as a keyboard or a mouse. Processor(s) 142 may be any technically feasible form of processing device configured to process data and execute program code. For example, any of the processor(s) 142 could be a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and so forth. In various embodiments, any of the operations and / or functions described herein may be performed by processor(s) 142, or any combination of these various processors, such as a CPU operating in cooperation with one or more GPUs. In various embodiments, the one or more GPU(s) perform parallel processing tasks, such as matrix multiplication and / or the like in LLM model computations. Processor(s) 142 may also receive user input from input devices, such as a keyboard or a mouse, and generate output on one or more displays.Similar to the memory 114 of the compression server 110, the memory 144 of the computing device 140 stores content, such as software applications and data, for use by processor(s) 142. System memory 144 may be any type of memory capable of storing data and software applications, such as RAM, ROM, EPROM, flash ROM, or any suitable combination of the foregoing. In some embodiments, memory (not shown) may supplement or replace system memory 144. The memory may include any number and type of external memory that can be accessed by the processor 142. For example, and without limitation, the memory may include a secure digital card, an external flash memory, a portable CD-ROM, an optical storage device, a magnetic storage device, and / or any suitable combination of the foregoing.To perform inference operations, the application 145 stored in the memory 144 accesses the student LLM 122 from the data store 120. Application 145 then presents input data to student LLM 122 to generate output data.FIG. 2 is a more detailed illustration of the architecture of the trained LLM 118 of FIG. 1, in accordance with various embodiments. As shown, the trained LLM 118 includes, without limitation, an embedding layer 210, multiple layers 215( 1)-(N), and a softmax layer 220. During execution, the input dataset 205 is input to the trained LLM 118, with the embedding layer 210, the layers 215( 1)-(N), and the softmax layer 220 then performing various operations to generate the LLM output 225.In various embodiments, the trained LLM 118 includes a transformer-based LLM configured to process the input dataset 205. In various embodiments, the input dataset 205 may be text data, such as words or phrases, or image or video data. More generally, the input data set 205 may include any technically possible data that may be processed by a transformer-based language model. After receiving the input data set 205, the embedding layer 210 converts the elements of the input data set 205 into numerical representations referred to as tokens and encodes each token as a vector. The vectors generated by the embedding layer 210 then pass through multiple layers 215( 1)-(N). Each layer 215 may comprise an attention layer or a multilayer perceptron (MLP)) layer with varying numbers of internal parameters including, without limitation, numbers of attention heads, key value projection dimensions, numbers of neurons, or types of activation functions. In various embodiments, each layer 215 may include a layer norm layer, a linear layer, a convolutional layer, a pooling layer, or any other type of viable layer of an artificial neural network. Each layer 215 generates a vector or matrix as a result of applying weight matrices and an activation function to the vector or matrix output of its preceding layer. The softmax layer 220 normalizes the output vector of the layer 215 (N) to a probability distribution of predicted results and generates the LLM output 225. In some embodiments where the goal of the trained LLM 118 is question answer, prediction of the next word / sentence, word / sentence translation, or image generation, the LLM output 225 may be the probability distribution of the next word / sentence coming / after the input word / sentence, translation of the input word / sentence, answer to the question input, or images generated in response to image and sub-title input.Generating Large Student Speech ModelsFIG. 3 is a more detailed illustration of the LLM architecture optimizer 116 of FIG. 1, according to various embodiments. As shown, the LLM architecture optimizer 116 includes, without limitation, an operator database 310, a knowledge distillation engine 320, a candidate architecture generator 330, and a candidate architecture selector 340. In operation, the LLM architecture optimizer 116 receives the trained LLM 118 and control parameters 302 from a user via a user interface (not shown) and generates student LLMs 122. The control parameters 302 include, without limitation, student layers 304 and target devices 306. The student layers 304 represent a list of candidate layers for student LLMs 122. In various embodiments, the student layers 304 may include a copy of each layer 215 of the trained LLM 118, cropped versions of each layer 215 of the trained LLM 118, quantized versions of each layer 215 of the trained LLM 118, identity layers, attention layers, or MPL layers. The target devices 306 are a list of devices on which student LLMs 122 may be deployed. The target devices 306 may include, without limitation, a server machine, a desktop machine, a laptop computer, a cellular phone or GPU, or another type of processor.Operator database 310 receives as input student layers 304 and target devices 306 from control parameters 302. The operator database 310 then generates a look-up table of performance metrics for each student layer 304 operating on any target device 306 and an associated target provisioning configuration, such as the hardware characteristics of the target device 306 and the usage regime associated with the target device 306. The performance metrics included in the look-up table may include, without limitation, processing latency, processing throughput, and memory space requirements. Processing latency typically measures the total time each student layer 304 takes to generate output based on a given input request. Processing latency may be measured in multiple phases, including the time each student layer 304 takes to process the input tokens, referred to as a pre-fill phase, and the time each student layer 304 takes to generate output tokens, referred to as a decode phase. Processing throughput typically measures the number of tokens each student layer 304 can process or generate in a particular time period. The storage space requirement typically measures the amount of storage required to store the parameters of each student layer 304. The performance metrics included in the operator database lookup table 310 may then be used by the candidate architecture generator 330 to generate one or more constraint equations of a constrained optimization problem, as described below.For a given layer 215 of the trained LLM 118, a knowledge distillation engine 320 receives one or more student layers 304 having the same input and output dimensions as the given layer 215 of the trained LLM 118. The knowledge distillation engine 320 then trains these one or more student layers 304 to mimic the given layer 215 of the trained LLM 118. The knowledge distillation engine 320 may use any feasible training technique to train the student layers 304, such as stochastic gradient descent with backpropagation, adaptive moment estimation (Adam), or root mean squared propagation (RMSprop). During training, the knowledge distillation engine 320 first calculates the layer-by-layer loss between the given layer 215 of the trained LLM 118 and the one or more student layers 304, referred to as operator score, according to equation (1): where X is a set of training samples, t is i the operation of the given layer 215 of the trained LLM 118, x is i the input to the given layer 215 of the trained LLM 118, { IS si,j} THE SET OF OPERATIONS OF THE ONE OR MORE STUDENT LAYERS 304, W={W IS i,j} THE SET OF WEIGHTS, which correspond to the set of operations of the one or more student layers 304, and is loss function. Examples of suitable loss functions include, without limitation, L1 norm, mean square error (MSE), and normalized MSE. The knowledge distillation engine 320 then updates the set of weights W of the one or more student layers 304 according to the training technique to mimic the operations of the given layer 215 of the trained LLM 118. The knowledge distillation engine 320 repeats the process for each of the remaining layers 215 of the trained LLM 118.The candidate architecture generator 330 receives the operator database 310, the operator scores and weights of the student layers 304 trained by the knowledge distillation engine 320, the trained LLM 118, and the target devices 306 from the control parameters 302. For a given device from the target devices 306, the candidate architecture generator 330 uses conditional optimization to determine a set of one or more candidate architectures for the student LLM 120. The operations of the candidate architecture generator 330 are described in more detail below in connection with FIG. 4.The candidate architecture selector 340 receives the various sets of candidate architectures generated by the candidate architecture generator 330. The candidate architecture selector 340 then trains each received candidate architecture to different calibration data. The calibration data may include, without limitation, the original training data used to train the trained LLM 118, a subset of the original training data used to train the trained LLM 118, or any other data not presented to the trained LLM 118 during training. After training all the various sets of candidate architectures received from the candidate architecture generator 330, the candidate architecture selector 340 selects, as the student LLM 122, the candidate architecture with the least error between the predicted outputs and the true outputs on the calibration dataset. The operations of the candidate architecture selector 340 are described in more detail below in connection with FIG. 5.FIG. 4 is a more detailed illustration of the candidate architecture generator 330 of FIG. 3, in accordance with various embodiments. As shown, the candidate architecture generator 330 includes, without limitation, the constrained optimization shaper 410 and the integer linear programming solver 415. As noted above, in operation, the candidate architecture generator 330 receives the operator database 310, the operator scores and weights of the student layers 304 trained by the knowledge distillation engine 320, the trained LLM 118, and the target devices 306 from the control parameters 302, and generates the various candidate architectures 420( 1)-(K). More specifically, the target devices 306, operator database 310, and operator scores and weights of the student layers 304 trained by the knowledge distillation engine 320 are input to the constrained optimization shaper 410. The constrained optimization shaper 410 then defines a constrained optimization problem to be solved by the integer linear programming solver 415, where the constrained optimization problem includes, without limitation, a target function to be minimized with respect to particular variables and one or more constraint equations that define conditions for those particular variables. In this regard, for a given target device 306, the constrained optimization shaper 410 formulates the target function included in the optimization problem as a loss function that estimates the error between the output of a candidate architecture 420 and the true value for a given dataset. In various embodiments, the loss function may be determined according to equation (2): where W is the set of weights of the student layers 304 trained by the knowledge distillation engine 320, Z={z i} is a set of binary vectors such that z i is a one-hot vector representing the choice of the student layer 304, C(x; Z, W) is the candidate architecture defined by Z and W, and X is a dataset whose elements are the designated pairs (x, y). The constrained optimization shaper 410 then uses the operator database lookup table 310 to determine at least one constraint equation according to equation (3): wherein a function defining the power budget for a given target device 306, p i is a vector whose components are the performance metrics for each student layer 304 corresponding to the given target device 306 included in the operator database lookup table 310, and is a user-defined constant.Constrained optimization shaper 410 passes equations (2)-(3) to integer linear programming solver 415. The integer linear programming solver 415, in turn, first approximates the loss function in equation (2) using a linear function according to equation (4): where T(x)=t N ◦ t N-1 ◦... ◦ t 1( x) is the set of operations performed by the trained LLM 118 in response to an input x, is a constant representing the error between the output of the trained LLM 118 and the true value y for the dataset X, and e i is a vector, wherein each component is the difference between the output of the trained LLM 118 for a small designated dataset and the output of the trained LLM 118 for the same small designated dataset when a given student layer 304 included in the student layers 304 replaces one of the layers 215 of the trained LLM 118 and all other layers 215 of the trained LLM 118 are unchanged (which is repeated across all different student layers 304 to obtain all different components of the vector e i). Integer linear programming solver 415 then generates a candidate architecture 420 by solving the following integer linear minimization problem given by equations (5)-(6):The integer linear programming solver 415 may use any feasible integer linear optimization technique to solve equations (5)-(6), such as a cut-plane algorithm (cutting plane algorithm) or a branch and bound algorithm (branch and bound algorithm). Integer linear programming solver 415 generates a set of candidate architectures 420 by solving the following linear minimization problem given by equations (7)-(9): where equation (9) acts as a limitation on maximum overlap with any other solution for equations (7)-(8). Upon completion of these operations, the integer linear programming solver 415 forwards candidate architectures 420 to the candidate architecture selector 340.FIG. 5 is a more detailed illustration of the candidate architecture selector 340 of FIG. 3, in accordance with various embodiments. As shown, candidate architecture selector 340 includes, without limitation, candidate architecture trainer 510, selected candidate architectures 520, and fine tuning selector 530. In operation, the candidate architecture selector 340 receives candidate architectures 420 from the candidate architecture generator 330 and generates student LLMs 122. In this regard, different sets of candidate architectures 420 are first input to candidate architecture trainer 510, which trains the different sets of candidate architectures 420 with a calibration dataset. The calibration data may include, without limitation, the original training data used to train the trained LLM 118, a subset of the original training data used to train the trained LLM 118, or any other data not presented to the trained LLM 118 during training. From each set of candidate architectures 420, the candidate architecture trainer 510 then selects the candidate architecture 420 with the least error between the predicted output and the true output for the calibration dataset as the selected candidate architecture 520. Each selected candidate architecture 520 is then input to fine tuner 530 to undergo further training. Fine tuner 530 trains each selected candidate architecture 520 with the same dataset used to train trained LLM 118 and uses the same learning rate plan used when trained LLM 118. The learning rate plan is a technique that adjusts the learning rate between iterations during the training process. The technique implemented by the learning rate plan may include, without limitation, a step-wise decay, an exponential decay, and a cosine annealing. After training, fine tuner 530 outputs student LLMs 122.FIG. 6 is a flow diagram of method steps for generating candidate architectures, according to various embodiments. Although the method steps are described in connection with the systems of FIGS. 1-5, those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the various embodiments.As shown, a method 600 begins at step 602, where the LLM architecture optimizer 116 receives the control parameters 302 from a user via a user interface (not shown). Examples of various control parameters 302 that may be input by the user include, without limitation, student layers 304 and target devices 306.At step 604, the LLM architecture optimizer 116 generates an operator database from the received control parameters 302. More specifically, the control parameters 302 are input to the operator database 310. The operator database 310 then generates a look-up table of performance metrics for each student layer 304 that is included in the control parameters 302 and that operates on any target device 306 included in the control parameters 302 in conjunction with an associated target provisioning configuration. Target provisioning configurations may include, without limitation, the hardware characteristics of the target device 306 and the usage regime associated with the target device 306. The performance metrics included in the look-up table may include, without limitation, processing latency, processing throughput, and memory space requirements.At step 606, the LLM architecture optimizer 116 receives the trained LLM model 118, which may be any type of machine learning model. For example, in various embodiments, the trained LLM 118 may be a transformer-based LLM, such as a GPT, with any suitable architecture. The LLM architecture optimizer 116 may receive the trained LLM 118 from any storage device, such as the data store 120.At step 608, the knowledge distillation engine 320 trains, for each given layer 215 of the trained LLM 118, the student layers 304 included in the control parameters 302 to mimic the given layer 215 of the trained LLM 118. The knowledge distillation engine 320 trains the various student layers 304 by calculating the layer-by-layer loss between the given layer 215 of the trained LLM 118 and the various student layers 304. The knowledge distillation engine 320 may use any loss function during these training operations, such as L 1- Norm, MSE, and normalized MSE. Likewise, the knowledge distillation engine 320 may use any feasible training technique to train the student layers 304, such as stochastic gradient descent with backpropagation, adam, or rmsprop.At step 610, for each target device 306 included in the control parameters 302, the constrained optimization shaper 410 generates a constrained optimization problem using the operator database 310 and the trained student layers generated by the knowledge distillation engine 320. The constrained optimization problem includes, without limitation, a target function to be minimized with respect to particular variables and one or more constraint equations that set conditions for those particular variables. The constrained optimization shaper 410 uses the trained student layers generated by the knowledge distillation engine 320 to generate the objective function of the constrained optimization problem and uses the performance metrics included in the operator database lookup table 310 to generate the one or more constrained equations of the constrained optimization problem.At step 612, for each constrained optimization problem generated by the constrained optimization shaper 410, the integer linear programming solver 415 approximates the objective function included in the constrained optimization problem as a linear function to generate a linear constrained optimization problem.At step 614, the integer linear programming solver 415 solves the various linear constrained optimization problems to generate candidate architectures 420. The integer linear programming solver 415 may use any feasible integer linear optimization technique to solve the linear constrained optimization problems, such as a slice plane algorithm or a branch and constraint algorithm.FIG. 7 is a flow diagram of method steps for generating student LLMs according to various embodiments. Although the method steps are described in connection with the systems of FIGS. 1-5, those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the various embodiments.As shown, a method 700 begins at step 702, where the candidate architecture selector 340 receives various sets of candidate architectures 420 from the candidate architecture generator 330.At step 704, the candidate architecture selector 340 trains each of the sets of candidate architectures 420 with calibration data. The calibration data may include, without limitation, the original training data used to train the trained LLM 118, a subset of the original training data used to train the trained LLM 118, or any other data not presented to the trained LLM 118 during training.At step 706, the candidate architecture trainer 510 selects, for each of the sets of candidate architectures 420, the candidate architecture having the least error as a selected candidate architecture 520. As described hereinabove, in various embodiments, the candidate architecture trainer 510 selects the candidate architecture 420 having the least error between the predicted output and the true output for the calibration dataset as the selected candidate architecture 520.At step 708, the fine tuner 530 trains the selected candidate architectures 520 to generate the student LLMs 122. Fine tuner 530 trains each selected candidate architecture 520 with the same dataset used to train trained LLM 118 and uses the same learning rate plan used when trained LLM 118. The fine tuning can use any feasible learning rate plan, such as a step-wise decay, an exponential decay, and a cosine annealing during this fine tuning phase. At step 710, the candidate architecture selector 340 outputs the student LLMs 122.Overall, the architecture of a trained LLM is optimized for execution on specified hardware and compressed based on target constraints such as latency and number of parameters to construct a smaller "student" LLM. First, a database of potential layer types, referred to as student layers, is constructed for the student LLM. Next, an operator score for each student layer of the student LLM is calculated based on how well that student layer mimics the corresponding layer in the original trained LLM. The operator scores and hardware-specific latency and memory constraints are used to construct a set of candidate architectures for the student LLM. Each candidate architecture is then trained with a calibration dataset and the minimum loss-of-accuracy architecture, when executed, is selected for deployment and then fine-tuned. The result is a smaller LLM that has improved execution latencies and reduced memory space requirements with minimized loss of accuracy with respect to the original trained LLM.At least one technical advantage of the disclosed techniques over the prior art is that the disclosed techniques may significantly facilitate training and provision of LLMs across a number of different hardware implementations. In this regard, the disclosed techniques may be used to generate a different trained student LLM for each different hardware implementation based on a single trained LLM. Thus, with the disclosed techniques, multiple different smaller trained student LLMs can be generated for multiple different hardware implementations without having to fundamentally train each student LLM, thereby reducing the time and computational resources required to provide new trained models. Moreover, the student LLMs generated using the disclosed techniques may improve the latency and storage space requirements of the original trained LLM without substantial loss of accuracy. The disclosed techniques also implement constrained optimization problems to automate the design of the various student LLM architectures, thereby eliminating the need for manual hyperparameter fine tuning. These technical advantages provide one or more technological improvements over prior art approaches.1. Some embodiments are directed to a computer-implemented method for generating trained artificial neural networks, the method comprising:for each model layer included in a trained model, training one or more student model layers to mimic the model layer;generating, for a first target device included in a plurality of target devices, one or more candidate architectures based on a constrained optimization problem and the one or more trained student model layers;training the one or more candidate architectures with a set of calibration data;selecting a first candidate architecture that is included in the one or more candidate architectures and that is associated with a least amount of error; andperforming a plurality of fine tuning training operations on the first candidate architecture to generate a first trained student model.2. The computer-implemented method of Set 1, wherein generating the one or more candidate architectures comprises generating a linear constrained optimization problem based on a objective function included in the constrained optimization problem and calculating a solution for the linear constrained optimization problem to generate at least one of the one or more candidate architectures.3. The computer-implemented method of set 1 or 2, wherein the linear constrained optimization problem includes a linear function comprising an approximation of the objective function included in the constrained optimization problem.4. The computer-implemented method of any of Sets 1-3, wherein the one or more student model layers and the plurality of target devices comprise user-defined control parameters.5. The computer-implemented method of any of sets 1-4, wherein the first candidate architecture has fewer errors between a predicted output and a true output generated using the set of calibration data than any other candidate architecture included in the one or more candidate architectures.6. The computer-implemented method of any of sets 1-5, wherein performing the plurality of fine tuning operations on the first candidate architecture comprises training the first candidate architecture with a dataset used to train the trained model.7. The computer-implemented method of any of Sets 1-6, wherein a learning rate plan used when the trained model is trained is implemented when the plurality of fine tuning operations are performed on the first candidate architecture.8. The computer-implemented method of any of Sets 1-7, wherein the one or more student layers include a copy of each layer included in the trained model, a cropped version of each layer included in the trained model, at least one identity layer, at least one attention layer, or at least one multi-layer perceptron layer.9. The computer-implemented method of any of Sets 1-8, wherein the one or more student model layers and the model layer have the same input dimensions and the same output dimensions.10. The computer-implemented method of any of sets 1-9, wherein the first trained student model has a lower execution latency relative to an execution latency associated with the trained model.11. Some other embodiments are directed to one or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:for each model layer included in a trained model, training one or more student model layers to mimic the model layer;generating, for a first target device included in a plurality of target devices, one or more candidate architectures based on a constrained optimization problem and the one or more trained student model layers;training the one or more candidate architectures with a set of calibration data;selecting a first candidate architecture that is included in the one or more candidate architectures and that is associated with a least amount of error; andperforming a plurality of fine tuning training operations on the first candidate architecture to generate a first trained student model.12. The one or more non-transitory computer readable media of Set 11, wherein the plurality of target devices includes at least one of a server machine, a desktop machine, a graphics processing unit, a laptop computer, or a mobile phone.13. The one or more non-transitory computer readable media of Set 11 or 12, wherein the first trained student model has a storage space requirement that is less than a storage space requirement associated with the trained model.14. The one or more non-transitory computer readable media of any of sets 11 - 13, wherein generating the one or more candidate architectures comprises generating a linear constrained optimization problem based on a objective function included in the constrained optimization problem and calculating a solution for the linear constrained optimization problem to generate at least one of the one or more candidate architectures.15. The one or more non-transitory computer readable media of any of sets 11 - 14, wherein the linear constrained optimization problem includes a linear function comprising an approximation of the objective function included in the constrained optimization problem.16. The one or more non-transitory computer readable media of any of sets 11 - 15, wherein the one or more student model layers and the plurality of target devices comprise user-defined control parameters.17. The one or more non-transitory computer readable media of any of sets 11 - 16, wherein the first candidate architecture has fewer errors between a predicted output and a true output generated using the set of calibration data than any other candidate architecture included in the one or more candidate architectures.18. The one or more non-transitory computer readable media of any of sets 11 - 17, wherein performing the plurality of fine tuning operations on the first candidate architecture comprises training the first candidate architecture with a dataset used to train the trained model.19. The one or more non-transitory computer readable media of any of sets 11 - 18, wherein a learning rate plan used when the trained model is trained is implemented when the plurality of fine tuning operations are performed on the first candidate architecture.20. Some embodiments are directed to a computer system comprising:one or more memories including instructions; andone or more processors coupled to the one or more memories and configured, when executing the instructions, to perform the steps of:for each model layer included in a trained model, training one or more student model layers to mimic the model layer;generating, for a first target device included in a plurality of target devices, one or more candidate architectures based on a constrained optimization problem and the one or more trained student model layers;training the one or more candidate architectures with a set of calibration data;selecting a first candidate architecture that is included in the one or more candidate architectures and that is associated with a least amount of error; andperforming a plurality of fine tuning training operations on the first candidate architecture to generate a first trained student model. All combinations of all claim elements recited in any one of the claims and / or all elements described in this application are in any way within the scope of the present invention of protection.Any and all combinations of any of the claim elements recited in any of the claims and / or any elements described in this application fall within the scope of the present invention and scope contemplated in any manner.The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.Aspects of the present embodiments may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of a full hardware embodiment, a full software embodiment (including firmware, resident software, microcode, etc.), or an embodiment that combines software and hardware aspects, all of which may be generally referred to herein as a "module", a "system", or a "computer". Moreover, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.Any combination of one or more computer readable medium(s) may be used. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.Aspects of the present disclosure are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowchart illustrations and / or the block diagrams, and combinations of blocks in the flowchart illustrations and / or the block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine. The instructions, when executed via the processor of the computer or other programmable data processing device, enable the functions / steps specified in the flowchart and / or block diagram block or blocks to be implemented. Such processors may include, without limitation, general purpose processors, special purpose processors, application specific processors, or field programmable gate arrays.The flowcharts and the block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions indicated in the block may occur out of the order indicated in the figures. For example, two blocks shown in succession may, in fact, be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It is also noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, may be implemented by special purpose hardware-based systems that perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is defined by the following claims.

Claims

A computer-implemented method for generating trained artificial neural networks, the method comprising: for each model layer included in a trained model, training one or more student model layers to mimic the model layer; for a first target device included in a plurality of target devices, generating one or more candidate architectures based on a constrained optimization problem and the one or more trained student model layers; training the one or more candidate architectures with a set of calibration data; selecting a first candidate architecture included in the one or more candidate architectures and associated with a least amount of error; and performing a plurality of fine tuning training operations on the first candidate architecture to generate a first trained student model.The computer-implemented method of claim 1, wherein generating the one or more candidate architectures comprises generating a linear constrained optimization problem based on a objective function included in the constrained optimization problem and calculating a solution for the linear constrained optimization problem to generate at least one of the one or more candidate architectures.The computer-implemented method of claim 2, wherein the linear constrained optimization problem includes a linear function comprising an approximation of the objective function included in the constrained optimization problem.The computer-implemented method of any preceding claim, wherein the one or more student model layers and the plurality of target devices comprise user-defined control parameters.The computer-implemented method of any preceding claim, wherein the first candidate architecture has fewer errors between a predicted output and a true output generated using the set of calibration data than any other candidate architecture included in the one or more candidate architectures.The computer-implemented method of any preceding claim, wherein performing the plurality of fine tuning operations on the first candidate architecture comprises training the first candidate architecture with a dataset used to train the trained model.The computer-implemented method of claim 6, wherein a learning rate plan used when the trained model is trained is implemented when the plurality of fine tuning operations are performed on the first candidate architecture.The computer-implemented method of any preceding claim, wherein the one or more student layers include a copy of each layer included in the trained model, a cropped version of each layer included in the trained model, at least one identity layer, at least one attention layer, or at least one multi-layer perceptron layer.The computer-implemented method of any preceding claim, wherein the one or more student model layers and the model layer have the same input dimensions and the same output dimensions.The computer-implemented method of any preceding claim, wherein the first trained student model has a lower execution latency relative to an execution latency associated with the trained model.One or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: for each model layer included in a trained model, training one or more student model layers to mimic the model layer; for a first target device included in a plurality of target devices, generating one or more candidate architectures based on a constrained optimization problem and the one or more trained student model layers; training the one or more candidate architectures with a set of calibration data; selecting a first candidate architecture included in the one or more candidate architectures and associated with a least amount of error; and performing a plurality of fine tuning training operations on the first candidate architecture to generate a first trained student model.The one or more non-transitory computer readable media of claim 11, wherein the plurality of target devices includes at least one of a server machine, a desktop machine, a graphics processing unit, a laptop computer, or a mobile phone.The one or more non-transitory computer readable media of claim 11 or 12, wherein the first trained student model has a storage space requirement that is less than a storage space requirement associated with the trained model.The one or more non-transitory computer readable media of any of claims 11 to 13, wherein generating the one or more candidate architectures comprises generating a linear constrained optimization problem based on a objective function included in the constrained optimization problem and calculating a solution for the linear constrained optimization problem to generate at least one of the one or more candidate architectures.The one or more non-transitory computer readable media of claim 14, wherein the linear constrained optimization problem includes a linear function comprising an approximation of the objective function included in the constrained optimization problem.The one or more non-transitory computer readable media of any of claims 11 to 15, wherein the one or more student model layers and the plurality of target devices comprise user-defined control parameters.The one or more non-transitory computer readable media of any of claims 11 to 16, wherein the first candidate architecture has fewer errors between a predicted output and a true output generated using the set of calibration data than any other candidate architecture included in the one or more candidate architectures.The one or more non-transitory computer readable media of any of claims 11 to 17, wherein performing the plurality of fine tuning operations on the first candidate architecture comprises training the first candidate architecture with a dataset used to train the trained model.The one or more non-transitory computer readable media of claim 18, wherein a learning rate plan used when the trained model is trained is implemented when the plurality of fine tuning operations are performed on the first candidate architecture.A computer system comprising: one or more memories including instructions; and one or more processors coupled to the one or more memories and configured, when executing the instructions, to perform the steps of: for each model layer included in a trained model, training one or more student model layers to mimic the model layer; for a first target device included in a plurality of target devices, generating one or more candidate architectures based on a constrained optimization problem and the one or more trained student model layers; training the one or more candidate architectures with a set of calibration data; selecting a first candidate architecture included in the one or more candidate architectures and associated with a least amount of error; and performing a plurality of fine tuning training operations on the first candidate architecture to generate a first trained student model.

Images