System and method of processing inference requests using a plurality of versions of a deep neural network quantized with different precisions

Abstract

A method of processing inference requests in a deep neural network is provided. The method includes storing, in a memory (102), a plurality of quantized versions (104A-N) of a deep neural network model. Each of the plurality of quantized versions of the model has a specific level of precision (106A-N). The method includes monitoring an inference request queue. The method includes selecting one of the plurality of quantized versions in accordance with the results of the monitoring step.

Description

[0001] SYSTEM AND METHOD OF PROCESSING INFERENCE REQUESTS IN A DEEP

[0002] NEURAL NETWORK

[0003] TECHNICAL FIELD

[0004] The disclosure relates generally to quantization in Deep Neural Network (DNN) models, and more particularly, the disclosure relates to a method of processing inference requests in a deep neural network. Moreover, the disclosure relates to a system of processing inference requests in the deep neural network.

[0005] BACKGROUND

[0006] Quantization is a model size reduction technique that converts Deep Neural Network (DNN) model weights from high-precision floating-point (FP) representations (e.g., FP16 (16-bit), FP32 (32-bit) or FP64 (64-bit)) to lower-precision floating-point (FP) or integer (INT) representations, such as INT8 (8-bit) or INT4 (4-bit). A major challenge in quantization includes the limited dynamic range of these representations, which can result in accuracy loss when converting from higher-precision floating-point representations.

[0007] During quantization, the high dynamic range of FP32 or FP16 needs to be compressed / squeezed into just 255 values of INT8 or even 15 values of INT4. This compression, typically done through a process called calibration, is essential for maintaining the model (e.g., the DNN model) performance.

[0008] The primary advantage of quantization is the reduction in memory and computation required to run the model by decreasing the precision of the model’s parameters and activations. Additionally, some types of quantization enable calculations to be performed in integers, leading to improved bandwidth and further performance gains. For example, quantizing a subset of model parameters from FP32 to INT8 can reduce the required memory space and computation time by a factor of four.

[0009] Quantization methods include a Post-Training Quantization (PTQ) method and a Quantization- Aware Training (QAT) method. The PTQ method involves converting the weights of an already trained model to a lower precision FP or INT representation without any retraining. While the PTQ method is straightforward and easy to implement, it may cause a slight degradation in model performance due to the loss of precision in the weights. The QAT method integrates the weight conversion process during the training stage. Although the QAT method is more computationally demanding, it often results in superior model performance by maintaining higher accuracy despite the reduced precision.

[0010] Quantization types include a naive quantization, a hybrid quantization, and a selective quantization. In the naive quantization, all operators of the deep neural network model are quantized to the same precision using the same data. In the hybrid quantization, some operators of the deep neural network model are quantized to a certain precision (e.g., INT8), while other operators are left in their original data type / precision. This approach requires prior knowledge of the structure of the deep neural network model and its quantization-sensitive layers. In the selective quantization, specific operators of the deep neural network model are quantized to INT8 precision using different calibration data and granularity (per channel or per tensor). For example, residuals are quantized to INT8, while sensitive or non-friendly layers remain in FP16 precision.

[0011] A post-training static quantization method quantizes both the weights and activations of the model ahead of time, before the model is deployed for inference. In a post-training dynamic quantization approach, the weights are quantized ahead of time, but the activations are dynamically quantized during inference. While dynamic quantization can offer flexibility, it can be slightly slower than the static quantization due to the overhead of computing the range for activations during each inference.

[0012] In both static and dynamic post-training quantization, once the quantized model is deployed for inference, the quantized weights maintain their precision level and do not adapt to lower or higher precision. Consequently, the accuracy loss remains constant and does not consider current system pressure.

[0013] In a Large Language Model (LLM) inference system with limited hardware capabilities, such as small High Bandwidth Memory (HBM) and computing power, the system is constrained by a number of inference requests that it can handle per run. Implementing a quantization method can reduce hardware requirements which may improve inference throughput and latency, but this often results in accuracy loss. The LLM inference system typically deploys a quantized model that lacks the ability to adapt to the number of inference requests in the queue. Once it is quantized, the model’s cost and accuracy loss remain static, without consideration for current system demands. Thus, there remains a need to balance inference throughput and accuracy, and to optimize both aspects as conditions change.

[0014] Therefore, there arises a need to address the aforementioned technical problems / drawbacks in processing inference requests in a deep neural network.

[0015] SUMMARY

[0016] It is an object of the disclosure to provide a method of processing inference requests in a deep neural network. Moreover, the disclosure relates to a system of processing inference requests in the deep neural network.

[0017] This object is achieved by the features of the independent claims. Further, implementation forms are apparent from the dependent claims, the description, and the figures.

[0018] The disclosure provides a method of processing inference requests in a deep neural network. Moreover, the disclosure relates to a system of processing inference requests in the deep neural network.

[0019] According to a first aspect, there is provided a method of processing inference requests in a deep neural network. The method includes storing in memory a plurality of quantized versions of a deep neural network model. Each of the plurality of quantized versions of the model has a specific level of precision. The method includes monitoring an inference request queue. The method includes selecting one of the plurality of quantized versions in accordance with the results of the monitoring step.

[0020] The method enables the dynamic switching between model precisions based on the inference request / system load. The method enables faster inference by reducing computation requirements, increases throughput by minimizing model size, and minimizes accuracy loss by adjusting precision in underutilized systems. Additionally, the method's ability to dynamically switch between sub-model precisions based on the inference request / system load optimizes resource utilization by enhancing efficiency and conserving memory and computational resources, thereby eliminating the need to load the entire model. The method also provides different levels of accuracy to users based on the availability of system resources, potentially supporting a pricing mechanism that reflects the varying accuracy levels, indicating the deployment of this advanced technology.

[0021] Preferably, the selecting step selects a first quantized version of the model having a first level of precision when the result of the monitoring step is that the inference request queue has a first capacity level of queued requests and selects a second quantized version of the model having a second level of precision when the result of the monitoring step is that the inference request queue has a second capacity level of queued requests. Preferably, the first level of precision is higher than the second level of precision when the first capacity level is lower than the second capacity level. Preferably, the first level of precision is lower than the second level of precision when the first capacity level is higher than the second capacity level.

[0022] Preferably, the method further comprises creating the plurality of quantized versions of the deep neural network model before storing the plurality of quantized versions of the deep neural network model in the memory. Optionally, the plurality of quantized versions are created by selecting a first subset of a plurality of layers of the deep neural network model and quantizing only the first subset. Optionally, the plurality of quantized versions are created by selecting a second subset of the plurality of layers of the deep neural network model and quantizing only the second subset. The first subset and the second subset may be quantized using a separate level of quantization precision.

[0023] Optionally, the quantization involves post training quantization. The model may be a large language model. Preferably, the processing is carried out using a Graphical Processing Unit. Preferably, the quantization quantizes the weights and activations of the model. The specific levels of precision may include FP16, INT8 and INT4. Preferably, the step of monitoring the inference request queue also monitors additional statistical parameters in generating the monitoring result.

[0024] According to a second aspect, there is provided a system comprising means adapted for carrying out all the steps of the method described above. The system enables the dynamic switching between model precisions based on the inference request / system load. The system enables faster inference by reducing computation requirements, increases throughput by minimizing model size, and minimizes accuracy loss by adjusting precision in underutilized systems. Additionally, the system's ability to dynamically switch between sub-model precisions based on the inference request / system load optimizes resource utilization by enhancing efficiency and conserving memory and computational resources, thereby eliminating the need to load the entire model. The system also provides different levels of accuracy to users based on the availability of system resources, potentially supporting a pricing mechanism that reflects the varying accuracy levels, indicating the deployment of this advanced technology.

[0025] According to a third aspect, there is provided a computer program including instructions for carrying out all the steps of the above described method, when the computer program is executed on a computer system.

[0026] Therefore, in contradistinction to the existing solutions, the method enables the dynamic switching between model precisions based on the inference request / system load. The method enables faster inference by reducing computation requirements, increases throughput by minimizing model size, and minimizes accuracy loss by adjusting precision in underutilized systems. Additionally, the method's ability to dynamically switch between sub-model precisions based on the inference request / system load optimizes resource utilization by enhancing efficiency and conserving memory and computational resources, thereby eliminating the need to load the entire model.

[0027] These and other aspects of the disclosure will be apparent from the implementation(s) described below.

[0028] BRIEF DESCRIPTION OF DRAWINGS

[0029] Implementations of the disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0030] FIG. 1 is a block diagram that illustrates a system of processing inference requests in a deep neural network in accordance with an implementation of the disclosure;

[0031] FIGS. 2A-2C illustrate exemplary views of the system of FIG. 1 for processing inference requests in a deep neural network in accordance with an implementation of the disclosure; FIG. 3 is a flow diagram that illustrates a method of processing inference requests in a deep neural network in accordance with an implementation of the disclosure; and

[0032] FIG. 4 is an illustration of a computer system in which the various architectures and functionalities of the various previous implementations may be implemented.

[0033] DETAILED DESCRIPTION OF THE DRAWINGS

[0034] Implementations of the disclosure provide a method of processing inference requests in a deep neural network. Moreover, the disclosure relates to a system of processing inference requests in the deep neural network.

[0035] To make solutions of the disclosure more comprehensible for a person skilled in the art, the following implementations of the disclosure are described with reference to the accompanying drawings.

[0036] Terms such as "a first", "a second", "a third", and "a fourth" (if any) in the summary, claims, and foregoing accompanying drawings of the disclosure are used to distinguish between similar objects and are not necessarily used to describe a specific sequence or order. It should be understood that the terms so used are interchangeable under appropriate circumstances, so that the implementations of the disclosure described herein are, for example, capable of being implemented in sequences other than the sequences illustrated or described herein. Furthermore, the terms "include" and "have" and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units, is not necessarily limited to expressly listed steps or units but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or device.

[0037] FIG. 1 is a block diagram that illustrates a system 100 of processing inference requests in a deep neural network in accordance with an implementation of the disclosure. The system includes a memory 102. The memory 102 may be a memory pool or a local Dynamic Random- Access Memory (DRAM). The system 100 creates one or more quantized versions 104A-N of a deep neural network model. The deep neural network model is a machine learning model that uses multiple artificial neural network layers between the input and the output layers. The deep neural network model may be a large language model (LLM). The LLM may be a language model notable for its ability to achieve general-purpose language generation and other natural language processing tasks.

[0038] Preferably, the system 100 creates the one or more quantized versions 104A-N by selecting a first subset of one or more layers of the deep neural network model and quantizing only the first subset. Preferably, the system 100 further creates the one or more quantized versions 104A-N by selecting a second subset of the one or more layers of the deep neural network model and quantizing only the second subset. The first subset and the second subset may be quantized using a separate level of quantization precision.

[0039] Optionally, the quantization involves post training quantization. In the post training quantization, the deep neural network model that is trained is applied with static quantization multiple times. Each time, all the parameters (e.g., weights and activations) of the deep neural network model are converted to different levels of precision 106A-N (e.g., FP16, INT8, INT4). This post training quantization may generate multiple quantized versions 104A-N of the deep neural network model. Preferably, the quantization quantizes the weights and activations of the deep neural network model.

[0040] The system 100 stores the one or more quantized versions 104A-N of the deep neural network model in the memory 102. Optionally, the one or more quantized versions 104A-N are stored in the memory pool or local DRAM of the system 100 during runtime. Each of the quantized versions 104A-N of the model has a specific level of precision (e.g., 106A-N). The specific levels of precision 106A-N may include FP16, INT8 and INT4. The system 100 monitors an inference request queue (i.e., a system pressure / load) and selects one of the quantized versions 104A-N of the deep neural network model in accordance with the results of the monitoring. That is, depending on the monitored inference request queue / system pressure or load, the system 100 dynamically switches / selects one of the quantized versions 104A-N of the deep neural network model and loads the appropriate quantized version of the deep neural network model with lower precision (e.g., INT8 or INT4) onto a GPU to increase throughput and reduce latency under high system load (i.e., overutilized systems). The system 100 may load the appropriate quantized version of the deep neural network model with higher precision (e.g., FP16 or FP32) onto the GPU when the system load is lower (i.e., underutilized systems). This approach optimizes the resource usage by balancing precision and system performance based on the system’s current load. The inference request queue / system pressure may be determined by the number of inference requests waiting in the queue, along with other statistical parameters of the system. The switch between the quantized versions of the deep neural network model may be triggered based on user-defined system pressure thresholds or may be automatically determined by the system 100. Preferably, the system 100 also monitors additional statistical parameters in generating the monitoring result. Preferably, the processing of the inference requests is carried out using the Graphical Processing Unit (GPU). The GPU may be an Artificial Intelligence (Al) accelerator.

[0041] Preferably, the post training quantization involves selective quantization. Optionally, the system 100 quantizes the deep neural network model incrementally / selectively multiple times by selecting different portions / a subset of layers (e.g., an operator, a layer, or a sublayer) of the deep neural network model each time for quantization with varying precision levels (e.g., FP16, INT8, and INT4), which may result in models of different sizes. During runtime, the system 100 may dynamically switch between the quantized versions / subsets of the deep neural network model instead of the entire deep neural network model. For example, if the system 100 gradually quantizes the model having FP32 precision with two layers (i.e., layer 1 and layer 2), the system 100 creates four distinct quantized versions of the model, each with varying degrees of quantization applied to different parts of the deep neural network model. The system 100 simplifies the process by performing the quantization for N iterations. The system 100 may dynamically adjust the precision levels of the deep neural network model that is running based on the user pricing preferences. Pricing may be controlled by setting a maximum price limit, which prompts the system 100 to adjust precision according to the system pressure / load, or by offering users different pricing tiers based on the desired precision.

[0042] Preferably, the system 100 selects a first quantized version (e.g., 104A) of the model having a first level of precision 106A when the result of the monitoring is that the inference request queue has a first capacity level of queued requests and the system 100 selects a second quantized version (e.g., 104B) of the model having a second level of precision 106B when the result of the monitoring is that the inference request queue has a second capacity level of queued requests. Preferably, the first level of precision 106A is higher than the second level of precision 106B when the first capacity level is lower than the second capacity level. Preferably, the first level of precision 106A is lower than the second level of precision 106B when the first capacity level is higher than the second capacity level. The system 100 enables the dynamic switching between model precisions based on the inference request / system load. The system 100 enables faster inference by reducing computation requirements, increases throughput by minimizing model size, and minimizes accuracy loss by adjusting precision in underutilized systems. Additionally, the system's ability to dynamically switch between sub-model precisions based on the inference request / system load optimizes resource utilization by enhancing efficiency and conserving memory and computational resources, thereby eliminating the need to load the entire model. The system 100 also provides different levels of accuracy to users based on the availability of system resources, potentially supporting a pricing mechanism that reflects the varying accuracy levels, indicating the deployment of this advanced technology.

[0043] FIGS. 2A-2C illustrate exemplary views of the system of FIG. 1 for processing inference requests in a deep neural network in accordance with an implementation of the disclosure. The system includes a memory. The memory may be a memory pool or a local Dynamic Random- Access Memory (DRAM) 202. The system may process the inference requests using a Graphical Processing Unit (GPU) 204. The system creates one or more quantized versions by selecting one or more subsets of one or more layers of the deep neural network model and quantizing the subsets of the one or more layers. The quantization of the subsets of the one or more layers includes post training quantization. In the post training quantization, the deep neural network model that is trained is applied with static quantization multiple times. Each time, all the parameters (i.e., weights and activations) of the deep neural network model are converted to different levels of precision (e.g., FP16, INT8, INT4) as shown in FIG. 2A. This post training quantization may generate multiple quantized versions of the deep neural network model. Preferably, the quantization quantizes the weights and activations of the model. FIG. 2A illustrates an example of the system which includes an inference system with a single GPU 204 under varying system loads (i.e., inference requests) at different times. At time 7 (7 < T2~) with a system load of (Lx< L2~), the system runs the deep neural network model with FP16 precision, while at time T2, the system runs the deep neural network model with INT8 precision. That is, at time the system runs the deep neural network model with FP16 precision and at time T2, the system converts the FP16 precision used by the deep neural network model to INT8 precision as shown in FIG. 2A.

[0044] The system stores the one or more quantized versions of the deep neural network model in the memory pool or local DRAM 202 of the system during runtime. Each of the quantized versions of the model has a specific level of precision (e.g., FP16, INT8 and INT4). The system monitors an inference request queue (i.e., a system pressure / load). Depending on the monitored inference request queue / system pressure or load, the system dynamically switches / selects one of the quantized versions of the deep neural network model and loads all the parameters of the selected quantized version of the model with lower precision (e.g., INT8 or INT4) onto the GPU 204 to increase throughput and reduce latency under high system load (i.e., overutilized systems) as illustrated in FIG. 2A. The system may load all the parameters of the quantized versions of the model with higher precision (e.g., FP16 or FP32) onto the GPU 204 when the system load is lower (i.e., underutilized systems). This approach optimizes the resource usage by balancing precision and system performance based on the current load. The inference request queue / system pressure may be determined by the number of inference requests waiting in the queue, along with other statistical parameters of the system. The switch between the quantized version of the model may be triggered based on user-defined system pressure thresholds or may be automatically determined by the system.

[0045] FIG. 2B illustrates an example of the system which includes an inference system with a single GPU 204 under varying system loads (i.e., inference requests) at different times. At time 7 (7 < T2< T3) with a system load (Lx< L2< L3), the system runs the deep neural network model with FP16 precision, while at time T2, the layer 1 of the deep neural network model is replaced with INT8 precision, and by time T3, the layer 2 of the deep neural network model is also replaced with INT8 precision. In this example, the system creates one or more quantized versions by selecting one or more subsets of one or more layers of the deep neural network model and quantizing the subsets of the one or more layers. The quantization of the subsets of the one or more layers includes post training quantization, similar to the approach as described in FIG. 2A. The post training quantization involves selective quantization. In selective quantization, the system quantizes the model incrementally / selectively multiple times. Each time, the system selects different portions (e.g., an operator, a layer, or a sublayer of the deep neural network model) for quantization with varying precision levels (e.g., FP16, INT8 and INT4), resulting in models of different sizes. During runtime, the system may dynamically switch between the quantized versions of the subsets of layers of the deep neural network model instead of the entire model. That is, the system gradually quantizes different portions of the model, such as specific operators, layers, or sub-layers. With each iteration of the quantization, an additional portion of the model is quantized, thereby gradually lowering the precision of more components / portions of the model over time. FIG. 2C illustrates an example of the quantized versions / iteration of the deep neural network model that are gradually quantized. For example, in a base model, starting with FP32 precision with two layers (i.e., layer 1 and layer 2), the system creates four distinct quantized versions, each with varying degrees of quantization applied to different parts / portions of the model. At a first iteration of quantization (VI), only the layer 1 is converted from FP32 precision to FP16 precision. At a second iteration (V2), only the layer 2 is converted from FP32 precision to FP16 precision. At a third iteration (V3), only the layer 1 is converted from FP16 precision to INT8 precision. At a fourth iteration (V4), only the layer 2 is converted from FP16 precision to INT8 precision. The system simplifies the process by performing the quantization for N iterations. The system may dynamically adjust the precision levels of the running model based on the user pricing preferences.

[0046] The system stores the one or more quantized versions of the deep neural network model in the memory pool or local DRAM 202 of the system during runtime. Each of the quantized versions of the model has a specific level of precision (e.g., FP16, INT8 and INT4). The system monitors an inference request queue (i.e., a system pressure / load) and selects one of the quantized versions of the deep neural network model in accordance with the results of the monitoring. That is, depending on the monitored inference request queue / system pressure, the system dynamically switches / selects one of the quantized versions 104A-N (e.g., INT32) of the model and loads the appropriate quantized version (e.g., INTI 6) of the deep neural network model (i.e., next low precision version of the model) onto the GPU 204 to increase throughput and reduce latency during high system load. The system may load the next quantized versions of the model with higher precision (e.g., FP16 or FP32) onto the GPU 204 when the system load is lower. Preferably, the system selects a first quantized version of the model having a first level of precision (e.g., FP32) when the result of the monitoring is that the inference request queue has a first capacity level of queued requests and the system selects a second quantized version of the model having a second level of precision (e.g., INT16) when the result of the monitoring is that the inference request queue has a second capacity level of queued requests. Preferably, the first level of precision (e.g., FP32) is higher than the second level of precision (e.g., INT16) when the first capacity level is lower than the second capacity level. Preferably, the first level of precision is lower than the second level of precision when the first capacity level is higher than the second capacity level. FIG. 3 is a flow diagram that illustrates a method of processing inference requests in a deep neural network in accordance with an implementation of the disclosure. At a step 302, one or more quantized versions of a deep neural network model is stored in a memory. Each of the quantized versions of the model has a specific level of precision. At a step 304, an inference request queue is monitored. At a step 306, one of the quantized versions is selected in accordance with the results of the monitoring step.

[0047] Preferably, the selecting step selects a first quantized version of the model having a first level of precision when the result of the monitoring step is that the inference request queue has a first capacity level of queued requests and selects a second quantized version of the model having a second level of precision when the result of the monitoring step is that the inference request queue has a second capacity level of queued requests. Preferably, the first level of precision is higher than the second level of precision when the first capacity level is lower than the second capacity level. Preferably, the first level of precision is lower than the second level of precision when the first capacity level is higher than the second capacity level.

[0048] The method enables the dynamic switching between model precisions based on the inference request / system load. The method enables faster inference by reducing computation requirements, increases throughput by minimizing model size, and minimizes accuracy loss by adjusting precision in underutilized systems. Additionally, the method's ability to dynamically switch between sub-model precisions based on the inference request / system load optimizes resource utilization by enhancing efficiency and conserving memory and computational resources, thereby eliminating the need to load the entire model. The method also provides different levels of accuracy to users based on the availability of system resources, potentially supporting a pricing mechanism that reflects the varying accuracy levels, indicating the deployment of this advanced technology.

[0049] Preferably, the method further includes creating the plurality of quantized versions of the deep neural network model before storing the plurality of quantized versions of the deep neural network model in the memory. Optionally, the plurality of quantized versions is created by selecting a first subset of a plurality of layers of the deep neural network model and quantizing only the first subset. Optionally, the plurality of quantized versions is created by selecting a second subset of the plurality of layers of the deep neural network model and quantizing only the second subset. The first subset and the second subset may be quantized using a separate level of quantization precision.

[0050] Optionally, the quantization involves post training quantization. The model may be a large language model. Preferably, the processing is carried out using a Graphical Processing Unit. Preferably, the quantization quantizes the weights and activations of the model. The specific levels of precision may include FP16, INT8 and INT4. Preferably, the step of monitoring the inference request queue also monitors additional statistical parameters in generating the monitoring result.

[0051] FIG. 4 is an illustration of a computer system in which the various architectures and functionalities of the various previous implementations may be implemented. As shown, the computer system 400 includes at least one processor 404 that is connected to a bus 402, wherein the computer system 400 may be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), Hyper Transport, or any other bus or point-to-point communication protocol (s). The computer system 400 also includes a memory 406.

[0052] Control logic (software) and data are stored in the memory 406 which may take a form of random-access memory (RAM). In the disclosure, a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip modules with increased connectivity which simulate on- chip operation, and make substantial improvements over utilizing a conventional central processing unit (CPU) and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user.

[0053] The computer system 400 may also include a secondary storage 410. The secondary storage 410 includes, for example, a hard disk drive and a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash memory. The removable storage drive at least one of reads from and writes to a removable storage unit in a well-known manner. Computer programs, or computer control logic algorithms, may be stored in at least one of the memory 406 and the secondary storage 410. Such computer programs, when executed, enable the computer system 400 to perform various functions as described in the foregoing. The memory 406, the secondary storage 410, and any other storage are possible examples of computer-readable media.

[0054] In an implementation, the architectures and functionalities depicted in the various previous figures may be implemented in the context of the processor 404, a graphics processor coupled to a communication interface 412, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the processor 404 and a graphics processor, a chipset (namely, a group of integrated circuits designed to work and sold as a unit for performing related functions, and so forth).

[0055] Furthermore, the architectures and functionalities depicted in the various previous-described figures may be implemented in a context of a general computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application-specific system. For example, the computer system 400 may take the form of a desktop computer, a laptop computer, a server, a workstation, a game console, an embedded system.

[0056] Furthermore, the computer system 400 may take the form of various other devices including, but not limited to a personal digital assistant (PDA) device, a mobile phone device, a smart phone, a television, and so forth. Additionally, although not shown, the computer system 400 may be coupled to a network (for example, a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, or the like) for communication purposes through an I / O interface 408.

[0057] It should be understood that the arrangement of components illustrated in the figures described are exemplary and that other arrangement may be possible. It should also be understood that the various system components (and means) defined by the claims, described below, and illustrated in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described figures. In addition, while at least one of these components are implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.

[0058] Although the disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims.

Claims

CLAIMS1. A method of processing inference requests in a deep neural network, comprising steps of: storing in memory (102) a plurality of quantized versions (104A-N) of a deep neural network model, each of the plurality of quantized versions of the model having a specific level of precision (106A-N); monitoring an inference request queue; and selecting one of the plurality of quantized versions in accordance with the results of the monitoring step.

2. The method of claim 1, wherein the selecting step selects a first quantized version of the model having a first level of precision when the result of the monitoring step is that the inference request queue has a first capacity level of queued requests and selects a second quantized version of the model having a second level of precision when the result of the monitoring step is that the inference request queue has a second capacity level of queued requests.

3. The method of claim 2, wherein the first level of precision is higher than the second level of precision when the first capacity level is lower than the second capacity level.

4. The method of claim 1, wherein the first level of precision is lower than the second level of precision when the first capacity level is higher than the second capacity level.

5. The method of claim 1, further comprising a step, prior to the storing step, of creating the plurality of quantized versions of the deep neural network model.

6. The method of claim 5, wherein the creating step creates the plurality of quantized versions by selecting a first subset of a plurality of layers of the deep neural network model and quantizing only the first subset.

7. The method of claim 6, wherein the creating step further creates the plurality of quantized versions by selecting a second subset of the plurality of layers of the deep neural network model and quantizing only the second subset, wherein the first subset and the second subset are quantized using a separate level of quantization precision.

8. The method of claim 1, wherein the quantization involves post training quantization.

9. The method of claim 1, wherein the model is a large language model.

10. The method of claim 8, wherein the processing is carried out using a Graphical Processing Unit (204).

11. The method of claim 9, wherein the quantization quantizes the weights and activations of the model.

12. The method of claim 1, wherein the specific levels of precision include FP16, INT8 and INT4.

13. The method of claim 1, wherein the monitoring step also monitors additional statistical parameters in generating the monitoring result.

14. A system (100) comprising means adapted for carrying out all the steps of the method according to any preceding method claim.

15. A computer program comprising instructions for carrying out all the steps of the method according to any preceding method claim, when said computer program is executed on a computer system.

Images