Quantization Methods for Accelerating Neural Network Inference

Abstract

The electronic device performs a method of quantizing a neural network, the method including the steps of: clipping values ​​used in the neural network beyond a range from a minimum to a maximum; simulating a quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing values ​​used in the neural network according to the updated minimum and maximum values. In some embodiments, the method of quantizing a neural network further includes the step of minimizing the range during training.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is related to a continuation of U.S. patent application Ser. No. 17 / 693,270, entitled "Quantization Method for Accelerating Neural Network Inference," filed on March 11, 2022, which is incorporated herein by reference in its entirety.

[0002]

[0002] Technical field FIELD OF THE DISCLOSURE The present disclosure relates generally to data processing techniques, and more particularly to quantization methods for accelerating inference in neural network systems. [Background technology]

[0003] Quantization in neural networks uses fewer bits to represent numbers in neural network storage and computation. For example, a neural network trained in a 32-bit floating-point (FP32) precision format can be converted to a format using 8-bit signed integers (INT8), resulting in a four-fold reduction in model storage and memory footprint. Furthermore, performing model inference in the INT8 format can be achieved using a Single Instruction Multiple Dataset (SIMD) mechanism, in which a single instruction, such as multiplication, can be performed simultaneously on four 8-bit integers rather than one 32-bit floating-point number. This results in a 75% reduction in computation time.

[0004]

[0004] The main challenge in converting neural network models from FP32 to INT8 format is the loss of model accuracy due to the loss of numerical precision. Various methods exist to address this issue: The loss of accuracy can be partially recovered by using post-training quantization (PTQ) or quantization-aware training (QAT) methods. PTQ uses a representative training data set to adjust the minimum and maximum quantization range in FP32 format. QAT adjusts the minimum and maximum quantization range, while fine-tuning the neural network model weights during the training process. What PTQ and QAT have in common is that the minimum and maximum ranges of layer weights and activations are adjusted to help recover model accuracy loss due to quantization. In practice, the minimum and maximum values ​​are updated according to batch statistics during the training process. Summary of the Invention

[0005]

[0005] To overcome the deficiencies or shortcomings of the above methods, there is a need for improved systems and methods for accelerating the inference of neural network systems.

[0006]

[0006] No previous research has provided a solution for optimizing the minimum and maximum values ​​of the quantization range. Existing research uses statistically summarized minimum and maximum values ​​during the calibration or training process. This is because the minimum and maximum values ​​are not differentiable and therefore cannot be learned from the training process of a neural network model.

[0007] A major limitation of previous work is that the quantization range, determined by the minimum and maximum values, is statistically summarized during the calibration or training process. While this may be sufficient if the training data is normalized, many deep learning applications, such as reinforcement learning (RL), may not have well-defined inputs, and therefore the input data is not normalized. In such cases, simply summarizing the minimum and maximum values ​​of a batch of training data is susceptible to sudden spikes in the input data. As a result, any subsequent spikes in the intermediate layers can dramatically increase the minimum and maximum values ​​of the quantization range, adversely affecting the training loss and accuracy of the neural network.

[0008]

[0008] The technical problem to be solved by the present invention is that the method and system disclosed herein address the limitations of prior work and optimize the minimum and maximum values ​​of the quantization range. In contrast to the methods used in prior work, the minimum and maximum values ​​are formulated as analytical functions that serve the purpose of clipping values ​​that exceed the range defined by the minimum and maximum values. Furthermore, the quantization range is minimized so that the resolution of quantization from floating-point numbers to integers can be optimized.

[0009]

[0009] Another technical problem to be solved by the present invention is that the method and system disclosed herein quantize a neural network model to reduce the calculation time for model inference. A method for optimizing a clipping function that defines the quantization range of the neural network weight parameters and activations is introduced. By using this method, the clipping function is optimized, so that a narrow quantization range can be reached to ensure the optimal quantization resolution.

[0010]

[0010] According to a first aspect of the present application, a method for quantizing a neural network includes: clipping values ​​used in the neural network that exceed a range from a minimum value to a maximum value; simulating a quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing values ​​used in the neural network according to the updated minimum and maximum values.

[0011]

[0011] In some embodiments, the method for quantizing a neural network further includes minimizing the range during training.

[0012] According to a second aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory, the programs, when executed by the one or more processing units, causing the electronic device to perform one or more of the methods described above.

[0013] According to a third aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units, the programs, when executed by the one or more processing units, causing the electronic device to perform one or more of the methods described above.

[0014]

[0014] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not all-inclusive, and many additional features and advantages will be apparent to those skilled in the art, especially in view of the drawings, specification, and claims. Furthermore, it should be noted that the language used herein has been selected primarily for readability and instructional purposes, and may not have been selected to delineate or limit the subject matter herein. [Brief explanation of the drawings]

[0015]

[0015] In order that the present disclosure may be more fully understood, a more particular description will be made by reference to features of various embodiments, some of which are illustrated in the accompanying drawings. However, the accompanying drawings merely illustrate relevant features of the present disclosure and therefore should not be considered limiting, as the present description may allow for other useful features. [Figure 1]

[0016] FIG. 1 illustrates an exemplary computing environment in which one or more networked client devices and one or more server systems interact with each other locally or remotely via one or more communications networks, in accordance with some implementations of the present disclosure. [Figure 2]

[0017] FIG. 2 is an exemplary neural network that may be implemented to process data in a neural network model according to some implementations of the present disclosure. [Figure 3A]

[0018] FIG. 3A illustrates an example symmetric clipping scenario with respect to α and β for shaping clipping functions according to some implementations of the present disclosure. [Figure 3B]

[0019] FIG. 3B illustrates an example asymmetric clipping scenario with respect to α and β for shaping clipping functions according to some implementations of the present disclosure. [Figure 3C]

[0020] FIG. 3C illustrates an example positive clipping scenario for α and β for shaping clipping functions according to some implementations of the present disclosure. [Figure 3D]

[0021] FIG. 3D illustrates an example negative clipping scenario for α and β for shaping clipping functions according to some implementations of the present disclosure. [Figure 4]

[0022] FIG. 4 illustrates the workflow and structural components of a neural network quantization method according to some implementations of the present disclosure. [Figure 5]

[0023] FIG. 5 is a block diagram illustrating an example process for quantizing a neural network according to some implementations of the present disclosure.

[0024] According to common practice, the various features illustrated in the drawings may not be drawn to scale. Accordingly, dimensions of various features may be arbitrarily increased or decreased for clarity. Moreover, some drawings may not depict all of the components of a given system, method, or apparatus. Finally, like reference numerals may be used throughout the specification and drawings to refer to like features. DETAILED DESCRIPTION OF THE INVENTION

[0016]

[0025] Reference is made to specific implementation details, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein may be practiced in many types of electronic devices.

[0017]

[0026] Before describing the embodiments of the present application in more detail, the names and terms associated with the embodiments of the present application will be described, and the names and terms associated with the embodiments of the present application will have the following meanings.

[0018]

[0027] NN (Neural Networks): Neural networks

[0028] FP32 (32-bit Floating Point): 32-bit floating point

[0029] INT8 (8-bit Integer): 8-bit integer

[0030] PTQ (Post-Training Quantization): Post-Training Quantization

[0031] QAT (Quantization-Aware Training): Training that takes quantization into account

[0032] SIMD (Single Instruction Multiple Data): Single instruction multiple data

[0033] SGD (Stochastic Gradient Descend): Stochastic gradient descent

[0034] RL (Reinforcement Learning):

[0035] min(minimum): minimum value

[0036] max(maximum): maximum value

[0037] FIG. 1 illustrates an exemplary computing environment 100 in which one or more networked client devices 102 and one or more server systems 104 interact with each other locally or remotely via one or more communications networks 106, in accordance with some implementations of the present disclosure.

[0019]

[0038] In some embodiments, a server system 104, such as 104A and 104B, is physically separate from but communicatively coupled to one or more client devices 102. In some embodiments, a client device 102 (e.g., 102A, 102B) comprises a desktop computer. In some embodiments, a client device 102 (e.g., 102C) comprises a mobile device, such as a mobile phone, a tablet computer, and a laptop computer. Each client device 102 can collect data or user input, run user applications, and present output on its user interface. The collected data or user input can be processed locally at the client device 102 and / or remotely by the server 104. Each client device 102 communicates with another client device 102 or the server system 104 using one or more communications networks 106. The communication network 106 can be one or more networks having one or more types of topologies, including, but not limited to, the Internet, an intranet, a local area network (LAN), a cellular network, an Ethernet, a telephone network, a Bluetooth personal area network (PAN), etc. In some embodiments, two or more client devices 102 within a sub-network are coupled via a wired connection, while at least some client devices 102 within the same sub-network are coupled via a local wireless communication network (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and other wireless communication networks). In one example, a client device 102 establishes a connection to one or more communication networks 106 directly (e.g., using 3G / 4G connectivity to a wireless carrier), through a network interface (e.g., a router, switch, gateway, hub, or intelligent, dedicated whole-home control node), or any combination thereof.

[0020]

[0039] Each of the server systems 104 includes one or more processors 110 and memory that stores instructions for execution by the one or more processors 110. The server systems 104 also include an input / output interface to the client devices 102 as 114. The one or more server systems 104 provide system data (e.g., boot files, operating system images, and user applications) to the client devices 102 and, in some embodiments, process data and user input received from the client devices 102 when the user applications run on the client devices 102. The one or more server systems 104 may enable real-time data communication with client devices 102 that are remote from each other or from one or more of the server systems 104. Server system 104A is configured to store data storage 112. Server system 104B is configured to store neural network models 116. In some embodiments, the neural network models and data storage may be within the same server 104. The neural network training method may be implemented on one or more server systems 104.

[0021]

[0040] The client device 102 includes one or more processors and memory that stores instructions for execution by the one or more processors. The instructions stored on the client device 102 enable the implementation of a user interface application and a web browser for the server 104. The web browser and user interface application are linked to a user account in the computing environment 100.

[0022]

[0041] Neural network training techniques are applied in the computing environment 100 to process data obtained by an application running on the client device 102 or loaded from another data storage or file to identify information contained in the data, match the data with other data, classify the data, or synthesize related data. The data may include text, images, audio, video, etc. A neural network model is trained using the training data before being applied to process the data. In some embodiments, the neural network model training method is implemented on the client device 102. In some embodiments, the neural network model training method is implemented jointly on the client device 102 and the server system 104. In some embodiments, the neural network model may be maintained on the client device 102. In some embodiments, the client device 102 is configured to identify, classify, or modify data information from the data storage 112 or from the neural network model 116 automatically and without user intervention.

[0023]

[0042] In some embodiments, both model training and data processing are performed locally at each individual client device 102 (e.g., client device 102C). Client device 102C retrieves training data from one or more server systems 104, including data storage 112, and applies the training data to train a neural network model. Following model training, client device 104C retrieves and processes data locally using the trained neural network model. Alternatively, in some embodiments, both model training and data processing are performed remotely at a server system 104 (e.g., server system 104B) associated with a client device 102 (e.g., client device 102A). Server 104B retrieves training data from itself, another server 104, or data storage 112, and applies the training data to train a neural network model 116. The client device 102A acquires data, transmits the data to the server 104B (e.g., within an application) for data processing using the trained neural network model, receives data processing results from the server 104B, and presents the results in a user interface (e.g., associated with the application). The client device 102A itself performs little or no data processing on the data before transmitting the data to the server 104B. Furthermore, in some embodiments, data processing is performed locally on the client device 102 (e.g., client device 102B), while model training is performed remotely on a server system 104 (e.g., server 104B) associated with the client device 102B. The trained neural network model is optionally stored in another data storage, such as server 104B or 112.The client device 102B imports the trained neural network model from the server 104B or the data storage 112, processes data using the neural network model, and generates data processing results that are presented locally in a user interface.

[0043] The neural network model system 116 may include one or more of a server, a client device, storage, or a combination thereof. The neural network model system 116 typically includes one or more processing units (CPUs), one or more network interfaces, memory, and one or more communication buses (sometimes referred to as chipsets) for interconnecting these components. The neural network model system 116 may also include one or more input devices for prompting user input, such as a keyboard, a mouse, a voice command input unit or microphone, a touchscreen display, a touch-sensitive input pad, a gesture-capturing camera, or other input buttons or controls. The neural network model system 116 may also include one or more output devices for enabling presentation of a user interface and display content, including one or more speakers and / or one or more visual displays.

[0024]

[0044] The memory includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. The memory optionally includes one or more storage devices located remotely from the one or more processing units. The memory, or alternatively, the non-volatile memory within the memory, includes a non-transitory computer-readable storage medium. In some embodiments, the memory, or the non-transitory computer-readable storage medium of the memory, stores programs, modules, and data structures, including an operating system, an input processing module for detecting and processing input data, a model training module for receiving training data and establishing a neural network model for processing the data, a neural network module for processing the data using the neural network model, and the like.

[0045] Each of the above-identified elements can be stored in one or more of the memory devices mentioned above and corresponds to a set of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) are not required to be implemented as separate software programs, procedures, modules, or data structures; thus, various subsets of these modules can be combined or otherwise rearranged in various embodiments. In some embodiments, the memory optionally stores a subset of the above-identified modules and data structures. Additionally, the memory optionally stores additional modules and data structures not described above.

[0025]

[0046] 2 illustrates an example neural network 200 that may be implemented to process data in the neural network model 116 according to some implementations of the present disclosure. The neural network model 116 is built based on the neural network 200. A corresponding model-based processing module in the server system 104B applies the neural network model 116, including the neural network 200, to process the data.

[0026]

[0047] In some examples, neural network 200 includes a collection of neuron nodes 220 connected by links 212. Each neuron node 220 receives one or more neuron node inputs and applies a propagation function to generate a neuron node output from the one or more neuron node inputs. A weight associated with each link 212 is applied to the neuron node output as it is transmitted through one or more links 212 to one or more other neuron nodes 220. One or more neuron node inputs are combined based on their corresponding weights according to the propagation function. In one example, the propagation function is the product of a nonlinear activation function and a linear weighted combination of one or more neuron node inputs.

[0027]

[0048] A neural network consists of one or more layers having neuron nodes 220. In some embodiments, the one or more layers include a single layer that acts as both an input layer and an output layer. In some embodiments, the one or more layers include an input layer 202 for receiving input, an output layer 206 for generating output, and zero or more hidden / latent layers 204 (e.g., 204A and 204B) between the input layer 202 and the output layer 206. A deep neural network has more than one hidden layer 204 between the input layer 202 and the output layer 206. In the neural network 200, each layer is only connected to its immediately preceding and / or succeeding layer. In some embodiments, layer 202 or 204B is a fully connected layer because each neuron node 220 in layer 202 or 204B is connected to every neuron node 220 in the immediately succeeding layer.

[0028]

[0049] In some embodiments, one or more neural networks may be used by the neural network model 116. The one or more neural networks may include a fully connected neural network, a multi-layer perceptron, a convolutional neural network, a recurrent neural network, a feed-forward neural network, a radial basis function neural network, a long short-term memory (LSTM), an auto-encoder, and a sequence-to-sequence model, among others.

[0029]

[0050] The training process is a process for calibrating all weights in each layer of the learning model using a training data set provided to the input layer 202. The training process typically includes two steps (forward propagation and backpropagation) that are repeated multiple times until a preset convergence condition is met. In forward propagation, a set of weights for various layers is applied to the input data and intermediate results from the previous layer. In backpropagation, the margin of error (e.g., a loss function) of the output is measured, and the weights are adjusted accordingly to reduce the error. The activation function is optionally linear, normalized linear unit, sigmoid, hyperbolic tangent, or other type. In some embodiments, a network bias term is added to the sum of weighted outputs from the previous layer before the activation function is applied. The network bias provides a perturbation that helps the neural network 200 avoid overfitting the training data. The training results include the network bias parameters for each layer.

[0030]

[0051] The methods and systems disclosed herein have many advantages, including: differentiable (or trainable or learnable) minimum and maximum quantization parameters are more robust to variations in input data and can quantize neural network models with higher accuracy compared to quantization techniques in which quantization parameters are simply statistically summarized from batch data.

[0031]

[0052] In some embodiments, the clipping functions disclosed herein are generalized solutions and are applicable to symmetric or asymmetric quantization.

[0032]

[0053] In some embodiments, the clipping function may be applied during the training process along with an additional L2 regularization method to minimize the quantization range determined by the minimum and maximum values ​​and improve the quantization resolution.

[0033]

[0054] In some embodiments, the values ​​of the methods and systems of the present invention are applied to weights and intermediate features in a neural network. Values ​​are not affected if they fall within a range between a minimum and maximum value. The analytical function is defined as follows:

[0034]

number

[0035]

[0056] In some embodiments, the analysis clipping function has the following characteristics:

[0036]

number

[0056] In some examples, the minimum and maximum values ​​are determined automatically during the training process. Figures 3A-3D show different scenarios of how different α and β shape clipping functions, according to some implementations of the present disclosure.

[0037] 3A illustrates an exemplary symmetric clipping scenario for α and β for forming a clipping function according to some implementations of the present disclosure, e.g., α=−6, β=6.

[0038] 3B illustrates an exemplary asymmetric clipping scenario for α and β for forming a clipping function according to some implementations of the present disclosure, e.g., α=−6, β=2.

[0039] 3C illustrates an example positive clipping scenario for α and β for forming a clipping function according to some implementations of the present disclosure, e.g., α=2, β=6.

[0040] 3D illustrates an example negative clipping scenario for α and β for forming a clipping function according to some implementations of the present disclosure, e.g., α=−6, β=−2.

[0041]

[0057] In some embodiments, additional techniques are used to minimize the range determined by α and β, i.e., |α-β|, along with the analytical clipping function. For example, an L2 regularization method to minimize the quantization range is applied to the loss function:

[0042]

number

[0058] In some embodiments, the goal of the L2 regularization method is to constrain the model to have optimal and minimal quantization ranges for the parameters and activations of all layers in the network.

[0043]

[0059] In some embodiments, new analytical functions are used to define generalized differentiable minimum and maximum values ​​for quantizing neural network models, providing symmetric or asymmetric quantization, resulting in a more robust solution to accuracy degradation caused by spikes in the input data and hidden layer activations.

[0044]

[0060] FIG. 4 illustrates the workflow and structural components of a neural network quantization method according to some implementations of the present disclosure.

[0045]

[0061] In some embodiments, the value to be quantized 410 is fed to a clipping function 420. The clipped value 430 from the clipping function 420 is further fed 440 to a quantization process along with α and β, the output of which is a quantized value 450.

[0046]

[0062] In some embodiments, a personal computer (PC) or mobile device performs the training and inference of the neural network model.

[0047]

[0063] In some embodiments, the quantization process follows the following steps: First, for each layer of the neural network, the layer parameters and activations (e.g., values ​​to be quantized 410 as shown in FIG. 4) are fed to a clipping function (e.g., clipping function 420 as shown in FIG. 4) defined by Eq. 1, respectively, in the forward propagation stage. The method clips the values ​​(e.g., values ​​to be quantized 410 as shown in FIG. 4) to a range of minimum and maximum values ​​determined by the clipping function (e.g., clipping function 420 as shown in FIG. 4) using parameters α and β.

[0048]

[0064] Second, the weight parameters and intermediate activations in each layer (e.g., clipped values ​​430 as shown in FIG. 4) are quantized using a fake or simulation quantization method, where the operations are still performed in FP32 format but are quantized to INT8 values ​​(e.g., quantized values ​​450 as shown in FIG. 4) to mimic the behavior of INT8 operations. In a fake or simulation quantization method, the quantization process is mimicked.

[0049]

[0065] In some embodiments, an L2 regularization method is added to the loss to optimize model accuracy, as shown in Eq. 2. In some cases, L2 regularization is optional and can be used to minimize the quantization range, resulting in an optimal high quantization resolution.

[0050]

[0066] In some embodiments, during the training process, the minimum and maximum values ​​are updated along with the neural network weight parameters through backpropagation using a gradient-based numerical method such as SGD, which updates the values ​​of α and β during the training process to converge to an optimal solution (e.g., a quantization process using α and β 440 as shown in FIG. 4).

[0051]

[0067] In some embodiments, structural components of the disclosed methods interact with each other. For example, in forward propagation, the output from the clipping function is fed into a simulated quantization process. In back propagation, a training process updates the values ​​of α and β. L2 regularization is part of the loss function that guides the optimization of the values ​​of α and β.

[0052]

[0068] In some embodiments, the data used in the disclosed methods and systems includes input training data and neural network weight parameters. Both types of data are quantized so that operations in model inference can be performed in a desired quantization format, such as from FP32 to INT8. In each layer, values ​​to be quantized, such as layer weights and activations, are first clipped using the clipping function shown in Eq. 1. The clipped values ​​are then quantized using a fake quantization method to mimic the quantization process. After calibration with PTQ or training with QAT is complete, the model inputs and model weights are converted from FP32 to INT8 so that model storage and inference can be performed in the target format, such as INT8.

[0053]

[0069] Alternatively, in some embodiments, quantization can be applied in a per-channel fashion, so that each channel has its own quantization parameter. For example, if the layer is a 2D convolutional layer and its weight dimension is [N k ,N k ,N i ,N o ] and the input activation dimension is [N x ,N y ,N i ] and the output activation dimension is [N x ’ ,N y ’ ,N o ], which is commonly used in per-layer quantization, where the quantization parameters are triplet scalars applied to the weights, input activations, and output activations, respectively. In per-channel quantization, the scalar quantization parameter pairs are replaced by vector pairs, where each element of the vector matches the channel dimension, and each channel is quantized by a different range.

[0054]

[0070] FIG. 5 is a block diagram illustrating an example process 500 for quantizing a neural network according to some implementations of the present disclosure.

[0055]

[0071] The process 500 of quantizing a neural network includes a step 502 of clipping values ​​used in the neural network that exceed a range from a minimum to a maximum value.

[0056]

[0072] The process 500 includes a step 504 of simulating a quantization process using clipped values.

[0057]

[0073] The process 500 then includes a step 506 of updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process.

[0058]

[0074] The process 500 additionally includes a step 508 of quantizing the values ​​used in the neural network according to the updated minimum and maximum values.

[0059]

[0075] For example, minimum and maximum values ​​are formulated in an analytical function, which acts to clip values ​​that exceed the range defined by the minimum and maximum values, which are then applied to the weights and intermediate feature values ​​in the neural network.

[0060]

[0076] In some embodiments, process 500 additionally includes a step 510 of minimizing the range during training. For example, the quantization range is minimized so that the resolution from quantization from floating point numbers to integers can be optimized.

[0061]

[0077] In some embodiments, the values ​​used within the neural network include one or more of the weights, layer activations, and intermediate features in the neural network. For example, the values ​​are applied to the weights and intermediate features in the neural network. Values ​​to be quantized, such as layer weights and layer activations, are first clipped using a clipping function.

[0062]

[0078] In some embodiments, clipping 502 values ​​used in the neural network beyond a minimum to maximum range is performed by the following clipping function:

[0063]

number

[0079] where α is the minimum value, β is the maximum value, x is the value used in the neural network, and f(x) is the clipping function. For example, if a value is within the range between the minimum and maximum values, it will not be affected.

[0064]

[0080] In some embodiments, clipping 502 values ​​used within the neural network beyond a minimum to maximum range includes at least one of symmetric clipping, asymmetric clipping, positive clipping, and negative clipping. For example, the minimum and maximum values ​​are determined automatically during training, and Figures 3A-3D show various scenarios of how different α and β may form the clipping function.

[0065]

[0081] In some embodiments, minimizing the range during training (510) includes: minimizing the range during training using L2 regularization applied to the loss function during training. For example, a clipping function can be applied with additional L2 regularization during training to minimize the range determined by the min / max range and increase the quantization resolution. An additional technique is used to minimize the range determined by α and β, i.e., |α-β|. L2 regularization to minimize the quantization range is applied to the loss function.

[0066]

[0082] In some embodiments, clipping 502 values ​​used within the neural network that are outside the range of a minimum to a maximum value is performed in forward propagation.

[0067]

[0083] In some embodiments, updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process (506) is performed with backpropagation.

[0068]

[0084] In some embodiments, simulating the quantization process using clipped values ​​(504) includes operating the simulated quantization in FP32 format and quantizing the clipped values ​​to INT8 format. For example, the weight parameters and intermediate activations in each layer are quantized using a fake quantization method, where operations are still performed in FP32 format but are quantized to INT8 values ​​to mimic the behavior of INT8 operations.

[0069]

[0085] In some embodiments, clipping (502) values ​​used in the neural network beyond a range from a minimum value to a maximum value includes: for each channel of the plurality of channels, clipping each value used in the neural network beyond a respective range from a respective minimum value to a respective maximum value. For example, quantization can be applied in a per-channel manner, where each channel has its own quantization parameter. In per-channel quantization, pairs of scalar quantization parameters are replaced by pairs of vectors, where each element is aligned with the channel dimension, and each channel is quantized in a different range.

[0070]

[0086] Additionally, further embodiments include various subsets of the above-described embodiments, including those illustrated in FIGS. 1-5, combined or otherwise rearranged in various other embodiments.

[0071]

[0087] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media, such as data storage media, or communication media, including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communications protocol. As such, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media or (2) communication media, such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the implementations described herein. A computer program product may include computer-readable media.

[0072]

[0088] The terminology used in the description of implementations herein is for the purpose of describing particular implementations only and is not intended to limit the scope of the claims. As used in the description of implementations and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or," as used herein, refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that, as used herein, the terms "comprises" and / or "comprising" specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0073]

[0089] It will also be understood that although terms such as "first," "second," and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first electrode can be referred to as a second electrode, and similarly, a second electrode can be referred to as a first electrode, without departing from the scope of the present invention. Although a first electrode and a second electrode are both electrodes, they are not the same electrode.

[0074]

[0090] The description in this application has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications, variations, and alternative implementations will be apparent to those skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. The embodiments have been chosen and described to best explain the principles and practical applications of the invention and to enable others skilled in the art to understand the invention in its various implementations and to best utilize these basic principles and various implementations, with various modifications, as suited to the particular use contemplated. It is therefore to be understood that the scope of the claims is not limited to the particular examples of implementations disclosed, and that modifications and other implementations are intended to be encompassed within the scope of the appended claims.

[0075]

[0091] <<Additional Notes>> (Appendix 1) 1. A method of quantizing a neural network, comprising: clipping values ​​used in said neural network that exceed a range from a minimum value to a maximum value; simulating the quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing the values ​​used in the neural network according to the updated minimum and maximum values; A method comprising:

[0076] (Appendix 2) The method of claim 1, further comprising: minimizing the range during the training; A method comprising:

[0077] (Appendix 3) 2. The method of claim 1, wherein the values ​​used in the neural network include values ​​of one or more of weights, layer activations, and intermediate features in the neural network.

[0078] (Appendix 4) 2. The method of claim 1, wherein the step of clipping values ​​used in the neural network that exceed the minimum to maximum range comprises using a clipping function:

[0079]

number

[0080] (Appendix 5) 2. The method of claim 1, wherein clipping values ​​used in the neural network that exceed the minimum to maximum range includes at least one of symmetric clipping, asymmetric clipping, positive clipping, and negative clipping.

[0081] (Appendix 6) 3. The method of claim 2, wherein minimizing the range during training comprises: minimizing the range during the training using L2 regularization applied to a loss function during the training; A method comprising:

[0082] (Appendix 7) 2. The method of claim 1, wherein the step of clipping values ​​used in the neural network that exceed the minimum to maximum range is performed in forward propagation.

[0083] (Appendix 8) 2. The method of claim 1, wherein the step of updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process is performed by backpropagation.

[0084] (Appendix 9) 10. The method of claim 1, wherein simulating a quantization process using the clipped values ​​includes performing a simulated quantization operation in FP32 format and quantizing the clipped values ​​to INT8 format.

[0085] (Appendix 10) 2. The method of claim 1, wherein clipping values ​​used in the neural network beyond a range from a minimum value to a maximum value includes clipping, for each channel of a plurality of channels, each value used in the neural network beyond a respective range from a respective minimum value to a respective maximum value.

[0086] (Appendix 11) 1. An electronic device comprising: one or more processing units; a memory coupled to the one or more processing units; and a plurality of programs stored in the memory, the plurality of programs, when executed by the one or more processing units, causing the electronic device to perform a plurality of operations for quantizing a neural network, the plurality of operations comprising: clipping values ​​used in said neural network that exceed a range from a minimum value to a maximum value; simulating the quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing the values ​​used in the neural network according to the updated minimum and maximum values; 2. An electronic device comprising:

[0087] (Appendix 12) 12. The electronic device of claim 11, wherein the operations of quantizing the neural network further include: minimizing the range during the training; 2. An electronic device comprising:

[0088] (Appendix 13) 12. The electronic device of claim 11, wherein the values ​​used in the neural network include values ​​of one or more of weights, layer activations, and intermediate features in the neural network.

[0089] (Appendix 14) 12. The electronic device of claim 11, wherein clipping values ​​used in the neural network that are outside the minimum to maximum range comprises using a clipping function:

[0090]

number

[0091] (Appendix 15) 12. The electronic device of claim 11, wherein clipping values ​​used in the neural network that exceed the minimum to maximum range includes at least one of symmetric clipping, asymmetric clipping, positive clipping, and negative clipping.

[0092] (Appendix 16) 13. The electronic device of claim 12, wherein minimizing the range during the training comprises: minimizing the range during the training using L2 regularization applied to a loss function during the training; 2. An electronic device comprising:

[0093] (Appendix 17) 12. The electronic device of claim 11, wherein clipping values ​​used in the neural network that are outside the minimum to maximum range is performed in forward propagation.

[0094] (Appendix 18) 12. The electronic device of claim 11, wherein updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process is performed by backpropagation.

[0095] (Appendix 19) 1. A non-transitory computer-readable storage medium storing a plurality of programs for execution by an electronic device having one or more processing units, the plurality of programs, when executed by the one or more processing units, causing the electronic device to perform a plurality of operations for quantizing a neural network, the plurality of operations comprising: clipping values ​​used in said neural network that exceed a range from a minimum value to a maximum value; simulating the quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing the values ​​used in the neural network according to the updated minimum and maximum values; 1. A non-transitory computer-readable storage medium, comprising:

[0096] (Appendix 20) 19. The non-transitory computer-readable storage medium of claim 19, wherein the operations for quantizing the neural network further include: minimizing the range during the training; 1. A non-transitory computer-readable storage medium, comprising:

Claims

1. 1. A method executed by one or more processing units of an electronic device for quantizing a neural network, the method comprising: clipping values ​​used in said neural network beyond a range from a minimum value to a maximum value; simulating a quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing the values ​​used in the neural network according to the updated minimum and maximum values; and further comprising minimizing the range by using L2 regularization applied to a loss function during training of the neural network.

2. 10. The method of claim 1, wherein the values ​​used in the neural network include values ​​of one or more of weights, layer activations, and intermediate features in the neural network.

3. 2. The method of claim 1, wherein the step of clipping values ​​used in the neural network beyond the minimum to maximum range comprises using a clipping function: [Equation 1] where α is the minimum value, β is the maximum value, x is a value used in the neural network, and f(x) is the clipping function.

4. 2. The method of claim 1, wherein clipping values ​​used in the neural network beyond the minimum to maximum range includes at least one of symmetric clipping, asymmetric clipping, positive clipping, and negative clipping.

5. 2. The method of claim 1, wherein the step of clipping values ​​used in the neural network beyond the minimum to maximum range is performed in forward propagation.

6. 2. The method of claim 1, wherein the step of updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process is performed by backpropagation.

7. 2. The method of claim 1, wherein simulating a quantization process using the clipped values ​​comprises performing a simulated quantization operation in FP32 format and quantizing the clipped values ​​to INT8 format.

8. A method executed by one or more processing units of an electronic device for quantizing a neural network, comprising: clipping values ​​used in said neural network beyond a range from a minimum value to a maximum value; simulating a quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing the values ​​used in the neural network according to the updated minimum and maximum values; wherein clipping values ​​used in the neural network beyond a range from a minimum value to a maximum value comprises clipping, for each channel of a plurality of channels, each value used in the neural network beyond a respective range from a respective minimum value to a respective maximum value.

9. 9. An electronic device comprising one or more processing units, a memory coupled to the one or more processing units, and a plurality of programs stored in the memory, the plurality of programs, when executed by the one or more processing units, causing the processing units to perform the method of any one of claims 1 to 8.

10. A computer program product that causes a computer to carry out the method according to any one of claims 1 to 8.

Images