Quantization Methods for Accelerating Neural Network Inference

Abstract

The electronic device performs a method of quantizing a neural network, the method including the steps of: clipping values ​​used in the neural network beyond a range from a minimum to a maximum; simulating a quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing values ​​used in the neural network according to the updated minimum and maximum values. In some embodiments, the method of quantizing a neural network further includes the step of minimizing the range during training.

Description

[Technical field]

[0001]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is related to a continuation of U.S. patent application Ser. No. 17 / 693,270, entitled "Quantization Method for Accelerating Neural Network Inference," filed on March 11, 2022, which is incorporated by reference in its entirety.

[0002]

[0002] TECHNICAL FIELD The present disclosure relates generally to data processing techniques, and more particularly to quantization methods for accelerating inference in neural network systems. [Background technology]

[0003] Quantization in neural networks uses fewer bits to represent numbers in neural network storage and computation. For example, a neural network trained in a 32-bit floating-point (FP32) precision format can be converted to a format using 8-bit signed integers (INT8), resulting in a four-fold reduction in model storage and memory footprint. Furthermore, performing model inference in the INT8 format can be achieved using a Single Instruction Multiple Dataset (SIMD) mechanism, where a single instruction, such as multiplication, can be performed simultaneously on four 8-bit integers instead of one 32-bit floating point. This results in a 75% reduction in computation time.

[0004]

[0004] The main challenge in converting neural network models from FP32 to INT8 format is the loss of model accuracy due to the loss of numerical precision. There are various methods to address this issue: The loss of accuracy can be recovered to some extent by using post-training quantization (PTQ) or quantization-aware training (QAT) methods. PTQ methods adjust the minimum and maximum quantization range in FP32 format using a representative training data set. In QAT, the minimum and maximum quantization range are adjusted, while the weights of the neural network model are fine-tuned during the training process. What PTQ and QAT have in common is that the minimum and maximum ranges of layer weights and activations are adjusted to help recover the model accuracy loss caused by quantization. In practice, the minimum and maximum values ​​are updated according to batch statistics during the training process. Summary of the Invention

[0005]

[0005] To overcome the deficiencies or shortcomings of the above methods, there is a need for improved systems and methods for accelerating inference in neural network systems.

[0006]

[0006] No prior work has provided a solution for optimizing the minimum and maximum values ​​of the quantization range. Existing works use statistically summarized minimum and maximum values ​​during the calibration or training process. This is because the minimum and maximum values ​​are not differentiable and therefore cannot be learned from the training process of the neural network model.

[0007]

[0007] The main limitation in previous work is that the quantization range determined by the minimum and maximum values ​​is statistically summarized during the calibration or training process. This may be sufficient if the training data is normalized, but many deep learning applications, such as reinforcement learning (RL), may not have well-defined inputs, and therefore the input data is not normalized. In such cases, simply summarizing the minimum and maximum values ​​of a batch of training data is susceptible to sudden spikes in the input data. Therefore, any subsequent spikes in the intermediate layers will dramatically increase the minimum and maximum values ​​of the quantization range, adversely affecting the training loss and accuracy of the neural network.

[0008]

[0008] The technical problem to be solved by the present invention is that the method and system disclosed herein address the limitations of the prior work and optimize the minimum and maximum values ​​of the quantization range. In contrast to the methods used in the prior work, the minimum and maximum values ​​are formulated with analytical functions that serve the purpose of clipping values ​​that exceed the range defined by the minimum and maximum values. Furthermore, the quantization range is minimized so that the resolution of the quantization from floating-point numbers to integers can be optimized.

[0009]

[0009] Another technical problem to be solved by the present invention is that the method and system disclosed in this application quantize the neural network model to reduce the calculation time in model inference. A method is introduced to optimize the clipping function that defines the quantization range of the weight parameters and activations of the neural network. By using this method, the clipping function is optimized so that it can reach a narrow quantization range to ensure the optimal quantization resolution.

[0010]

[0010] According to a first aspect of the present application, a method for quantizing a neural network includes the steps of: clipping values ​​used in the neural network beyond a range from a minimum value to a maximum value; simulating the quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing the values ​​used in the neural network according to the updated minimum and maximum values.

[0011]

[0011] In some embodiments, the method of quantizing a neural network further includes minimizing the range during training.

[0012] According to a second aspect of the present application, an electronic device includes one or more processing units, a memory, and a number of programs stored in the memory, the programs, when executed by the one or more processing units, causing the electronic device to perform one or more of the methods described above.

[0013]

[0013] According to a third aspect of the present application, a non-transitory computer-readable storage medium stores a number of programs for execution by an electronic device having one or more processing units, the programs, when executed by the one or more processing units, causing the electronic device to perform one or more of the methods described above.

[0014]

[0014] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not all-inclusive, and many additional features and advantages will be apparent to those skilled in the art, especially when considering the drawings, the specification, and the claims. Furthermore, it should be noted that the words used in this specification are primarily selected for readability and instructional purposes, and may not be selected to describe or limit the subject matter of this specification. [Brief description of the drawings]

[0015]

[0015] In order that the present disclosure may be more particularly understood, a more particular description will now be made by reference to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the accompanying drawings merely illustrate the relevant features of the present disclosure and therefore should not be considered limiting, since the present description may allow for other useful features. [Figure 1]

[0016] FIG. 1 illustrates an exemplary computing environment in which one or more networked client devices and one or more server systems interact with each other locally or remotely via one or more communications networks in accordance with some implementations of the present disclosure. [Diagram 2]

[0017] FIG. 2 is an example neural network that may be implemented to process data in a neural network model according to some implementations of the present disclosure. [Figure 3A]

[0018] FIG. 3A illustrates an example symmetric clipping scenario with respect to α and β for shaping clipping functions according to some implementations of this disclosure. [Figure 3B]

[0019] FIG. 3B illustrates an example asymmetric clipping scenario with respect to α and β for shaping clipping functions according to some implementations of the present disclosure. [Figure 3C]

[0020] FIG. 3C illustrates an example positive clipping scenario for α and β for shaping clipping functions according to some implementations of the present disclosure. [Figure 3D]

[0021] FIG. 3D illustrates an example negative clipping scenario for α and β for shaping clipping functions according to some implementations of the present disclosure. [Figure 4]

[0022] FIG. 4 illustrates the workflow and structural components of a neural network quantization method according to some implementations of the present disclosure. [Diagram 5]

[0023] FIG. 5 is a block diagram illustrating an example process for quantizing a neural network according to some implementations of the present disclosure.

[0024] According to common practice, the various features illustrated in the figures may not be drawn to scale. As such, dimensions of the various features may be arbitrarily expanded or reduced for clarity. Further, some drawings may not depict all of the components of a given system, method, or apparatus. Finally, like reference numerals may be used throughout the specification and figures to denote like features. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016]

[0025] Reference is made to specific implementation details, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to one skilled in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to one skilled in the art that the subject matter presented herein may be practiced in many types of electronic devices.

[0017]

[0026] Before describing the embodiments of the present application in further detail, the names and terms associated with the embodiments of the present application are described, and the names and terms associated with the embodiments of the present application have the following meanings.

[0018]

[0027] NN(Neural Networks): Neural networks

[0028] FP32 (32-bit Floating Point): 32-bit floating point

[0029] INT8 (8-bit Integer): 8-bit integer

[0030] PTQ (Post-Training Quantization): Post-Training Quantization

[0031] QAT (Quantization-Aware Training): Training that takes quantization into account

[0032] SIMD (Single Instruction Multiple Data): Single instruction multiple data

[0033] SGD (Stochastic Gradient Descend): Stochastic gradient descent method

[0034] RL (Reinforcement Learning):

[0035] min(minimum): minimum value

[0036] max(maximum): Maximum value

[0037] FIG. 1 illustrates an exemplary computing environment 100 in which one or more networked client devices 102 and one or more server systems 104 interact with each other locally or remotely via one or more communications networks 106 in accordance with some implementations of the present disclosure.

[0019]

[0038] In some embodiments, a server system 104, such as 104A and 104B, is physically separate from one or more client devices 102 but communicatively coupled thereto. In some embodiments, a client device 102 (e.g., 102A, 102B) comprises a desktop computer. In some embodiments, a client device 102 (e.g., 102C) comprises a mobile device, such as a mobile phone, a tablet computer, and a laptop computer. Each client device 102 can collect data or user input, execute user applications, and present output on its user interface. The collected data or user input can be processed locally at the client device 102 and / or remotely by the server 104. Each client device 102 communicates with another client device 102 or with the server system 104 using one or more communications networks 106. The communication network 106 may be one or more networks having one or more types of topologies, including, but not limited to, the Internet, an intranet, a local area network (LAN), a cellular network, an Ethernet, a telephone network, a Bluetooth personal area network (PAN), and the like. In some embodiments, two or more client devices 102 in a sub-network are coupled via a wired connection, while at least some client devices 102 in the same sub-network are coupled via a local wireless communication network (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and other wireless communication networks). In one example, the client device 102 establishes a connection to one or more communication networks 106 directly (e.g., using 3G / 4G connectivity to a wireless carrier), or through a network interface (e.g., a router, switch, gateway, hub, or intelligent dedicated whole-home control node), or through any combination thereof.

[0020]

[0039] Each of the server systems 104 includes one or more processors 110 and memory that stores instructions for execution by the one or more processors 110. The server systems 104 also include an input / output interface to the client as 114. The one or more server systems 104 provide system data (e.g., boot files, operating system images, and user applications) to the client devices 102 and, in some embodiments, process data and user input received from the client devices 102 when the user applications execute on the client devices 102. The one or more server systems 104 may enable real-time data communication with client devices 102 that are remote from each other or from the one or more server systems 104. The server system 104A is configured to store the data storage 112. The server system 104B is configured to store the neural network model 116. In some embodiments, the neural network model and the data storage may be within the same server 104. The neural network training method may be implemented on one or more server systems 104.

[0021]

[0040] The client device 102 includes one or more processors and memory that stores instructions for execution by the one or more processors. The instructions stored on the client device 102 enable implementation of a user interface application and a web browser on the server 104. The web browser and user interface application are linked to a user account in the computing environment 100.

[0022]

[0041] Neural network training techniques are applied in the computing environment 100 to process data obtained by an application running on the client device 102 or data loaded from another data storage or file to identify information contained in the data, match the data with other data, classify the data, or synthesize related data. The data may include text, images, audio, video, etc. A neural network model is trained with the training data before being applied to process the data. In some embodiments, the neural network model training method is implemented on the client device 102. In some embodiments, the neural network model training method is implemented jointly on the client device 102 and the server system 104. In some embodiments, the neural network model may be maintained on the client device 102. In some embodiments, the client device 102 is configured to identify, classify, or modify data information from the data storage 112 or from the neural network model 116 automatically and without user intervention.

[0023]

[0042] In some embodiments, both model training and data processing are performed locally at each individual client device 102 (e.g., client device 102C). Client device 102C retrieves training data from one or more server systems 104, including data storage 112, and applies the training data to train the neural network model. Following model training, client device 104C retrieves and processes data locally using the trained neural network model. Alternatively, in some embodiments, both model training and data processing are performed remotely at a server system 104 (e.g., server system 104B) associated with a client device 102 (e.g., client device 102A). Server 104B retrieves training data from itself, another server 104, or data storage 112, and applies the training data to train the neural network model 116. The client device 102A acquires data, transmits the data to the server 104B (e.g., within an application) for data processing using the trained neural network model, receives data processing results from the server 104B, and presents the results in a user interface (e.g., associated with the application). The client device 102A itself performs no or little data processing on the data before transmitting the data to the server 104B. Furthermore, in some embodiments, data processing is performed locally at the client device 102 (e.g., client device 102B), while model training is performed remotely at a server system 104 (e.g., server 104B) associated with the client device 102B. The trained neural network model is optionally stored in another data storage, such as the server 104B or 112.The client device 102B imports the trained neural network model from the server 104B or the data storage 112, processes data using the neural network model, and generates data processing results that are presented locally in a user interface.

[0043] The neural network model system 116 includes one or more of a server, a client device, storage, or a combination thereof. The neural network model system 116 typically includes one or more processing units (CPUs), one or more network interfaces, memory, and one or more communication buses for interconnecting these components (sometimes referred to as chipsets). The neural network model system 116 includes one or more input devices for facilitating user input, such as a keyboard, a mouse, a voice command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capture camera, or other input buttons or controls. The neural network model system 116 also includes one or more output devices for enabling presentation of a user interface and display content, including one or more speakers and / or one or more visual displays.

[0024]

[0044] The memory includes high speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. The memory optionally includes one or more storage devices located remotely from the one or more processing units. The memory, or alternatively the non-volatile memory in the memory, includes a non-transitory computer readable storage medium. In some embodiments, the memory, or the non-transitory computer readable storage medium of the memory, stores programs, modules, and data structures, including an operating system, an input processing module for detecting and processing input data, a model training module for receiving training data and establishing a neural network model for processing the data, a neural network module for processing the data using the neural network model, and the like.

[0045] Each of the above-identified elements may be stored in one or more of the memory devices discussed above and corresponds to a set of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) are not required to be implemented as separate software programs, procedures, modules, or data structures, and thus various subsets of these modules may be combined or otherwise rearranged in various embodiments. In some embodiments, the memory optionally stores a subset of the above-identified modules and data structures. Additionally, the memory optionally stores additional modules and data structures not described above.

[0025]

[0046] 2 is an example neural network 200 that may be implemented to process data in the neural network model 116 according to some implementations of the present disclosure. The neural network model 116 is built based on the neural network 200. A corresponding model-based processing module in the server system 104B applies the neural network model 116, including the neural network 200, to process data.

[0026]

[0047] In some examples, the neural network 200 includes a collection of neuron nodes 220 connected by links 212. Each neuron node 220 receives one or more neuron node inputs and applies a propagation function to generate a neuron node output from the one or more neuron node inputs. A weight associated with each link 212 is applied to the neuron node output as the neuron node output is transmitted through one or more links 212 to one or more other neuron nodes 220. The one or more neuron node inputs are combined based on the corresponding weights according to the propagation function. In one example, the propagation function is the product of a nonlinear activation function and a linear weighted combination of one or more neuron node inputs.

[0027]

[0048] A neural network consists of one or more layers having neuron nodes 220. In some embodiments, the one or more layers include a single layer that acts as both an input layer and an output layer. In some embodiments, the one or more layers include an input layer 202 for receiving input, an output layer 206 for generating output, and zero or more hidden / latent layers 204 (e.g., 204A and 204B) between the input layer 202 and the output layer 206. A deep neural network has more than one hidden layer 204 between the input layer 202 and the output layer 206. In the neural network 200, each layer is only connected to its immediately preceding layer and / or the layer immediately following it. In some embodiments, layer 202 or 204B is a fully connected layer because each neuron node 220 in layer 202 or 204B is connected to all neuron nodes 220 in the layer immediately following it.

[0028]

[0049] In some embodiments, one or more neural networks may be used by the neural network model 116, including a fully connected neural network, a multi-layer perceptron, a convolutional neural network, a recurrent neural network, a feed-forward neural network, a radial basis function neural network, a Long Short-Term Memory (LSTM), an auto-encoder, and a sequence-to-sequence model, among others.

[0029]

[0050] The training process is a process for calibrating all weights of each layer of the learning model using a training data set provided to the input layer 202. The training process typically includes two steps (forward propagation and back propagation) that are repeated multiple times until a preset convergence condition is met. In forward propagation, a set of weights for various layers is applied to the input data and intermediate results from the previous layers. In back propagation, the margin of error (e.g., loss function) of the output is measured, and the weights are adjusted accordingly to reduce the error. The activation function is optionally linear, normalized linear unit, sigmoid, hyperbolic tangent, or other type. In some embodiments, a network bias term is added to the sum of the weighted outputs from the previous layers before the activation function is applied. The network bias provides a perturbation that helps the neural network 200 avoid overfitting the training data. The results of the training include the network bias parameters for each layer.

[0030]

[0051] The methods and systems disclosed herein have many advantages, including: differentiable (or trainable or learnable) minimum and maximum quantization parameters are more robust to variations in the input data and can quantize neural network models with greater accuracy compared to quantization techniques in which the quantization parameters are simply statistically summarized from batch data.

[0031]

[0052] In some embodiments, the clipping functions disclosed herein are generalized solutions and are applicable to symmetric or asymmetric quantization.

[0032]

[0053] In some embodiments, the clipping function may be applied during the training process along with an additional L2 regularization method to minimize the quantization range determined by the minimum and maximum values ​​and improve the quantization resolution.

[0033]

[0054] In some embodiments, the values ​​of the methods and systems of the present invention are applied to weights and intermediate features in a neural network. Values ​​are not affected if they are within the range between a minimum and maximum value. The analytical function is defined as follows:

[0034]

number

[0035]

[0056] In some embodiments, the analysis clipping function has the following characteristics:

[0036]

number

[0056] In some examples, the minimum and maximum values ​​are determined automatically during a training process. Figures 3A-3D show different scenarios of how different α and β shape clipping functions according to some implementations of the present disclosure.

[0037] 3A illustrates an example symmetric clipping scenario for α and β for forming a clipping function according to some implementations of the present disclosure, e.g., α=−6, β=6.

[0038] 3B illustrates an example asymmetric clipping scenario for α and β for forming a clipping function according to some implementations of the present disclosure, e.g., α=−6, β=2.

[0039] 3C illustrates an example positive clipping scenario for α and β for forming a clipping function according to some implementations of the present disclosure, e.g., α=2, β=6.

[0040] 3D illustrates an example negative clipping scenario for α and β for forming a clipping function according to some implementations of the present disclosure, e.g., α=−6, β=−2.

[0041]

[0057] In some embodiments, additional techniques are used to minimize the range determined by α and β, i.e., |α-β|, along with the analytical clipping function. For example, an L2 regularization method to minimize the quantization range is applied to the loss function:

[0042]

number

[0058] In some embodiments, the goal of the L2 regularization method is to constrain the model to have optimal and minimal quantization ranges for the parameters and activations of all layers in the network.

[0043]

[0059] In some embodiments, a new analytical function is used to define generalized differentiable minimum and maximum values ​​for quantizing neural network models, which provides symmetric or asymmetric quantization, resulting in a more robust solution against accuracy degradation caused by spikes in the input data and hidden layer activations.

[0044]

[0060] FIG. 4 illustrates the workflow and structural components of a neural network quantization method according to some implementations of the present disclosure.

[0045]

[0061] In some embodiments, the value to be quantized 410 is provided to a clipping function 420. The clipped value 430 from the clipping function 420 is further provided 440 to a quantization process along with α and β, and the output of the quantization process 440 is the quantized value 450.

[0046]

[0062] In some embodiments, a personal computer (PC) or mobile device performs the training and inference of the neural network model.

[0047]

[0063] In some embodiments, the quantization process follows the following steps: First, for each layer of the neural network, the layer parameters and activations (e.g., values ​​to be quantized 410 as shown in FIG. 4) are fed in a forward propagation stage to a clipping function (e.g., clipping function 420 as shown in FIG. 4) defined in Eq. 1, respectively. The method clips the values ​​(e.g., values ​​to be quantized 410 as shown in FIG. 4) to a range of minimum and maximum values ​​determined by the clipping function (e.g., clipping function 420 as shown in FIG. 4) using parameters α and β.

[0048]

[0064] Second, the weight parameters and intermediate activations in each layer (e.g., clipped values ​​430 as shown in FIG. 4) are quantized using a fake or simulation quantization method, where the operations are still performed in FP32 format, but are quantized to INT8 values ​​(e.g., quantized values ​​450 as shown in FIG. 4) to mimic the behavior of INT8 operations. In a fake or simulation quantization method, the quantization process is mimicked.

[0049]

[0065] In some embodiments, an L2 regularization method is added to the loss to optimize the model accuracy, as shown in Eq. 2. In some cases, L2 regularization is optional and can be used to minimize the quantization range, resulting in an optimal high quantization resolution.

[0050]

[0066] In some embodiments, during the training process, the minimum and maximum values ​​are updated along with the neural network weight parameters through backpropagation using a gradient-based numerical method such as SGD, which updates the values ​​of α and β during the training process to converge to an optimal solution (e.g., a quantization process using α and β 440 as shown in FIG. 4).

[0051]

[0067] In some embodiments, the structural components of the disclosed methods interact with each other. For example, in forward propagation, the output from the clipping function is fed into a simulated quantization process. In back propagation, a training process updates the values ​​of α and β. L2 regularization is part of the loss function that guides the optimization of the values ​​of α and β.

[0052]

[0068] In some embodiments, the data used in the disclosed methods and systems include input training data and neural network weight parameters. Both types of data are quantized so that operations in model inference can be performed in a desired quantization format, such as from FP32 to INT8. In each layer, values ​​to be quantized, such as layer weights and activations, are first clipped using the clipping function shown in Eq. 1, and then the clipped values ​​are quantized using a fake quantization method to mimic the quantization process. After calibration with PTQ or training with QAT is completed, the model inputs and model weights are converted from FP32 to INT8, so that model storage and inference can be performed in the target format, such as INT8.

[0053]

[0069] Alternatively, in some embodiments, quantization can be applied in a per-channel fashion, such that each channel has its own quantization parameter. For example, if the layer is a 2D convolutional layer whose weight dimension is [N k ,N k ,N i ,N o ] and the input activation dimension is [N x ,N y ,N i ] and the output activation dimension is [N x ’ ,N y ’ ,N o ], commonly used in per-layer quantization, the quantization parameters are triplet scalars applied to the weights, input activations, and output activations, respectively. In per-channel quantization, the scalar quantization parameter pairs are replaced by vector pairs, where each element of the vector is aligned to the channel dimension, and each channel is quantized by a different range.

[0054]

[0070] FIG. 5 is a block diagram illustrating an example process 500 for quantizing a neural network according to some implementations of the present disclosure.

[0055]

[0071] The process 500 of quantizing a neural network includes a step 502 of clipping values ​​used in the neural network that are outside a range from a minimum to a maximum value.

[0056]

[0072] The process 500 includes a step 504 of simulating a quantization process using the clipped values.

[0057]

[0073] The process 500 then includes a step 506 of updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process.

[0058]

[0074] The process 500 additionally includes a step 508 of quantizing values ​​used in the neural network according to the updated minimum and maximum values.

[0059]

[0075] For example, the minimum and maximum values ​​are formulated in an analytical function that acts to clip values ​​that are outside the range defined by the minimum and maximum values, which are then applied to the weights and intermediate feature values ​​in the neural network.

[0060]

[0076] In some embodiments, process 500 additionally includes a step of minimizing ranges during training 510. For example, the quantization ranges are minimized so that the resolution from quantization from floating point numbers to integers can be optimized.

[0061]

[0077] In some embodiments, the values ​​used in the neural network include one or more of the values ​​of weights, layer activations, and intermediate features in the neural network. For example, the values ​​are applied to the values ​​of weights and intermediate features in the neural network. Values ​​to be quantized, such as layer weights and layer activations, are first clipped using a clipping function.

[0062]

[0078] In some embodiments, clipping 502 values ​​used in the neural network beyond a minimum to maximum range is performed by the following clipping function:

[0063]

number

[0079] where α is the minimum value, β is the maximum value, x is the value used in the neural network, and f(x) is the clipping function, i.e. if a value is in the range between the minimum and maximum values, it will not be affected.

[0064]

[0080] In some embodiments, clipping 502 values ​​used within the neural network beyond a minimum to maximum range includes at least one of symmetric clipping, asymmetric clipping, positive clipping, and negative clipping. For example, the minimum and maximum values ​​are determined automatically during training, and Figures 3A-3D show various scenarios of how different α and β may form the clipping function.

[0065]

[0081] In some embodiments, minimizing the range during training (510) includes: minimizing the range during training using L2 regularization applied to the loss function during training. For example, a clipping function can be applied with additional L2 regularization during training to minimize the range determined by the min / max range and increase the quantization resolution. An additional technique is used to minimize the range determined by α and β, i.e., |α-β|. The L2 regularization to minimize the quantization range is applied to the loss function.

[0066]

[0082] In some embodiments, clipping 502 values ​​used within the neural network that are outside the range of a minimum to a maximum value is performed in forward propagation.

[0067]

[0083] In some embodiments, updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process (506) is performed with backpropagation.

[0068]

[0084] In some embodiments, simulating a quantization process using clipped values ​​(504) includes operating simulated quantization in FP32 format and quantizing the clipped values ​​to INT8 format, for example, weight parameters and intermediate activations in each layer are quantized using a fake quantization method, where operations are still performed in FP32 format, but quantized to INT8 values ​​to mimic the behavior of INT8 operations.

[0069]

[0085] In some embodiments, clipping (502) values ​​used in the neural network beyond a range from a minimum to a maximum value includes: for each channel of the plurality of channels, clipping each value used in the neural network beyond a respective range from a respective minimum to a respective maximum value. For example, quantization can be applied in a per-channel manner, where each channel has its own quantization parameter. In per-channel quantization, pairs of scalar quantization parameters are replaced by pairs of vectors, where each element is aligned to the channel dimension, and each channel is quantized at a different range.

[0070]

[0086] Further embodiments include various subsets of the above embodiments, including those illustrated in FIGS. 1-5, which may be combined or otherwise rearranged in various other embodiments.

[0071]

[0087] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium, which includes any medium that facilitates transfer of a computer program from one place to another, for example according to a communication protocol. As such, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the implementations described in the present application. A computer program product may include a computer-readable medium.

[0072]

[0088] The terms used in the description of implementations herein are for the purpose of describing particular implementations only and are not intended to limit the scope of the claims. As used in the description of implementations and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and / or "comprising" as used herein specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0073]

[0089] It will also be understood that although terms such as first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first electrode can be called a second electrode, and similarly, a second electrode can be called a first electrode, without departing from the scope of the implementation. Although a first electrode and a second electrode are both electrodes, they are not the same electrode.

[0074]

[0090] The description of the present application has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications, variations, and alternative implementations will be apparent to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. The embodiments have been chosen and described in order to best explain the principles of the invention, its practical applications, and to enable those skilled in the art to understand the invention for various implementations and to best utilize the underlying principles and various implementations with various modifications as appropriate for the particular applications contemplated. It is therefore to be understood that the scope of the claims is not limited to the particular examples of implementations disclosed, and that modifications and other implementations are intended to be encompassed within the scope of the appended claims.

[0075]

[0091] <<Additional Notes>> (Appendix 1) 1. A method of quantizing a neural network, comprising: clipping values ​​used in said neural network that are outside a range of a minimum value to a maximum value; simulating the quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing values ​​used in said neural network according to the updated minimum and maximum values; The method includes:

[0076] (Appendix 2) The method of claim 1, further comprising: minimizing said range during said training; The method includes:

[0077] (Appendix 3) 2. The method of claim 1, wherein the values ​​used in the neural network include values ​​of one or more of weights, layer activations, and intermediate features in the neural network.

[0078] (Appendix 4) 2. The method of claim 1, wherein the step of clipping values ​​used in the neural network that are outside the minimum to maximum range comprises using a clipping function:

[0079]

number

[0080] (Appendix 5) 2. The method of claim 1, wherein clipping values ​​used in the neural network that exceed the minimum to maximum range includes at least one of symmetric clipping, asymmetric clipping, positive clipping, and negative clipping.

[0081] (Appendix 6) 3. The method of claim 2, wherein the step of minimizing the range during training comprises: minimizing the range during the training using an L2 regularization applied to a loss function during the training; A method comprising:

[0082] (Appendix 7) 2. The method of claim 1, wherein the step of clipping values ​​used in the neural network that are outside the minimum to maximum range is performed in forward propagation.

[0083] (Appendix 8) 2. The method of claim 1, wherein the step of updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process is performed by backpropagation.

[0084] (Appendix 9) 2. The method of claim 1, wherein simulating a quantization process using the clipped values ​​includes performing a simulated quantization operation in FP32 format and quantizing the clipped values ​​to INT8 format.

[0085] (Appendix 10) 2. The method of claim 1, wherein clipping values ​​used in the neural network beyond a range from a minimum value to a maximum value includes clipping, for each channel of a plurality of channels, each value used in the neural network beyond a respective range from a respective minimum value to a respective maximum value.

[0086] (Appendix 11) 1. An electronic device comprising: one or more processing units; a memory coupled to the one or more processing units; and a plurality of programs stored in the memory, the plurality of programs, when executed by the one or more processing units, causing the electronic device to perform a plurality of operations for quantizing a neural network, the plurality of operations including: clipping values ​​used in said neural network that are outside a range of a minimum value to a maximum value; simulating the quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing values ​​used in said neural network according to the updated minimum and maximum values; 23. An electronic device comprising:

[0087] (Appendix 12) 12. The electronic device of claim 11, wherein the operations of quantizing the neural network further include: minimizing said range during said training; 23. An electronic device comprising:

[0088] (Appendix 13) 12. The electronic device of claim 11, wherein the values ​​used in the neural network include values ​​of one or more of weights, layer activations, and intermediate features in the neural network.

[0089] (Appendix 14) 12. The electronic device of claim 11, wherein the step of clipping values ​​used in the neural network that are outside the minimum to maximum range comprises using a clipping function:

[0090]

number

[0091] (Appendix 15) 12. The electronic device of claim 11, wherein clipping values ​​used in the neural network beyond the minimum to maximum range includes at least one of symmetric clipping, asymmetric clipping, positive clipping, and negative clipping.

[0092] (Appendix 16) 13. In the electronic device of claim 12, the step of minimizing the range during the training comprises: minimizing the range during the training using an L2 regularization applied to a loss function during the training; 23. An electronic device comprising:

[0093] (Appendix 17) 12. The electronic device of claim 11, wherein the step of clipping values ​​used in the neural network that are outside the minimum to maximum range is performed in forward propagation.

[0094] (Appendix 18) 12. The electronic device of claim 11, wherein updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process is performed by backpropagation.

[0095] (Appendix 19) 1. A non-transitory computer readable storage medium storing a plurality of programs for execution by an electronic device having one or more processing units, the plurality of programs, when executed by the one or more processing units, causing the electronic device to perform a plurality of operations for quantizing a neural network, the plurality of operations including: clipping values ​​used in said neural network that are outside a range of a minimum value to a maximum value; simulating the quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing values ​​used in said neural network according to the updated minimum and maximum values; A non-transitory computer readable storage medium comprising:

[0096] (Appendix 20) 20. The non-transitory computer-readable storage medium of claim 19, wherein the operations of quantizing the neural network further include: minimizing said range during said training; A non-transitory computer readable storage medium comprising:

Claims

1. 1. A method of quantizing a neural network, comprising: clipping values ​​used in said neural network beyond a range of a minimum to a maximum value; simulating a quantization process using the clipped values; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing values ​​used in the neural network according to the updated minimum and maximum values; The method includes:

2. 13. The method of claim 1 further comprising: minimizing the range during the training; The method includes:

3. 2. The method of claim 1, wherein the values ​​used in the neural network include values ​​of one or more of weights, layer activations, and intermediate features in the neural network.

4. 2. The method of claim 1, wherein the step of clipping values ​​used in the neural network beyond the minimum to maximum range comprises using a clipping function: [0010] where α is the minimum value, β is the maximum value, x is a value used in the neural network, and f(x) is the clipping function.

5. 2. The method of claim 1, wherein clipping values ​​used in the neural network beyond the minimum to maximum range includes at least one of symmetric clipping, asymmetric clipping, positive clipping, and negative clipping.

6. 3. The method of claim 2, wherein the step of minimizing the range during training comprises: minimizing the range during the training using an L2 regularization applied to a loss function during the training; A method comprising:

7. 2. The method of claim 1, wherein the step of clipping values ​​used in the neural network that are outside the minimum to maximum range is performed in forward propagation.

8. 2. The method of claim 1, wherein the step of updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process is performed by backpropagation.

9. 2. The method of claim 1, wherein simulating a quantization process using the clipped values ​​comprises performing a simulated quantization operation in FP32 format and quantizing the clipped values ​​to INT8 format.

10. 2. The method of claim 1, wherein clipping values ​​used in the neural network beyond a range from a minimum value to a maximum value comprises clipping, for each channel of a plurality of channels, each value used in the neural network beyond a respective range from a respective minimum value to a respective maximum value.

11. 11. An electronic device comprising one or more processing units, a memory coupled to the one or more processing units, and a plurality of programs stored in the memory, the plurality of programs, when executed by the one or more processing units, causing the processing unit to perform a method according to any one of claims 1 to 10.

12. A computer program product which, when run on a computer, causes the computer to carry out the method according to any one of claims 1 to 10.

Images