Quantization method for accelerating reasoning of neural network

By limiting the quantization range and updating the minimum and maximum values ​​during training of neural networks, the problem of reduced accuracy in FP32 to INT8 conversion is solved, achieving higher accuracy and lower computing time, and enhancing The robustness of the model.

CN118901068APending Publication Date: 2024-11-05TENCENT AMERICA LLC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202280093048.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-11
Filing Date
2022-08-26
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

When converting neural network models from FP32 format to INT8 format, the numerical accuracy loss leads to a decrease in model accuracy and the minimum and maximum values ​​of the quantization range cannot be effectively optimized, resulting in increased computing time and training losses.

Method used

By limiting the minimum and maximum values ​​of the quantization range during training of the neural network, optimize the quantization range using the limiting function to make it more refined, thereby improving the quantization resolution and updating the minimum and Maximum value to accommodate fluctuations in input data.

Benefits of technology

It achieves higher accuracy and lower computing time in neural network models, improves quantitative resolution, and enhances the robustness of the model to input data fluctuations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118901068A_ABST
    Figure CN118901068A_ABST
Patent Text Reader

Abstract

An electronic device performs a method of quantizing a neural network. The method comprises the following steps: carrying out amplitude limiting on values which are used in a neural network and exceed a range from a minimum value to a maximum value; simulating a quantization process using the clipped value; during the training period of the neural network, updating the minimum value and the maximum value to optimize the quantization processing; and quantizing the value used in the neural network according to the updated minimum value and maximum value. In some embodiments, the method of quantizing the neural network further includes minimizing the range during training.
Need to check novelty before this filing date? Find Prior Art

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation application of U.S. patent application No. 17 / 693,270, filed on March 11, 2022, entitled “QUANTIZATION METHOD FOR ACCELERATING THE INFERENCE OF NEURAL NETWORKS,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to data processing techniques, and in particular, to quantization methods for accelerating inference of neural network systems. Background Art

[0003] Quantization in neural networks uses fewer bits to represent numerical values ​​in the storage and calculation of neural networks. For example, a neural network trained in a 32-bit floating point (FP32) precision format can be converted to a format using an 8-bit signed integer (INT8). This results in a 4x reduction in model storage and memory usage. In addition, a Single Instruction Multiple Dataset (SIMD) mechanism can be used to implement running model reasoning in INT8 format, where a single instruction such as multiplication can be executed simultaneously on four 8-bit integers instead of one 32-bit floating point. This results in a 75% reduction in computation time.

[0004] The main challenge of converting a neural network model from FP32 format to INT8 format is the reduction in model accuracy due to the loss of numerical precision. There are various ways to solve this problem: Post-training Quantization (PTQ) or Quantization-aware Training (QAT) methods can be used to recover the accuracy loss to a certain extent. The PTQ method uses a representative training data set and adjusts the minimum and maximum values ​​of the quantization range in FP32 format. In QAT, the minimum and maximum values ​​of the quantization range are adjusted during the training process, and the weights of the neural network model are fine-tuned. The common part of PTQ and QAT is to adjust the minimum and maximum ranges of layer weights and layer activations to facilitate the recovery of model accuracy loss caused by quantization. In fact, during the training process, the minimum and maximum values ​​are updated according to the batch statistics. Summary of the invention

[0005] In order to overcome the deficiencies or shortcomings of the above-mentioned methods, improved systems and methods for accelerating the inference of neural network systems are needed.

[0006] No prior work provides a solution to optimize the minimum and maximum values ​​of the quantization range. Existing work uses the minimum and maximum values ​​statistically summarized during the calibration or training process. This is because the minimum and maximum values ​​are non-differentiable and therefore cannot be learned from the training process for the neural network model.

[0007] The main limitation of previous work is the statistical summary of the quantization range determined by the minimum and maximum values ​​during the calibration or training process. Although this may be sufficient if the training data is normalized, many deep learning applications such as reinforcement learning (RL) may not have well-defined inputs, and therefore the input data is not normalized. In such cases, simply summarizing the minimum and maximum values ​​of a batch of training data is susceptible to sudden spikes in the input data. As a result, it subsequently causes some spikes in the intermediate layers, resulting in a significant increase in the minimum and maximum values ​​of the quantization range and a negative impact on the training loss and accuracy of the neural network.

[0008] The technical problem to be solved by the present invention is that the method and system disclosed herein solve the limitations in previous work and optimize the minimum and maximum values ​​of the quantization range. In contrast to the methods used in previous work, the minimum and maximum values ​​are formulated in an analytical function for the purpose of clipping values ​​that exceed the range defined by the minimum and maximum values. In addition, the quantization range is minimized so that the resolution from quantized floating point numbers to integers can be optimized.

[0009] Another technical problem to be solved by the present invention is that the method and system disclosed herein quantizes a neural network model to reduce the computation time in model inference. A method for optimizing a clipping function that limits the quantization range of neural network weight parameters and activations is introduced. Using this method, the clipping function is optimized so that a narrow quantization range can be achieved to ensure optimal quantization resolution.

[0010] According to a first aspect of the present application, a method for quantizing a neural network includes: limiting values ​​used in the neural network that are outside a range from a minimum value to a maximum value; using the limited values ​​to simulate a quantization process; updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process; and quantizing the values ​​used in the neural network based on the updated minimum and maximum values.

[0011] In some implementations, the method of quantizing a neural network further comprises: minimizing the range during training.

[0012] According to a second aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. When the programs are executed by the one or more processing units, the electronic device executes one or more methods as described above.

[0013] According to a third aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs executed by an electronic device having one or more processing units. When the programs are executed by the one or more processing units, the electronic device executes one or more methods as described above.

[0014] Note that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in the specification are not all inclusive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art from the drawings, description, and claims. In addition, it should be noted that the language used in this specification is primarily selected for readability and instructional purposes, and is not selected to describe or limit the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order that the present disclosure may be understood in more detail, a more specific description may be made by reference to the features of various embodiments, some of which are shown in the accompanying drawings. However, the accompanying drawings only illustrate the relevant features of the present disclosure and are therefore not to be considered limiting, as the description may allow for other effective features.

[0016] Figure 1 is an exemplary computing environment according to some implementations of the present disclosure in which one or more network-connected client devices and one or more server systems interact with each other locally or remotely via one or more communication networks.

[0017] Figure 2 is an exemplary neural network implemented to process data in a neural network model according to some implementations of the present disclosure.

[0018] Figure 3A An exemplary symmetric clipping scenario is shown in which α and β form a clipping function according to some implementations of the present disclosure.

[0019] Figure 3B An exemplary asymmetric clipping scenario is shown in which α and β form a clipping function according to some implementations of the present disclosure.

[0020] Figure 3C An exemplary positive clipping scenario is shown in which α and β form a clipping function according to some implementations of the present disclosure.

[0021] Figure 3D An exemplary negative clipping scenario is shown in which α and β form a clipping function according to some implementations of the present disclosure.

[0022] Figure 4 The workflow and structural components of a neural network quantization method according to some implementations of the present disclosure are shown.

[0023] Figure 5 is a block diagram illustrating an exemplary process for quantizing a neural network according to some implementations of the present disclosure.

[0024] According to common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, similar reference numerals may be used to represent similar features throughout the specification and drawings. DETAILED DESCRIPTION

[0025] Reference will now be made in detail to specific implementations, examples of which are shown in the accompanying drawings. In the detailed description that follows, many non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on many types of electronic devices.

[0026] Before describing the embodiments of the present application in further detail, the names and terms involved in the embodiments of the present application are described, and the names and terms involved in the embodiments of the present application have the following explanations.

[0027] NN: Neural Network

[0028] FP32: 32-bit floating point

[0029] INT8: 8-bit integer

[0030] PTQ: Post-training quantization

[0031] QAT: Quantization Aware Training

[0032] SIMD: Single Instruction Multiple Data

[0033] SGD: Stochastic Gradient Descent

[0034] RL: Reinforcement Learning

[0035] min: minimum value

[0036] max: maximum value

[0037] Figure 1is an exemplary computing environment 100 in which one or more network-connected client devices 102 and one or more server systems 104 interact with each other locally or remotely via one or more communication networks 106 according to some implementations of the present disclosure.

[0038] In some embodiments, the server system 104, such as 104A and 104B, is physically remote but communicatively coupled to one or more client devices 102. In some embodiments, the client devices 102 (e.g., 102A, 102B) include desktop computers. In some embodiments, the client devices 102 (e.g., 102C) include mobile devices, such as mobile phones, tablet computers, and laptop computers. Each client device 102 can collect data or user input, execute user applications, and present output on its user interface. The collected data or user input can be processed locally at the client device 102 and / or remotely by the server 104. Each client device 102 communicates with another client device 102 or server system 104 using one or more communication networks 106. The communication network 106 can be one or more networks with one or more topology types, including but not limited to the Internet, intranet, local area network (Local Area Network, LAN), cellular network, Ethernet, telephone network, Bluetooth Personal Area Network (Bluetooth Personal Area Network, PAN), etc. In some embodiments, two or more client devices 102 in a subnet are coupled via a wired connection, and at least some of the client devices 102 in the same subnet are coupled via a local radio communication network (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and other radio communication networks). In an example, the client devices 102 establish a connection to one or more communication networks 106 either directly (e.g., using a 3G / 4G connection to a wireless carrier), through a network interface (e.g., a router, switch, gateway, hub, or smart, dedicated whole-home control node), or through any combination thereof.

[0039] Each of the server systems 104 includes one or more processors 110 and a memory storing instructions for execution by the one or more processors 110. The server systems 104 also include input / output interfaces such as 114 to the client. The one or more server systems 104 provide system data (e.g., boot files, operating system images, and user applications) to the client devices 102, and in some embodiments, process data and user input received from the client devices 102 when the user applications are executed on the client devices 102. The one or more server systems 104 can enable real-time data communication with client devices 102 that are remote from each other or remote from one or more server systems 104. Server system 104A is configured to store data storage 112. Server system 104B is configured to store neural network model 116. In some embodiments, the neural network model and the data storage can be in the same server 104. The neural network training method can be implemented at one or more server systems 104.

[0040] Each client device 102 includes one or more processors and memory storing instructions for execution by the one or more processors. The instructions stored on the client device 102 enable a web browser and user interface application for the server 104. The web browser and user interface application are linked to a user account in the computing environment 100.

[0041] A neural network training technique is applied to the computing environment 100 to process data obtained by an application executed at a client device 102 or loaded from another data storage device or file to identify information contained in the data, match the data with other data, classify the data, or synthesize related data. The data may include text, images, audio, video, etc. The neural network model is trained with training data before the neural network model is applied to process the data. In some embodiments, the neural network model training method is implemented at the client device 102. In some embodiments, the neural network model training method is jointly implemented at the client device 102 and the server system 104. In some embodiments, the neural network model can be stored at the client device 102. In some embodiments, the client device 102 is configured to automatically and without user intervention identify, classify or modify data information from the data storage device 112 or from the neural network model 116.

[0042] In some embodiments, both model training and data processing are implemented locally at each individual client device 102 (e.g., client device 102C). Client device 102C obtains training data from one or more server systems 104 including a data storage device 112, and applies the training data to train a neural network model. After model training, client device 104C obtains data and processes the data locally using the trained neural network model. Alternatively, in some embodiments, both model training and data processing are implemented remotely at a server system 104 (e.g., server system 104B) associated with a client device 102 (e.g., client device 102A). Server 104B obtains training data from itself, another server 104, or a data storage device 112, and applies the training data to train a neural network model 116. Client device 102A obtains data, sends the data to server 104B (e.g., in an application) for data processing using the trained neural network model, receives the data processing results from server 104B, and presents the results on a user interface (e.g., a user interface associated with an application). The client device 102A itself performs no or little data processing on the data before sending the data to the server 104B. In addition, in some embodiments, the data processing is implemented locally at the client device 102 (e.g., client device 102B), while the model training is implemented remotely at the server system 104 (e.g., server 104B) associated with the client device 102B. The trained neural network model is optionally stored in the server 104B or another data storage device such as 112. The client device 102B imports the trained neural network model from the server 104B or the data storage device 112, uses the neural network model to process the data, and generates data processing results to be presented locally on the user interface.

[0043] The neural network model system 116 includes one or more of a server, a client device, a storage device, or a combination thereof. The neural network model system 116 typically includes one or more processing units (CPUs), one or more network interfaces, memory, and one or more communication buses for interconnecting these components (sometimes referred to as a chipset). The neural network model system 116 includes one or more input devices for facilitating user input, such as a keyboard, a mouse, a voice command input unit or a microphone, a touch screen display, a touch-sensitive input board, a gesture capture camera, or other input buttons or controls. The neural network model system 116 also includes one or more output devices capable of presenting a user interface and display content, including one or more speakers and / or one or more visual displays.

[0044] The memory includes high-speed random access memory, such as DRAM (Dynamic Random Access Memory, DRAM), SRAM (Static Random Access Memory, SRAM), DDR RAM (Double Data Rate Random Access Memory, DDR RAM) or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. The memory optionally includes one or more storage devices located away from one or more processing units. The memory or alternatively the non-volatile memory in the memory includes a non-transitory computer-readable storage medium. In some embodiments, the memory or the non-transitory computer-readable storage medium of the memory stores programs, modules and data structures, including an operating system, an input processing module for detecting and processing input data, a model training module for receiving training data and establishing a neural network model for processing data, a neural network module for processing data using a neural network model, etc.

[0045] Each of the above-mentioned identified elements can be stored in one or more of the previously mentioned memory devices, and corresponds to the instruction set for performing the above-mentioned functions. The above-mentioned identified modules or programs (that is, instruction sets) do not need to be implemented as independent software programs, processes, modules or data structures, and therefore the various subsets of these modules can be combined or otherwise rearranged in various embodiments. In some embodiments, the memory optionally stores the above identified modules and the subset of data structure. In addition, the memory optionally stores additional modules and data structures not described above.

[0046] Figure 2 2 is an exemplary neural network 200 implemented to process data in a neural network model 116 according to some implementations of the present disclosure. The neural network model 116 is built based on the neural network 200. A corresponding model-based processing module within the server system 104B applies the neural network model 116 including the neural network 200 to process data.

[0047] In some examples, the neural network 200 includes a collection of neuron nodes 220 connected by links 212. Each neuron node 220 receives one or more neuron node inputs and applies a propagation function to generate a neuron node output based on the one or more neuron node inputs. When the neuron node output is transmitted to one or more other neuron nodes 220 through one or more links 212, the weight associated with each link 212 is applied to the neuron node output. According to the propagation function, one or more neuron node inputs are combined based on corresponding weights. In an example, the propagation function is the product of a nonlinear activation function and a linear weighted combination of one or more neuron node inputs.

[0048] The neural network is composed of one or more layers with neuron nodes 220. In some embodiments, one or more layers include a single layer that acts as an input layer and an output layer. In some embodiments, one or more layers include an input layer 202 for receiving input, an output layer 206 for generating output, and zero or more hidden layers / potential layers 204 (e.g., 204A and 204B) between the input layer 202 and the output layer 206. The deep neural network has more than one hidden layer 204 between the input layer 202 and the output layer 206. In the neural network 200, each layer is only connected to its immediately preceding layer and / or immediately following layer. In some embodiments, layer 202 or layer 204B is a fully connected layer because each neuron node 220 in layer 202 or layer 204B is connected to each neuron node 220 in its immediately following layer.

[0049] In some embodiments, the neural network model 116 may utilize one or more neural networks, including fully connected neural networks, multi-layer perceptrons, convolutional neural networks, recurrent neural networks, feed-forward neural networks, radial basis function neural networks, LSTM-long short-term memory, autoencoders, and sequence-to-sequence models, etc.

[0050] The training process is a process for calibrating all weights in the weights of each layer of the learning model using the training data set provided in the input layer 202. The training process generally includes two steps-forward propagation and backward propagation, which are repeated many times until the predefined convergence conditions are met. In the forward propagation, the weight sets of different layers are applied to the input data and the intermediate results from the previous layer. In the backward propagation, the error tolerance (e.g., loss function) of the output is measured, and the weights are adjusted accordingly to reduce the error. The activation function is optionally linear, rectified linear unit, sigmoid, hyperbolic tangent or other types. In some embodiments, before applying the activation function, a network bias term is added to the sum of the weighted outputs from the previous layer. The network bias provides a disturbance that helps the neural network 200 avoid overfitting the training data. The results of the training include the network bias parameters of each layer.

[0051] The methods and systems disclosed herein have many advantages. For example, differentiable (or trainable or learnable) minimum and maximum quantization parameters are more robust to fluctuations in input data, and neural network models can be quantized to a higher degree of accuracy than quantization techniques in which quantization parameters are simply statistically summarized from batches of data.

[0052] In some embodiments, the clipping functions disclosed herein are generalized solutions and are applicable to symmetric or asymmetric quantization.

[0053] In some embodiments, a clipping function may be applied during the training process along with an additional L2 regularization method to minimize the quantization range determined by the minimum and maximum values ​​and to increase the quantization resolution.

[0054] In some embodiments, the values ​​of the present methods and systems are applied to weights and intermediate features in a neural network. If the value falls within the range between the minimum and maximum values, the value is not affected. The analysis function is defined as follows: Among them, α and β define the minimum and maximum values ​​of the quantization range. Whether α or β is the minimum or maximum value of the quantization range is not predetermined. Instead, they are determined by the training process of the neural network. This gives the neural network training process some flexibility - no inequality constraints are imposed, such as requiring α to be greater than β. Instead, their values ​​are automatically updated during backpropagation using optimization methods such as stochastic gradient descent (SGD).

[0055] In some embodiments, the analytical clipping function has the following characteristics. If α>β, then If α<β, then

[0056] In some examples, the minimum and maximum values ​​are automatically determined during the training process. FIG. 3A to FIG. 3D Different scenarios of how different α and β form the clipping function according to some implementations of the present disclosure are shown. Figure 3A An exemplary symmetric clipping scenario is shown in which α and β form a clipping function according to some implementations of the present disclosure. For example, α=-6, and β=6. Figure 3B An exemplary asymmetric clipping scenario is shown in which α and β form a clipping function according to some implementations of the present disclosure. For example, α=-6, and β=2. Figure 3C An exemplary positive clipping scenario is shown in which α and β form a clipping function according to some implementations of the present disclosure. For example, α=2, and β=6. Figure 3D An exemplary negative clipping scenario is shown in which α and β form a clipping function according to some implementations of the present disclosure. For example, α=-6, and β=-2.

[0057] In some embodiments, in conjunction with the analytical clipping function, additional techniques are used to minimize the range determined by α and β, i.e., |α-β|. For example, an L2 regularization method that minimizes the quantization range is applied to the loss function: argmin x {∑ l ||α l -β l || 2 Formula 2

[0058] In some embodiments, the goal of the L2 regularization method is to constrain the model to have optimal and minimum quantization ranges for the parameters and activations of each layer within the network.

[0059] In some embodiments, a new analytical function is used to define the minimum and maximum values ​​of the generalized differentiable for quantizing the neural network model. It provides symmetric or asymmetric quantization and introduces a more robust solution to the accuracy loss caused by spikes in the input data and activations in the intermediate layers.

[0060] Figure 4 The workflow and structural components of a neural network quantization method according to some implementations of the present disclosure are shown.

[0061] In some embodiments, the value to be quantized 410 is fed into a clipping function 420. The clipped value 430 from the clipping function 420 is further fed into a quantization process 440 with α and β, and the output of the quantization process 440 is a quantized value 450.

[0062] In some embodiments, a personal computer (PC) or mobile device runs neural network model training and inference.

[0063] In some embodiments, the quantization process follows the following steps: First, for each layer in the neural network, the layer parameters and layer activations (e.g., Figure 4 The values ​​to be quantized 410 shown in FIG. 1 are fed into the clipping function defined in Equation 1 (for example, as Figure 4 The method converts the value (e.g., Figure 4 The value to be quantized 410 shown in FIG. 4 is clipped to a value determined by a clipping function having parameters α and β (e.g., Figure 4 The range of minimum and maximum values ​​is determined by the clipping function 420 shown.

[0064] Second, pseudo-quantization or simulated quantization is used to quantize the weight parameters and intermediate activations in each layer (e.g. Figure 4 430), where the computation is still performed in FP32 format but quantized to INT8 values ​​(e.g., as shown in FIG. Figure 4 quantized values ​​450 shown) to simulate the behavior of INT8 calculations. In pseudo-quantization or simulated quantization methods, the quantization process is simulated.

[0065] In some embodiments, an L2 regularization method is added to optimize the loss of model accuracy, as shown in Equation 2. In some examples, L2 regularization is optional and can be used to minimize the quantization range to give an optimal high quantization resolution.

[0066] In some embodiments, during the training process, the minimum and maximum values ​​and the neural network weight parameters are jointly updated by back propagation using a gradient-based numerical method such as SGD. The method updates the values ​​of α and β during the training process to converge to an optimal solution (e.g., Figure 4 Quantization process 440 with α and β is shown).

[0067] In some embodiments, the structural components of the disclosed method interact with each other. For example, in the forward propagation, the output from the clipping function is fed into the analog quantization process. In the backward propagation, the training process updates the α and β values. L2 regularization is part of the loss function that guides the optimization of the α and β values.

[0068] In some embodiments, the data used in the methods and systems disclosed herein include input training data and neural network weight parameters. These two types of data are quantized so that calculations in model reasoning can be performed in the desired quantization format, such as from FP32 to INT8. In each layer, the values ​​to be quantized, such as layer weights and layer activations, are first limited using the limiting function shown in Formula 1, and then the limited values ​​are quantized using a pseudo-quantization method to simulate the quantization process. After calibration in PTQ or training in QAT is completed, the model input and model weights are converted from FP32 to INT8 so that model storage and reasoning are performed in a target format such as INT8.

[0069] Alternatively, in some embodiments, quantization can be applied in a channel-by-channel manner, so that each channel has its own quantization parameter. For example, if the layer is a 2D convolutional layer, where the weight dimension of the layer is [N k , N k , N i , N o ], the input activation dimension is [N x , N y , N i ], and the output activation dimension is [N x ',N y ',N o ], then in the commonly used layer-by-layer quantization, the quantization parameters are three pairs of scalars applied to weights, input activations, and output activations respectively. In channel-by-channel quantization, a pair of scalar quantization parameters is replaced by a pair of vectors, where each element is aligned with the channel dimension and each channel is quantized to a different range.

[0070] Figure 5 is a block diagram illustrating an exemplary process 500 for quantizing a neural network according to some implementations of the present disclosure.

[0071] The process 500 of quantizing a neural network includes a step 502 of clipping values ​​used within the neural network that are outside a range from a minimum value to a maximum value.

[0072] Process 500 includes a step 504 of simulating a quantization process using the clipped values.

[0073] Process 500 then includes step 506 of updating the minimum and maximum values ​​during training of the neural network to optimize the quantization process.

[0074] Process 500 additionally includes a step 508 of quantizing values ​​used within the neural network based on the updated minimum and maximum values.

[0075] For example, minimum and maximum values ​​are formulated in analytical functions for the purpose of clipping values ​​outside the range defined by the minimum and maximum values. These values ​​are applied to the values ​​of weights and intermediate features in neural networks.

[0076] In some embodiments, process 500 additionally includes step 510 of minimizing range during training. For example, the quantization range is minimized so that the resolution from quantized floating point numbers to integers can be optimized.

[0077] In some embodiments, the values ​​used in the neural network include one or more values ​​of weights, layer activations, and intermediate features in the neural network. For example, these values ​​are applicable to the values ​​of weights and intermediate features in the neural network. A clipping function is first used to clip the values ​​to be quantized, such as layer weights and layer activations.

[0078] In some implementations, clipping (502) values ​​used within the neural network that are outside of a range from a minimum value to a maximum value is performed by a clipping function:

[0079] Where α is the minimum value, β is the maximum value, x is the value used within the neural network, and f(x) is the clipping function. For example, if a value falls within the range between the minimum and maximum values, the value is not affected.

[0080] In some embodiments, clipping (502) values ​​used within the neural network that are outside the range from a minimum value to a maximum value includes at least one of symmetrical clipping, asymmetrical clipping, positive clipping, and negative clipping. For example, the minimum and maximum values ​​are automatically determined during training, and Figure 1 A to Figure 1 D shows different scenarios of how different α and β form the clipping function.

[0081] In some embodiments, minimizing the range during training (510) includes: using L2 regularization applied to the loss function during training to minimize the range during training. For example, a clipping function can be applied with additional L2 regularization during training to minimize the range determined by the range of minimum / maximum values ​​and increase the quantization resolution. Additional techniques are used to minimize the range determined by α and β, i.e., |α-β|. L2 regularization for minimizing the quantization range is applied to the loss function.

[0082] In some implementations, clipping (502) values ​​outside of a range from a minimum value to a maximum value used within the neural network is performed during a forward pass.

[0083] In some implementations, updating the minimum and maximum values ​​to optimize the quantization process (506) during training of the neural network is performed in a back-propagation.

[0084] In some embodiments, using the clipped values ​​to simulate the quantization process (504) includes calculating the simulated quantization in FP32 format and quantizing the clipped values ​​to INT8 format. For example, the weight parameters and intermediate activations in each layer are quantized using a pseudo-quantization method, where the calculation is still performed in FP32 format but quantized to INT8 values ​​to simulate the behavior of INT8 calculations.

[0085] In some embodiments, clipping values ​​used within the neural network that exceed a range from a minimum value to a maximum value (502) includes: clipping respective values ​​used within the neural network that exceed respective ranges from respective minimum values ​​to respective maximum values ​​for each of a plurality of channels. For example, quantization can be applied in a channel-by-channel manner with each channel having its own quantization parameter. In channel-by-channel quantization, a pair of scalar quantization parameters is replaced with a pair of vectors, where each element is aligned with a channel dimension and each channel is quantized to a different range.

[0086] Other embodiments also include various subsets of the above embodiments, including Figures 1 to 5 The illustrated embodiments may be combined or otherwise rearranged in various other embodiments.

[0087] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted through a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include: computer-readable storage media, which corresponds to tangible media such as data storage media; or communication media, including any media that facilitates, for example, transmitting a computer program from one place to another according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the implementations described in this application. A computer program product may include a computer-readable medium.

[0088] The terms used in the description of the implementations herein are only for the purpose of describing specific implementations and are not intended to limit the scope of the claims. As used in the description of the implementations and the appended claims, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should also be understood that when used in this specification, the terms "comprises" and / or "comprising" specify the presence of the features, elements, and / or parts, but do not exclude the presence or addition of one or more other features, elements, parts, and / or groups thereof.

[0089] It should also be understood that although the terms first, second, etc. can be used to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from other elements. For example, without departing from the scope of the implementation, the first electrode can be referred to as the second electrode, and similarly, the second electrode can be referred to as the first electrode. The first electrode and the second electrode are both electrodes, but they are not the same electrode.

[0090] The description of the present application is presented for the purpose of illustration and description, and is not intended to be exhaustive or limited to the invention in the disclosed form. With the benefit of the teachings presented in the foregoing description and the associated drawings, many modifications, variations, and alternative implementations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to best explain the principles of the invention, the practical application, and to enable other persons of ordinary skill in the art to understand the various implementations of the invention, and to best utilize the basic principles and various implementations, with various modifications suitable for the intended specific use. Therefore, it should be understood that the scope of the claims is not limited to the specific examples of the disclosed implementations, and modifications and other implementations are intended to be included within the scope of the appended claims.

Claims

1. A method for quantizing neural networks, include: Clipping values ​​used within the neural network that are outside a range from a minimum value to a maximum value; Using the clipped values ​​to simulate a quantization process; updating the minimum value and the maximum value during training of the neural network to optimize the quantization process; as well as Values ​​used within the neural network are quantized according to the updated minimum value and the maximum value.

2. The method according to claim 1, further comprising: include: The range is minimized during the training.

3. The method according to claim 1, in, The values ​​used within the neural network include one or more of weights, layer activations, and intermediate features in the neural network.

4. The method according to claim 1, in, Clipping of values ​​used within the neural network that are outside the range from the minimum value to the maximum value is performed by the following clipping function: Where α is the minimum value, β is the maximum value, x is the value used within the neural network, and f(x) is the clipping function.

5. The method according to claim 1, in, The clipping of values ​​outside the range of the minimum value to the maximum value used in the neural network includes at least one of symmetrical clipping, asymmetrical clipping, positive clipping and negative clipping.

6. The method according to claim 2, in, Minimizing the range during the training includes: The range is minimized during the training using L2 regularization applied to a loss function during the training.

7. The method according to claim 1, in, Clipping of values ​​used within the neural network that are outside the range from the minimum value to the maximum value is performed in the forward propagation.

8. The method according to claim 1, in, Updating the minimum and maximum values ​​to optimize the quantization process is performed in a back-propagation during training of the neural network.

9. The method according to claim 1, in, Emulating the quantization process using the clipped values ​​includes calculating the simulated quantization in FP32 format and quantizing the clipped values ​​to INT8 format.

10. The method according to claim 1, in, Clipping values ​​used within the neural network that are outside the range from the minimum value to the maximum value includes clipping respective values ​​used within the neural network that are outside the respective range from the respective minimum value to the respective maximum value of each of the plurality of channels.

11. An electronic device, include: one or more processing units; a memory coupled to the one or more processing units; and a plurality of programs stored in the memory, which, when executed by the one or more processing units, cause the electronic device to perform a plurality of operations for quantizing a neural network, the plurality of operations comprising: Clipping values ​​used within the neural network that are outside a range from a minimum value to a maximum value; Using the clipped values ​​to simulate a quantization process; updating the minimum value and the maximum value during training of the neural network to optimize the quantization process; and Values ​​used within the neural network are quantized according to the updated minimum value and the maximum value.

12. The electronic device according to claim 11, in, The multiple operations of quantizing the neural network also include: The range is minimized during the training.

13. The electronic device according to claim 11, in, The values ​​used within the neural network include one or more of weights, layer activations, and intermediate features in the neural network.

14. The electronic device according to claim 11, in, Clipping the values ​​of the range from the minimum value to the maximum value used within the neural network is performed by the following clipping function: Where α is the minimum value, β is the maximum value, x is the value used within the neural network, and f(x) is the clipping function.

15. The electronic device according to claim 11, in, Clipping values ​​used within the neural network that are outside the range from the minimum value to the maximum value includes at least one of symmetrical clipping, asymmetrical clipping, positive clipping, and negative clipping.

16. The electronic device according to claim 11, in, Minimizing the range during the training includes: The range is minimized during the training using L2 regularization applied to a loss function during the training.

17. The electronic device according to claim 11, in, Clipping of values ​​used within the neural network that are outside the range from the minimum value to the maximum value is performed in the forward propagation.

18. The electronic device according to claim 11, in, Updating the minimum and maximum values ​​to optimize the quantization process is performed in a back-propagation during training of the neural network.

19. A non-transitory computer-readable storage medium storing a plurality of programs for execution by an electronic device having one or more processing units, in, When the plurality of programs are executed by the one or more processing units, the electronic device performs a plurality of operations of quantizing a neural network, the plurality of operations comprising: Clipping values ​​used within the neural network that are outside a range from a minimum value to a maximum value; Using the clipped values ​​to simulate a quantization process; updating the minimum value and the maximum value during training of the neural network to optimize the quantization process; and Values ​​used within the neural network are quantized according to the updated minimum value and the maximum value.

20. The non-transitory computer readable storage medium of claim 19, in, The multiple operations of quantizing the neural network also include: The range is minimized during the training.

Citation Information

Patent Citations

  • Quantization method for accelerating the inference of neural networks

    US20230289558A1