Method and apparatus for post-training quantization of neural networks

Abstract

A method and apparatus (100) for dynamic point quantization of neural networks to achieve higher precision and lower storage requirements is provided. Traditional neural network models require vast amounts of disk space, which increases the computational cost associated with these models, requiring user devices to provide large amounts of performance and power. The present invention focuses on quantizing deep learning models, thereby reducing storage requirements compared to traditional models without compromising precision, while also achieving higher performance. The neural network is quantized by determining whether an input is positive or negative (901); determining an exponent range of the input (902); determining a maximum range of layer parameters of the input (903); determining an offset of a layer of the input (904); performing exponent adjustment by converting the input to its corresponding binary form (905), and determining an exponent representation of the input by adding the offset to the exponent adjustment value (906).

Description

TECHNICAL FIELD

[0001] This application claims priority to Indian Patent Application No. IN201931043851 filed on October 30, 2019 in the Indian Patent Office, titled “METHOD AND APPARATUS FOR QUANTIZATION OF NEURAL NETWORKS POST TRAINING,” the entire disclosure of which is incorporated herein by reference.

[0002] The subject matter described herein relates generally to deep learning models, and more particularly to methods and apparatus for quantizing neural networks, such as weights and parameters of stored neural network models. BACKGROUND

[0003] Machine learning is an application of artificial intelligence (AI) that utilizes statistical techniques to provide computer systems with the ability to “learn” from data, e.g., to progressively improve performance on a particular task, without being explicitly programmed. Machine learning models receive inputs and generate outputs, e.g., a prediction output, based on the received inputs. Some machine learning models are parametric models that generate outputs based on the received inputs and parameter values of the model.

[0004] State-of-the-art techniques in the field of machine learning include deep neural networks that employ computational models composed of multiple processing layers that learn representations of data (typically very large amounts of data) with multiple levels of abstraction, giving rise to the term “deep learning” or “deep networks.” Deep learning, also known as “deep structured learning” or “hierarchical learning,” is a part of machine learning methods that is based on learning a data representation, rather than on task-specific algorithms. The “learning” can be “supervised,” “semi-supervised,” or “unsupervised.”

[0005] Deep learning architectures, such as “deep neural networks,” “deep belief networks,” and “recurrent neural networks,” have been applied to a number of fields, including but not limited to computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, bioinformatics, drug design, board game programs, etc., and have produced results comparable to, and in some cases superior to, human experts.

[0006] Artificial neural networks (referred to as "neural networks" in this paper) are an information processing paradigm inspired by how biological nervous systems, such as the brain, process information. A key element of this paradigm is the novel structure of the information processing system. This paradigm consists of a large number of highly interconnected processing elements (neurons) that work together to solve specific problems. Neural networks learn from examples, just as humans do. Neural networks can be configured for specific applications, such as pattern recognition or data classification, through a learning process. Just as learning in biological systems involves adjusting the synaptic connections between neurons, so too does learning in neural networks.

[0007] A problem associated with existing neural network models is that they consume a significant amount of disk space, resulting in very large pre-trained models. This leads to difficulties in placing such models on edge devices or user devices, such as, but not limited to, mobile phones, tablets, Raspberry Pi boards, wearable devices, etc. Typically, such devices have limited storage space and processing power. Currently, these models are stored using 32-bit floating-point data structures. Examples of typical pre-trained model sizes are shown in Table 1 below. It can be consistently inferred that almost all of the size is occupied by the weights of the neural network connections, as these weights are distinct floating-point numbers. For the same reason, simple compression techniques (such as zip compression) cannot adequately compress these weights.

[0008] Table 1: List of space occupied by some pre-trained neural network models

[0009] Serial number Model Weight 1 AlexNet 285MB 2 VGG-16 528MB 3 Resnet 152 220MB 4 Extraction 90MB

[0010] Another problem associated with these neural network models is their very high computational cost, which consumes a significant amount of performance and power from user devices. Because these models typically require 32-bit floating-point operations and multiplication, the associated costs are substantial in terms of both power and performance requirements.

[0011] One approach to overcoming these drawbacks associated with neural networks is to increase the computing power of the hardware used to deploy these networks. However, this approach is costly and impractical for large-scale deployment on user devices due to size limitations. An alternative for large-scale deployment is to implement these neural network models in fixed-point environments, which offers advantages in terms of reduced memory, bandwidth, power consumption, computation time, and storage requirements.

[0012] For some of the most advanced technologies, refer to TensorFlow. TMThis approach introduces fixed-point quantization as a scheme for quantizing pre-trained models to reduce memory usage during the inference phase. The technique examines the range of weights, i.e., the minimum and maximum values, and attempts to map each weight value to an 8-bit equivalent integer representation tensor. For example, if the minimum = –10.0, the maximum = 30.0, and it's an 8-bit array, the quantized values ​​are represented below. While this approach saves 75% of space and improves performance due to its 8-bit computation, some real-time issues associated with this approach will be discussed in detail below.

[0013] Quantization Float 0 –10.0 255 30.0 128 10.0

[0014] With TensorFlow TM The disadvantages of association are as follows:

[0015] (a) Addition between different layers: Some neural networks use a simple element-wise addition layer type, which simply adds the outputs of two activation layers from different layers (RESNET-32 is one such example), as shown in Figures 1A-1C. In RESNET-32, 32 convolutional layers with a filter size of 3×3 are stacked on top of each other, and there are shortcut connections that add the inputs from two activation layers. Dashed shortcut connection arrows indicate that the input is upsampled to match the size of the second addition input, while solid shortcut connection arrows have addition inputs of the same size. If the two arrays have the same quantization scale, they can be added. Because this approach involves different quantization ranges of different layers, two activation layers at different quantization scales cannot be matched for addition. Therefore, dequantization is required to bring the layer values ​​back to their original values, perform the addition, and then perform quantization again. This repeated quantization and dequantization slows down the model and involves resource consumption, making the performance advantage impractical.

[0016] (b) Cascading: Full support for cascading layers introduces the same dequantization and requantization issues as for addition layers. Since such rescaling of a uint 8 data type would be a lossy operation, and cascading should appear to be a lossless operation, this implementation is also subject to the same impact.

[0017] (c) True Zero Problem: In this implementation, because quantization is range-based and incrementally incremented, the zero value is shifted, for example:

[0018] Quantization Float 0 –10.0 255 30.0 128 10.0

[0019] The floating-point value –10.0 is now represented by zero. Representing zero with some other number is problematic because zero is an important operation, appearing frequently in neural network operations. For example, convolutions can pad edges with zeros when filters overlap, the ReLU activation function replaces any negative number with zero, and so on. Therefore, the quantization operation is dequantized again, and then requantized, as shown in Figure 2. The dark shaded boxes represent the dequantization and quantization of the input data. This repeated quantization and dequantization causes the model's performance to fall below that of the original model. Figure 3 This demonstrates the quantization TensorFlow for three variants of the CIFARNET image classification model. TM Some performance data comparing it to a normal model. Performance is measured by the inference time required to classify a single image. From the charts, it can be inferred that TensorFlow... TM The performance of the quantized model is much lower than that of the original model, thus failing to achieve its intended purpose. Based on the data listed in Tables 2 and 3, it can be inferred that the performance of the MNIST dataset is also affected.

[0020] Table 2: Lists the performance of the original network on each operation of the MNIST dataset.

[0021] Node type Average ms Average % Conv2D 62.098 73.109％ MatMul 16.089 18.942％ Add 4.248 5.001％ Maxpool 1.453 1.711％ Relu 1.006 1.184％ Const 0.022 0.026％ Reshape 0.008 0.009％ Retval 0.004 0.005％ Arg 0.004 0.005％ NoOp 0.004 0.005％ Identity 0.003 0.004％

[0022] Table 3 lists TensorFlow. TM Performance of quantization operations on the MNIST dataset

[0023] Node type Average ms Average % QuantizedConv2D 637.514 74.337％ QuantizedMatMul 188.589 21.990％ QuantizeV2 7.491 0.873％ RequantizationRange 5.128 0.598％ Dequantized 4.476 0.522％ Add 4.296 0.501％ Requantize 3.801 0.443％ QuantizedMaxPool 2.203 0.257％ QuantizedRelu 1.398 0.163％ Min 1.241 0.145％ Max 1.224 0.143％ NoOp 0.147 0.017％ Const 0.042 0.005％ Reshape 0.020 0.002％ QuantizedReshape 0.014 0.002％ Arg 0.007 0.001％ Retval 0.005 0.001％ Summary of the Invention

[0024] This summary is provided to introduce concepts related to methods and apparatus for quantizing neural networks, which are further described in the detailed embodiments. This summary is not intended to define essential characteristics of the claimed subject matter, nor is it intended to define or limit the scope of the claimed subject matter.

[0025] As can be inferred from the above discussion, existing fixed-point quantization schemes have not solved the problem of associating with neural networks, making these models too large to be used on user devices.

[0026] Therefore, compared to the original model size, it is necessary to reduce the storage size without excessively compromising the accuracy and performance of the neural network. Therefore, this invention discloses a method, apparatus, and system for performing dynamic point quantization on neural networks to achieve higher accuracy with lower storage requirements. Furthermore, the time penalties associated with quantization and requantization are also reduced.

[0027] The aforementioned need for dynamic point quantization neural networks is merely intended to provide an overview of some drawbacks of conventional systems and techniques, and is not intended to be exhaustive. Other drawbacks of conventional systems and techniques, as well as the corresponding benefits of the various non-limiting embodiments described herein, will become more apparent after reading the following description.

[0028] The purpose of this invention is to provide quantization for deep learning models.

[0029] Another objective of this invention is to reduce the storage size compared to the original model size without significantly compromising the accuracy and performance of the neural network.

[0030] Another object of the present invention is to provide a method for quantizing neural networks.

[0031] Another object of the present invention is to provide an apparatus for quantizing neural networks.

[0032] Specifically, the present invention relates to quantifying existing models in order to reduce the storage requirements of the model compared to existing models without compromising accuracy, while also achieving better performance.

[0033] According to a first aspect of the invention, a method for quantizing a neural network is provided. The input to the method is the weights and biases of each layer of a pre-trained neural network. As a pre-trained neural network, these inputs may initially be related to real-world data (e.g., image information) and / or features of real-world data. The method includes the steps of: determining whether the input is positive or negative, wherein the input is a floating-point number including a signed bit, at least one exponent bit, and at least one mantissa bit; determining the exponent range of the input; determining the maximum range of layer parameters of the input for representing the input in quantized form; determining the layer offset of the input; performing exponent adjustment by converting the input to the corresponding binary form; and determining the exponent representation of the input by adding the offset to the exponent adjustment value.

[0034] According to the first aspect, in a first possible implementation of the method, if the input is a positive number, the signed bit is 0; if the input is a negative number, the signed bit is 1.

[0035] According to the first aspect, in a second possible implementation of the method, the n-bit wide exponent range is calculated by taking an exponent of n–1 base 2 and subtracting 1 from it to obtain a range starting from zero. 1 is subtracted from the exponent portion n to avoid overflow, where n is the bit width assigned to the exponent portion.

[0036] According to the first aspect, in a third possible implementation of the method, the maximum range is defined as range = numerical value ≤ 2. LevelDetermine, where Level is the number of bits required to represent the entire range of the layer parameters.

[0037] According to the first aspect, in a fourth possible implementation of the method, the offset is determined by subtracting the level required to represent the maximum range from the exponential range and adding half of the mantissa.

[0038] According to the first aspect, in a fifth possible implementation of the method, converting the input to the corresponding binary form includes the step of determining whether the bit length of the input is 10 bits; wherein, if the bit length of the input in binary form is greater than 10 bits, the lower binary position bits are discarded, and the value of the exponent portion is adjusted by adding the discarded bits; wherein, if the bit length of the input in binary form is less than 10 bits, zeros are appended to the end of the binary form of the input, and the exponent of the input is adjusted by subtracting the number of added zeros.

[0039] According to the first aspect, in a sixth possible implementation of the method, a step of multiplying any two quantized inputs is also included.

[0040] According to the first aspect, in a seventh possible implementation of the method, a step of adding any two quantized inputs is also included.

[0041] According to a second aspect of the present invention, an apparatus for quantizing a neural network is provided. The apparatus includes a sign determination module, an exponent range determination module, a maximum range determination module, an offset determination module, an exponent adjustment module, and an exponent representation module. The sign determination module is used to determine whether the input is positive or negative, wherein the input is a floating-point number including a signed bit, at least one exponent bit, and at least one mantissa bit. The exponent range determination module is used to determine the exponent range of the input. The maximum range determination module is used to determine the maximum range of layer parameters of the input. The offset determination module is used to determine the layer offset of the input. The exponent adjustment module is used to perform exponent adjustment by converting the input into a corresponding binary form. The exponent representation module is used to determine the exponent representation of the input by adding the offset to the exponent adjustment value.

[0042] According to the second aspect, in a first possible implementation of the device, if the input is a positive number, the signed bit is 0; if the input is a negative number, the signed bit is 1.

[0043] According to the second aspect, in a second possible implementation of the device, the n-bit wide exponent range is calculated by taking an exponent of base 2 (n–1) and subtracting 1 from it to obtain a range starting from zero. In the exponent portion, n is subtracted by 1 to avoid overflow, where n is the bit width assigned to the exponent portion.

[0044] According to the second aspect, in a third possible implementation of the device, the maximum range is defined as range = numerical value ≤ 2. Level Determine, where Level is the number of bits required to represent the entire range of the layer parameters.

[0045] According to the second aspect, in a fourth possible implementation of the device, the offset is determined by subtracting the level required to represent the maximum range from the exponential range and adding half of the mantissa.

[0046] According to the second aspect, in a fifth possible implementation of the device, converting the input to the corresponding binary form involves determining whether the bit length of the input is 10 bits; wherein, if the bit length of the input in binary form is greater than 10 bits, the lower binary position bits are discarded, and the value of the exponent portion is adjusted by adding the discarded bits; wherein, if the bit length of the input in binary form is less than 10 bits, zeros are appended to the end of the binary form of the input, and the exponent of the input is adjusted by subtracting the number of added zeros.

[0047] According to the second aspect, in a sixth possible implementation of the device, the device is also used to perform a multiplication operation on any two quantized inputs.

[0048] According to the second aspect, in a seventh possible implementation of the device, the device is also used to perform an addition operation on any two quantized inputs.

[0049] Other aspects, advantages and distinctive features of the invention will become apparent to those skilled in the art from the following detailed description, which discloses exemplary embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0050] This detailed description is made with reference to the accompanying drawings. In each drawing, the numerals on the reference numerals identify the drawing in which the numeral first appears. All drawings use the same numerals to refer to the same features and components.

[0051] Figure 1 shows a schematic diagram of the RESNET architecture with 34 layers.

[0052] Figure 2 shows a block diagram of the Tensorflow quantization and dequantization operations.

[0053] Figure 3 The graph shows a comparison of the performance of the Tensorflow quantized model with the original model.

[0054] Figure 4 A block diagram illustrating the processing flow of weight analysis provided by an embodiment of the present invention is shown.

[0055] Figure 5 A block diagram illustrating the processing flow of floating-point representation provided by the present invention is shown.

[0056] Figure 6 A flowchart illustrating how memory space is allocated according to the present invention is shown.

[0057] Figure 7 A graphical representation of the speed comparison between integer arithmetic multiplication and floating-point multiplication is shown.

[0058] Figure 8 A block diagram illustrating the floating-point to mantissa exponent representation provided by the present invention is shown.

[0059] Figure 9 A flowchart of a method for quantizing neural networks provided by an embodiment of the present invention is shown.

[0060] Figure 10 A block diagram of an apparatus for quantizing a neural network, according to another embodiment of the present invention, is shown.

[0061] It should be understood that the accompanying drawings are for illustrative purposes and may not be drawn to scale. Detailed Implementation

[0062] The following description, in conjunction with the accompanying drawings of the embodiments of the present invention, clearly illustrates some technical solutions of the embodiments of the present invention. Those skilled in the art will understand that the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0063] This invention can be implemented in various ways, such as processes, apparatus, systems, computer-readable media (e.g., computer-readable storage media), or computer networks, wherein program instructions are stored in (non-transitory) memory or transmitted via optical or electronic communication links. In this specification, these implementations or any other form in which the invention may take are referred to as techniques. Generally, the order of steps in the disclosed processes can be modified within the scope of this invention.

[0064] The following provides a detailed description of one or more embodiments of the invention, along with accompanying drawings illustrating the principles of the invention. While the invention has been described in conjunction with these embodiments, it is not limited to any particular embodiment. The scope of the invention is limited only by the claims, and the invention includes many alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description to provide a thorough understanding of the invention. These details are provided for illustrative purposes, and the invention may be practiced without some or all of these specific details. For clarity, known technical information in the art related to this invention has not been described in detail to avoid unnecessarily obscuring the invention.

[0065] Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced without these specific details. In other instances, well-known methods, processes, components, modules, units, and / or circuits have not been described in detail so as not to obscure the invention.

[0066] While embodiments of the invention are not limited in this respect, discussions using terms such as “processing,” “computing / calculating,” “determining,” “establishing,” “analyzing,” and “checking” can refer to one or more operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulate and / or convert data (represented as physical (e.g., electronic) quantities within computer registers and / or memory) into other data, which are similarly represented as physical quantities within computer registers and / or memory or other informational non-transitory storage media that may store instructions to perform the operations and / or processes.

[0067] While embodiments of the invention are not limited in this respect, the term "multiple" as used herein may include "a plurality" or "two or more," etc. The term "multiple" may be used throughout the specification to describe two or more components, devices, elements, units, parameters, etc. Unless explicitly stated otherwise, the method embodiments described herein are not limited to a particular order or sequence. Furthermore, some of the described method embodiments or elements thereof may occur or be performed simultaneously at the same point in time.

[0068] In this invention, "quantization" refers to representing a quantity using fewer bits than floating-point bits.

[0069] In this invention, "deep learning model" refers to an artificial neural network with layer depth.

[0070] In this invention, "neural network" refers to a computational model based on the structure and function of biological neural networks.

[0071] In this invention, "dequantization" refers to converting quantized values ​​into floating-point representations.

[0072] In this invention, a "pre-trained model" refers to an artificial neural network whose weights are trained to perform a specific task.

[0073] In this invention, "weight parameter" refers to the multiplication factor of the edge connecting one neuron to another in an artificial neural network.

[0074] In this invention, "bias parameter" refers to the additive factor of neurons in an artificial neural network.

[0075] In this invention, a "convolutional layer" refers to a layer that applies a convolution operation to the input.

[0076] In this invention, "Residual Network (RESNET)" refers to a deep learning model.

[0077] In this invention, "Modified National Institute of Standards and Technology (MNIST)" refers to a large dataset of handwritten digits used to train various image processing systems.

[0078] In this invention, "Rectified Linear Unit (ReLU)" refers to an activation function similar to half-wave rectification in electrical engineering.

[0079] In this invention, "TensorFlow" TM "" refers to an open-source machine learning framework.

[0080] In this invention, "fixed-point quantization" refers to a quantization method that assigns a signal value falling between two levels to one of two levels with a fixed total number of levels.

[0081] In this invention, "weight analysis" refers to finding the range and standard deviation of the layer weights.

[0082] A method and apparatus for quantizing neural networks are disclosed. While aspects relating to quantizing deep learning models in order to reduce the storage requirements of these models without unduly compromising accuracy and to achieve better performance compared to conventional models, the invention can be implemented in any number of different computing systems, environments, and / or configurations, and embodiments are described in the context of the following exemplary systems, devices / nodes / apparatus, and methods.

[0083] The problems associated with neural network models are that they consume a significant amount of disk space, and pre-trained models are extremely large, making it difficult to place such models on user devices. Furthermore, the computational costs associated with these models are very high, demanding substantial performance and power from user devices. These models require 32-bit floating-point operations and multiplication. Therefore, the costs associated with these operations are high in terms of both power and performance requirements.

[0084] Furthermore, existing fixed-point quantization schemes do not address the issue of associating with neural networks, resulting in models that are too large to be used on user devices. Therefore, it is necessary to reduce the storage size compared to the original model without excessively compromising the accuracy and performance of the neural network. Current solutions involve repeated quantization and dequantization, which makes the neural model perform worse than the original model, thus severely impacting the performance of the neural model.

[0085] Therefore, this invention discloses a method and apparatus for dynamically point-quantizing neural networks to achieve higher accuracy with lower storage requirements. This invention specifically focuses on quantizing deep learning models to reduce storage requirements compared to traditional models without significantly compromising accuracy, while also achieving higher performance.

[0086] It should be noted that the solution provided by this invention focuses on pre-trained models to quantize weights. An embodiment of this solution is implemented through the steps briefly described below.

[0087] 1. Weight Analysis: Unlike traditional fixed-point quantization methods, a variable bit-width method is used based on the weight range in each layer; the quantized weights are represented in 8-bit or 16-bit schemes, such as... Figure 4 As shown.

[0088] 2. Floating-point representation: Unlike traditional fixed-point quantization methods, this method examines the range and represents numbers falling within a specific range with unique numbers in the quantization domain, employing an alternative representation method. It is worth noting that, since all weights in the pre-trained model are observed to fall within a smaller range, the weights are represented using... Figure 5 The representation is shown below. Compared to five convolutional layers, three fully connected layers contain most of the weights. The final 1000-way softmax layer helps classify the image into one of 1000 categories. This completely eliminates the requirement for floating-point multiplication associated with the current solution. An equivalent representation is provided below.

[0089] Uint32 = uint16 * uint16

[0090] The result of the operation on two uint 16 operands is stored in a uint 32 to avoid any overflow that may occur during the operation, and then the result is normalized again to an 8-bit or 16-bit format. The technical solutions with different use cases and precision data are described in detail below.

[0091] As mentioned above, popular pre-trained models of convolutional neural networks are too large to be installed on user devices or edge devices, and are currently stored using 32-bit floating-point data structures, such as, but not limited to, mobile phones, tablets, Raspberry Pi boards, wearable devices, etc. In this invention, these weight and bias parameters are converted into a smaller bit representation, i.e., 8 bits or 16 bits, depending on the distribution of weight and bias parameters for a specific layer. For layers with large standard deviations in weights and biases, 16 bits are chosen to represent the quantized weights; otherwise, 8 bits are chosen. This will reduce the model's memory footprint by at least 50%.

[0092] Figure 6 This invention illustrates that in each of the weights and biases in layers A, B, and C, a dynamic bit width (8 bits or 16 bits) is selected based on the range. Therefore, each layer can have a different bit width depending on the parameter range in the corresponding layer. In the case of multiplication, only the mantissa is multiplied, both for integers and floating-point numbers, which are computationally intensive compared to integers.

[0093] Figure 7 The experimental results are presented graphically, illustrating how the outcome of mathematical operations varies depending on the 16-bit or 32-bit data size and the data type of the integer or floating-point number. For integer arithmetic operations, both 8-bit and 16-bit instructions are available, which are significantly faster than floating-point operations, as shown in the graph. Furthermore, it can be easily deduced from the graph that floating-point multiplication of the same data size is much slower than integer multiplication.

[0094] In this invention, an exponent portion is added, and an XOR operation is performed on the signed bits. Since the mantissa portion is relatively small, the cost of multiplication is low in this case, thus improving computational performance. This representation has a smaller relative error compared to other techniques. Furthermore, this representation does not require dequantization because all weights are quantized at the same scale or can be easily converted to a higher bit representation. In the case of addition, a right-shift or left-shift operator is used in the mantissa portion to make the exponent portion identical. Then, the mantissa portion is added to obtain the final result. Therefore, addition has little overhead compared to existing quantization algorithms.

[0095] The dynamic dot quantization of the present invention includes the following steps, which are interpreted for 16-bit representation.

[0096] Step 1: For each weight value of a layer and all layers of the pre-trained neural network model, depending on the sign of the weight value to be quantized, the first bit will be 0 if the value to be quantized is positive, and 1 if the weight value to be quantized is negative.

[0097] Step 2: Determine the Exponent Range: Calculate the standard deviation of the parameter distribution in the layer. Based on the required output precision, set a threshold for the standard deviation. If the deviation exceeds this threshold, the layer parameters will be quantized to 16 bits; if it falls below, they will be quantized to 8 bits. If the layer parameters are in a 16-bit representation, the exponent will be 5 bits, therefore the range will be 2^4 – 1 = 15 (2^4 has been considered, not 2^5, to avoid overflow). Calculate the n-bit wide exponent range by taking the base-2 exponent n – 1 and subtracting 1 from it to obtain a range starting from zero. Subtract 1 from n in the exponent portion to avoid overflow. The formula is provided below, where n is the bit width assigned to the exponent portion.

[0098] Index range = 2 n-1 -1

[0099] Step 3: Determining the Maximum Range: To represent a value in quantized form, the range of the layer parameters must be determined. The maximum value between the modulus of the minimum and maximum layer parameter values ​​is the maximum range of the layer. The maximum range is used to calculate the number of bits required to represent the layer parameter. For example, if the minimum parameter value is -20 and the maximum parameter value is 150, then the maximum value of 20 and 150, i.e., 150, is the maximum range. Therefore, 2^8 = 256 levels are needed to represent the number 150, since 2^7 can only represent 128 levels. The formula is provided below, where level is the number of bits required to represent the entire range of the layer parameters. The level is the number of bits required to represent the entire range of the layer parameters, for example, for a single layer.

[0100] Range = value ≤ 2 Level

[0101] Step 4: Offset Determination: Define an offset value for each layer of the model. The offset value is taken from the exponent portion so that the layer parameters can fit into an 8-bit or 16-bit representation. The offset for a particular layer is calculated by subtracting the exponent range of the level required to represent the maximum range of the layer parameters. To keep the weight values ​​centered within the quantization domain range, half the mantissa bits of the chosen format (8-bit or 16-bit format) are added to the offset. Therefore, for the example above, the offset would be 15 (exponent range) – 8 (level) + 5 (half the mantissa bits) = 12. The formula is provided below, where n is the bit width assigned to the exponent portion, and level is the number of bits required to represent the entire range of layer parameters.

[0102] Offset = 2 (n-1) -1-Level + last digit / 2

[0103] Step 5: Locate the Mantissa and Adjust the Exponent: Now, convert each value in the layer parameters to binary form, ensuring that each value conforms to the 10-bit mantissa format (if the value is 16 bits wide). If the binary form of a value in the layer parameters has more than 10 bits, ignore the lower binary positions of the binary value and adjust the exponent by adding the ignored bits. If the binary form of a value in the layer parameters has less than 10 bits, append zeros at the end and adjust the exponent by subtracting the number of zeros added. For example, 150 can be represented as 1001 0110, so this will be represented in 10 bits as 1001 0110 00, with two extra bits added at the end to represent the full 10 bits, making the exponent 2. (-2) Therefore, the index adjustment value is -2.

[0104] Step 6: Find the exponent: The exponent can be represented by adding the offset to the exponent adjustment value from Step 5. The formula is provided below:

[0105] Exponent 1 = Offset + Exponent Adjustment Value from Step 5

[0106] Calculate the exponent for all values ​​in the operation. In the example value of 150, the exponent becomes 12 + (–2) = 10.

[0107] When multiplying any two values ​​using the quantization method of this claim, steps 1-6 remain the same. These steps represent the values ​​in floating-point format. After step 6, the following steps are performed for multiplication.

[0108] Step 7: Multiplication Representation: The mantissa portions of the two quantized values ​​are multiplied by integers, and the result is stored in a 32-bit accumulator to avoid any overflow. The operation is then normalized to 10 bits. For example, 150*150 will be 1010 1111 1100 1000 000 based on the mantissa portions described above. Therefore, this will be normalized to 1010 1111 11 (decimal 703), and will produce an exponent offset because we will discard the last 9 bits. Therefore, the mantissa exponent adjustment value is the number of bits discarded in the mantissa. The formula is provided below:

[0109] Mantissa index adjustment value = Number of bits discarded

[0110] Step 8: Final Representation of the Exponent: The exponent of the product is the sum of the exponents of the two quantized values ​​and the bits discarded in Step 7. The formula is provided below:

[0111] The exponent of the product = exponent 1 + exponent 2 + the number of bits discarded in step 7

[0112] Therefore, when multiplying 150 by 150, the exponent of the product becomes (10) + (10) + 9 = 29.

[0113] Step 9: Offset Calculation: The offset of the product will be the sum of the offsets of the two quantized values.

[0114] Offset of the product = Offset 1 + Offset 2

[0115] In the case of 150*150, the offset of the product becomes 12+12=24.

[0116] Step 10: Verify the product value: To verify that the product value is correct, the value needs to be converted back to real number format. This can be done by multiplying the mantissa by the power of the result of subtracting the offset from the base 2 exponent. Start with the sign bit and add a sign at the end. The formula is provided below:

[0117] Real value = -1 符号 ×last digit × 2 (指数-偏移量)

[0118] In the case of 150*150, the real value becomes 22496. This value is very close to the actual product value of 22500. Quantization multiplication introduces a relative error of 0.017%.

[0119] When adding any two values ​​using the quantization method of this claim, steps 1-6 remain the same. These steps represent the values ​​in floating-point format. After step 6, the following steps are performed to perform the addition.

[0120] Step 7: Normalization: To add two quantized values, the exponent portions (the final exponent portions) should be the same. As mentioned above, the smaller value is normalized to the larger value by adjusting the mantissa portion, using the same offset and exponent portion value.

[0121] Step 8: Addition Representation: Add the mantissas of the two quantized values. The result is stored in a 32-bit accumulator. Then, the 32-bit value is normalized to a 10-bit value using a mantissa exponent adjustment. For example, 150 + 150, adding the mantissas will produce 10010110 00 + 1001 0110 00 = 1001 0110 000. The normalization process discards the last bit. Therefore, the mantissa exponent adjustment becomes 1.

[0122] Step 9: Determine the exponent: The exponent of the sum will be the sum of the exponent of a numerical value and the number of bits discarded in step 8.

[0123] The exponent of the sum = exponent 1 + the number of bits discarded in step 8

[0124] Step 10: Offset determination: The offset value should be the same as the numerical value.

[0125] The addition algorithm can be verified in the same way as the multiplication algorithm. The addition result is verified using the formula from step 10 of the multiplication algorithm. The quantization error of the proposed solution is superior to that of TensorFlow. TM The quantization error of the method used. Table 4 shows the quantization error when quantizing 159.654 in both schemes. It shows that the error improvement is greater than 10 times.

[0126] Table 4: Quantization error of the two experimental schemes

[0127]

[0128] For this scheme, the maximum value is 2. 10 The precision is ±0.5. Any value greater than this, as well as any floating-point number with a distance greater than 0.5, is considered a numeric value. The largest number that can be represented using this scheme is 2. 171 -1.

[0129] Figure 10 A block diagram of an apparatus (100) for quantizing a neural network according to the present invention is shown. The apparatus includes a sign determination module (101), an exponent range determination module (102), a maximum range determination module (103), an offset determination module (104), an exponent adjustment module (105), and an exponent representation module (106). The sign determination module (101) is used to determine whether the input is positive or negative, wherein the input is a floating-point number including a sign bit, at least one exponent bit, and at least one mantissa bit. The exponent range determination module (102) is used to determine the exponent range of the input. The maximum range determination module (103) is used to determine the maximum range of the layer parameters of the input. The offset determination module (104) is used to determine the layer offset of the input. The exponent adjustment module (105) is used to perform exponent adjustment by converting the input to its corresponding binary form. The exponent representation module (106) is used to determine the exponent representation of the input by adding the offset to the exponent adjustment value. The apparatus is also used to perform multiplication and / or addition operations on any two quantized inputs. The connections between modules are merely illustrative, and it is understood that modules are interconnected based on which output should be provided to which input to achieve the corresponding function.

[0130] Compared to existing fixed-point quantization techniques, the dynamic point quantization technique of this invention ensures higher accuracy, as clearly demonstrated in the example discussed above, where a value is quantized and then dequantized back to its original value. Besides accuracy, dynamic point quantization ensures significantly faster computational performance than fixed-point quantization because it does not involve transforming or representing the value in another form, thus eliminating all problems associated with fixed-point quantization, such as the true zero problem, addition of different layers, cascading, etc. Dynamic point quantization eliminates the need for dequantization, thereby improving performance by several times.

[0131] The following mentions some non-limiting advantages of the present invention:

[0132] It reduces the storage requirements associated with traditional neural networks without compromising accuracy too much.

[0133] It offers better and faster performance compared to existing models.

[0134] It can implement neural networks on user devices.

[0135] Compared to existing models, it can save up to 75% of disk space.

[0136] It reduces the model's memory footprint by at least 50%.

[0137] Those skilled in the art will understand that any known or new algorithm can be used to implement this invention. However, it should be noted that this invention provides a method and apparatus for quantizing neural networks to achieve the aforementioned benefits and technical advantages, regardless of the use of any known or new algorithm.

[0138] Those skilled in the art will recognize that the various examples, units, and algorithm steps described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the function is executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described systems, devices, and units can be referred to the corresponding process in the above-described method embodiments, and will not be repeated here.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, method, or system can be implemented in other ways. For example, the described apparatus embodiments are merely exemplary. For example, the unit division is merely a logical functional division, and in actual implementation, it can be other divisions. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed can be implemented through some interface. Direct coupling or communication connection between apparatuses or units can be implemented electronically, mechanically, or otherwise.

[0141] When these functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The basic computer software product, stored in the storage medium, includes several instructions for instructing a computer node (which may be a personal computer, server, or network node) to perform all or part of the steps of the methods described in the embodiments of the present invention. The aforementioned storage medium includes any medium capable of storing program code, such as a USB flash drive, a removable hard disk, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0142] Devices that communicate with each other are not required to maintain continuous communication unless otherwise expressly stated. Furthermore, devices that communicate with each other may communicate directly or indirectly through one or more media.

[0143] When this document describes a single device or item, it is obvious that more than one device / item (whether or not they cooperate) can be used in place of that single device / item. Similarly, in the case of more than one device or item described herein (whether or not they cooperate), it is obvious that a single device / item can be used in place of more than one device or item, or that a different number of devices / items can be used in place of the number of devices or programs shown. Alternatively, the functionality and / or features of a device may be implemented by one or more other devices that are not explicitly described as having such functionality / features. Therefore, other embodiments of the invention do not necessarily need to include the device itself.

[0144] Finally, the language used in this specification has been chosen primarily for readability and guidance, rather than for recounting or limiting the subject matter of the invention. Therefore, the scope of the invention is not limited by this detailed description, but rather by any claims upon which this application is based. Consequently, the disclosure of embodiments of the invention is intended to be illustrative and not to limit the scope of the invention, which is set forth in the claims.

[0145] Regarding the use of virtually any plural and / or singular terms in this document, those skilled in the art can convert from plural to singular and / or from singular to plural depending on the context and / or application. For clarity, various singular / plural substitutions may be explicitly described herein.

[0146] While implementations of methods and apparatuses for quantizing neural networks have been described in language specific to structural features and / or methods to reduce storage requirements without unduly compromising accuracy and to achieve better performance compared to existing models, it should be understood that the appended claims are not necessarily limited to the specific features or methods described. Specific features and methods are disclosed as examples of implementations of quantizing deep learning models.

Claims

1. A method for quantizing neural networks in the field of computer vision, characterized in that, The method includes the following steps: The device (100) determines whether the input is positive or negative, wherein the input is a floating-point number including a signed bit, at least one exponent bit and at least one mantissa bit, and the input is associated with first image information; The device (100) determines the exponential range of the input; The device (100) determines the maximum range of layer parameters of the input for representing the input in quantized form; The device (100) determines the offset of the input layer; the offset is determined by subtracting the Level required to represent the maximum range from the exponential range and adding half of the mantissa; the Level is the number of bits required to represent the entire range of the layer parameters; The device (100) performs exponential adjustment by converting the input into the corresponding binary form; The device (100) determines the exponential representation of the input by adding the offset to the exponential adjustment value.

2. The method according to claim 1, characterized in that, If the input is positive, the signed bit is 0; if the input is negative, the signed bit is 1.

3. The method according to claim 1 or 2, characterized in that, The range of the index is determined by the following equation: Index range = 2 n-1 -1 Where n is the bit width.

4. The method according to claim 1 or 2, characterized in that, The maximum range is determined by the following equation: Range = Value ≤ 2 Level .

5. The method according to claim 1 or 2, characterized in that, Converting the input to its corresponding binary form includes determining whether the input has a bit length of 10 bits; wherein, if the bit length of the input in binary form is greater than 10 bits, the lower binary bits are discarded, and the value of the exponent is adjusted by adding the discarded bits; if the bit length of the input in binary form is less than 10 bits, zeros are appended to the end of the binary form of the input, and the exponent of the input is adjusted by subtracting the number of zeros added.

6. The method according to claim 1 or 2, characterized in that, It also includes the step of multiplying any two quantized inputs.

7. The method according to claim 1 or 2, characterized in that, It also includes the step of adding any two quantized inputs.

8. An apparatus (100) for quantizing neural networks in the field of computer vision, characterized in that, The device includes: The sign determination module (101) is used to determine whether the input is a positive number or a negative number, wherein the input is a floating-point number including a sign bit, at least one exponent bit and at least one mantissa bit, and the input is associated with the first image information; The exponent range determination module (102) is used to determine the exponent range of the input; The maximum range determination module (103) is used to determine the maximum range of the input layer parameters; An offset determination module (104) is used to determine the offset of the input layer; the offset is determined by subtracting the Level required to represent the maximum range from the exponential range and adding half of the mantissa; the Level is the number of bits required to represent the entire range of the layer parameters; An exponent adjustment module (105) is used to perform exponent adjustment by converting the input into a corresponding binary form; An exponential representation module (106) is used to determine the exponential representation of the input by adding the offset to the exponential adjustment value.

9. The apparatus according to claim 8, characterized in that, If the input is positive, the signed bit is 0; if the input is negative, the signed bit is 1.

10. The apparatus according to claim 8 or 9, characterized in that, The range of the index is determined by the following equation: Index range = 2 n-1 -1 Where n is the bit width.

11. The apparatus according to claim 8 or 9, characterized in that, The maximum range is determined by the following equation: Range = Value ≤ 2 Level .

12. The apparatus according to claim 8 or 9, characterized in that, The exponent adjustment module (105) is used to determine whether the bit length of the input is 10 bits; wherein, if the bit length of the input in binary form is greater than 10 bits, the lower binary position bits are discarded, and the value of the exponent portion is adjusted by adding the discarded bits; if the bit length of the input in binary form is less than 10 bits, zeros are appended to the end of the binary form of the input, and the exponent of the input is adjusted by subtracting the number of added zeros.

13. The apparatus according to claim 8 or 9, characterized in that, It is also used to perform multiplication on any two quantized inputs.

14. The apparatus according to claim 8 or 9, characterized in that, It is also used to perform addition operations on any two quantized inputs.

15. A neural network model, characterized in that, The weights of the layers are stored in an 8-bit or 16-bit floating-point representation obtained by the method according to any one of claims 1 to 7.

16. A method for quantizing neural networks, characterized in that, The neural network includes a graphics file and a parameter file, and the method employs the method according to any one of claims 1 to 7.

Images