Systems and methods for quantizing neural networks and identifying systems and methods

Abstract

Systems and methods for quantizing neural networks and identification systems and methods are provided. A system and method for quantizing a pre-trained neural network includes determining, for each layer or channel of the pre-trained neural network, for each main bit width value in a set of predetermined main bit width values, a minimum quantization noise of the layer or channel by a layer / channel bit width determiner; and selecting, for the layer or channel, a main bit width value having the minimum quantization noise of the layer or channel by a bit width selector. In an example embodiment, the minimum quantization noise of the layer or channel is based on the square of the range of weights of the layer or channel multiplied by a "negative power of the current main bit width value of a constant".

Description

[0001] This application claims priority to U.S. Provisional Application No. 62 / 735,839, filed on September 24, 2018, and U.S. Application No. 16 / 181,326, filed on November 5, 2018, the disclosures of which are incorporated herein by reference in their entireties. Technical Field

[0002] The subject matter disclosed herein relates to neural networks. More specifically, the subject matter disclosed herein relates to quantization techniques that can be used to provide guidance for selecting optimal layer-by-layer or channel-by-channel bit-width quantization values ​​for weights and / or activations of deep neural networks. Background Art

[0003] Recently, neural networks (e.g., deep neural networks (DNNs)) are widely used to process data (e.g., multimedia data such as image data, voice data, text data, etc.). However, the accuracy of DNN in processing data is relatively low, and the demand for computing resources is also large, which cannot meet the growing needs of users.

[0004] In particular, quantization of weights and / or activations of pre-trained deep neural networks (DNNs) provides a way to run DNNs in resource-constrained devices (e.g., but not limited to smartphones). The typical approach has been to assign the same bit-width quantization value to each layer or each channel for weights and / or activations, without considering the fact that the weights and / or activations of each layer or each channel contribute differently to the overall accuracy of the DNN. Summary of the invention

[0005] An example embodiment provides a system for quantizing a pre-trained neural network, which may include a range determiner, a squarer, a multiplier, and a comparator. The range determiner may determine the range of weights of a selected layer or channel of the pre-trained neural network. The squarer may square the weight range of the selected layer or channel. The multiplier may multiply the square of the range of weights of the selected layer or channel by a negative power of a current main bit width value selected from a set of main bit width values ​​of a constant. The comparator may compare the output of the multiplier for each current main bit width value selected from the set of main bit width values ​​for the selected layer or channel, and may select a main bit width value corresponding to the minimum value output from the multiplier for the selected layer or channel. In one embodiment, the constant may be 4. In another embodiment, the set of main bit width values ​​may include values ​​between 4 and 10, including 4 and 10. In yet another embodiment, the weight range of each layer or channel may also include the range of weights and activations of each layer or channel of the pre-trained neural network.

[0006] An example embodiment provides a system for quantizing a pre-trained neural network, which may include a range determiner, a first multiplier, a second multiplier, and a bit width selector. The range determiner may determine the range of weights of a selected layer or channel of the pre-trained neural network. The first multiplier may square the range of weights of the selected layer or channel. The second multiplier may multiply the square of the range of weights of the selected layer or channel by a negative power of a current main bit width value selected from a set of main bit width values ​​of a constant. The bit width selector may select a main bit width value corresponding to a minimum value output from the second multiplier for the selected layer or the selected channel. In one embodiment, the system may also include: a controller that may control the range determiner, the first multiplier, the second multiplier, and the bit width selector to select a bit width for each layer or channel of the pre-trained neural network.

[0007] An example embodiment provides a method for quantizing a pre-trained neural network, wherein the method may include: determining, by a layer / channel bit width determiner, for each layer or channel of the pre-trained neural network, for each main bit width value in a set of predetermined main bit width values, a minimum quantization noise of the layer or channel; and selecting, by a bit width selector, a main bit width value having the minimum quantization noise of the layer or channel for the layer or channel. In one embodiment, the minimum quantization noise of the layer or channel is based on the square of a range of weights of the layer or channel multiplied by a "negative power of a current main bit width value of a constant". In another embodiment, the constant may be 4. In yet another embodiment, the set of main bit width values ​​includes values ​​between 4 and 10, inclusive. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the following sections, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the accompanying drawings, in which:

[0009] Figure 1 Depicted are example embodiments of a "one-shot bit determination" technique or method for quantizing a pre-trained DNN in accordance with the subject matter disclosed herein;

[0010] Figure 2 depicts a functional block diagram of an example embodiment of a one-shot bit determination system in accordance with the subject matter disclosed herein;

[0011] Figure 3 is a graph showing the results of a one-shot bit-determined quantization technique performed on a commercial DNN compared to a typical quantization technique; and

[0012] Figure 4 is a graph showing the results of a one-shot bit-determined quantization technique performed on another commercial DNN in comparison with a typical quantization technique. DETAILED DESCRIPTION

[0013] In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, those skilled in the art will appreciate that the disclosed aspects may be practiced without these specific details. In other cases, well-known methods, processes, components, and circuits are not described in detail so as not to obscure the subject matter disclosed herein. In addition, the described aspects may be implemented to perform low-power 3D depth measurements in any imaging device or system (including but not limited to smart phones, user devices (UEs), and / or laptop computers).

[0014] References throughout this specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment disclosed herein. Therefore, the appearance of the phrase "in one embodiment" or "in an embodiment" or "according to an embodiment" (or other phrases with similar meanings) in different places throughout this specification may not necessarily all indicate the same embodiment. In addition, in one or more embodiments, particular features, structures, or characteristics may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" should not be interpreted as necessarily being preferred or advantageous over other embodiments. In addition, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In addition, depending on the context discussed herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. Similarly, hyphenated terms (e.g., "two-dimensional", "predetermined", "specific-pixel", etc.) may occasionally be used interchangeably with corresponding non-hyphenated versions (e.g., "two-dimensional", "predetermined", "specific pixel", etc.), and capitalized items (e.g., "Counter Clock", "Row Select", "PIXOUT", etc.) may be used interchangeably with corresponding non-capitalized versions (e.g., "counter clock", "row select", "pixout", etc.). Such occasional interchangeable usage should not be considered inconsistent with each other.

[0015] In addition, depending on the context discussed herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. It is further noted that the various drawings (including component drawings) shown and discussed herein are for illustrative purposes only and are not drawn to scale. Similarly, various waveforms and timing diagrams are shown for illustrative purposes only. For example, for clarity, the size of some elements may be exaggerated relative to other elements. In addition, if deemed appropriate, reference numerals are repeated in the drawings to indicate corresponding and / or similar elements.

[0016] The terms used herein are only for the purpose of describing some example embodiments, and are not intended to limit the subject matter of the claim. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that the terms "include" and / or "comprise" when used in this specification indicate the presence of stated features, wholes, steps, operations, elements and / or components, but do not exclude the presence or increase of one or more other features, wholes, steps, operations, elements, components and / or their groups. The terms "first", "second", etc. used herein are used as labels for the nouns after them, and unless so clearly defined, do not represent any type of order (e.g., space, time, logic, etc.). In addition, the same reference numerals may be used between two or more figures to represent parts, components, blocks, circuits, units or modules with the same or similar functions. However, such usage is only for the purpose of simplifying the description and facilitating discussion, and does not mean that the construction or structural details of such components or units are the same between all embodiments, or that such commonly referenced parts / modules are the only way to implement some example embodiments in the multiple example embodiments disclosed herein.

[0017] It will be understood that when an element or layer is referred to as being on, "connected to" or "bonded to" another element or layer, it may be directly on, directly connected to or directly bonded to the other element or layer, or there may be intermediate elements or layers. Conversely, when an element or layer is referred to as being "directly on," "directly connected to" or "directly bonded to" another element or layer, there are no intermediate elements or layers. The same reference numerals always represent the same element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0018] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the subject matter belongs. It will also be understood that, unless expressly so defined herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal manner.

[0019] As used herein, the term "module" means any combination of software, firmware, and / or hardware configured to provide the functionality described herein in conjunction with the module. Software may be implemented as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in any embodiment described herein may, for example, include hardwired circuits, programmable circuits, state machine circuits, and / or firmware storing instructions executed by programmable circuits, alone, or may include any combination of hardwired circuits, programmable circuits, state machine circuits, and / or the firmware. Modules may be implemented collectively or individually as circuits that form part of a larger system (e.g., but not limited to, an integrated circuit (IC), a system on a chip (SoC), etc.). The various components and / or functional blocks disclosed herein may be implemented as modules that may include software, firmware, and / or hardware that provide the functionality described herein in conjunction with the various components and / or functional blocks.

[0020] The subject matter disclosed herein provides a fast, simple, and efficient quantization technique for pre-trained DNNs without retraining the DNNs after quantization. The quantization technique (referred to herein as the "one-bit determination" technique) can be used to provide guidance for selecting the best layer-by-layer or channel-by-channel bit-width quantization values ​​for weights and / or activations. The technique disclosed herein provides better accuracy than typical quantization methods because it is based on the fact that the loss caused by quantization can be directly related to the range and weight gradients of the weights and / or activations being quantized.

[0021] The bit width quantization techniques disclosed herein account for the effects of quantization of individual layers within a DNN that can be quantized through information statistics of each layer (e.g., but not limited to, the range of weights / activation values ​​to be quantized and the variance of gradients). The errors caused by quantization can be analyzed to determine how quantization of each layer affects the overall accuracy of the DNN.

[0022] The subject matter disclosed herein can use the gradients of weights and / or activations or statistical information of the range of weights and / or activations as a guide for selecting the optimal bit width for each layer on a layer-by-layer or channel-by-channel basis. For example, consider the problem of minimizing the loss L(w) of a DNN, which can be defined as

[0023]

[0024] Where t is the target, y(w) is the normalized softmax output, N is the size of the training sample, and w is a long vector of the concatenation of the weights of each layer of the deep neural network. In other words, w is a vector of continuous weights for each layer.

[0025] The change of loss L(w) based on the change of w can be viewed as

[0026]

[0027] so,

[0028]

[0029] and

[0030]

[0031] as well as

[0032]

[0033] Where T is the partial derivative and n is the number of iterations.

[0034] Therefore, minimizing the quantization error is equivalent to reducing the impact of quantization on the accuracy of the DNN. Therefore, in order to minimize the variance of the quantization error ΔL, or var(Δw i ) should be minimized, where, in one embodiment, Where b is the current main bit width value.

[0035] Figure 1 An example embodiment of a "one-time bit determination" technique or method for quantizing a pre-trained DNN according to the subject matter disclosed herein is depicted. The technique begins at 101. At 102, a DNN is pre-trained. At 103, a validation dataset is run on the DNN to obtain global information about the performance of the DNN, such as network accuracy and error information. In one embodiment, the global information may include statistical information (e.g., but not limited to, the standard deviation of the gradients of weights and / or activations, and the range of weights and / or activations to be quantized). At 104, statistical information is collected for each layer and / or channel (e.g., but not limited to, the range of weights and / or activations for each layer and / or each channel). In one embodiment, the statistical information that can be obtained and used for each layer and / or channel may include (but not limited to) the standard deviation of the gradients of weights and / or activations, and the range of weights and / or activations to be quantized.

[0036] At 105, an index i may be initialized to iteratively quantize the layers / channels of the DNN. In addition, at 105, an index j may be initialized, which may be used to iteratively select a master bit-width value from a set of master bit-width values ​​for evaluation. In one embodiment, the set of master bit-width values ​​may include values ​​within the interval [4, 10]. In another embodiment, the set of master bit-width values ​​may be different.

[0037] At 106, a first primary bit width j is selected to quantize the layers / channels of the pre-trained DNN. At 107, the weights and / or activations of layer / channel i may be quantized based on the current primary bit width j selected from the set of primary bit width values. More specifically, the bit widths of the weights and / or activations of layer / channel i may be quantized based on the selected primary bit width, global statistics obtained for the pre-trained DNN, and statistics obtained for a specific layer / channel. In one embodiment, the bit width of layer / channel i may be quantized based on the product of the currently selected primary bit width multiplied by the square root of the ratio of the layer / channel statistic to the global statistic.

[0038] At 108, it is determined whether the weights and / or activations of all layers / channels of the pre-trained DNN have been quantized to the currently selected primary bit width. If not, the process continues to 109 where index i is incremented, and the process returns to 105. If at 108, the weights and / or activations of all layers / channels of the pre-trained DNN have been quantized with the currently selected primary bit width, the process continues to 110 where a validation data set is run on the DNN where the weights / activations of all layers / channels have been quantized to the currently selected primary bit width j. At 111, the validation error of the DNN is determined and recorded or stored, for example, in a memory.

[0039] At 112, it is determined whether all primary bit widths in the set of primary bit widths have been evaluated. If not, the process continues to 113 where the index j is incremented, and the process returns to 106 where the next primary bit width is selected and the weights and / or activations of all layers / channels of the pre-trained DNN are quantized to the currently selected primary bit width. If at 112, all primary bit widths in the set of primary bit widths have been selected, the process continues to 114.

[0040] At 114, for each layer / channel, a bit width corresponding to the lowest quantized DNN validation error is selected for the layer / channel. Flow continues to 115, at which the method ends. Other considerations may apply when selecting bit width values, so at 114, the results can be used as a guide for selecting the best bit width value for the layer / channel.

[0041] Figure 2A functional block diagram of an example embodiment of a one-time bit determination system 200 according to the subject matter disclosed herein is depicted. The system 200 may include a memory 202, a range determiner 203, a first multiplier 204, a second multiplier 205, and a comparator 206. In some embodiments, the system 200 may include a controller 207, which may provide iterative type functionality. In one embodiment, the system 200 may be implemented as a module, which may be any combination of software, firmware, and / or hardware configured to provide the functionality of the system 200. In another embodiment, one or more of the functional blocks of the system 200 may be a module, in which case such a module may be any combination of software, firmware, and / or hardware configured to provide the functionality described herein in conjunction with the particular module.

[0042] The validation dataset may be run on a pre-trained DNN 201 that will be quantized by the one-shot bit determination system 200. Global statistics may be collected and stored in the memory 202. In addition, statistics for each layer / channel may also be stored in the memory 202.

[0043] For each different layer / channel and for each different primary bit width b in the set of primary bit widths, a validation dataset is run on the DNN 201 and used by the system 200, e.g., in Figure 1 1. The technique 100 depicted in FIG. 1 is evaluated. For each iteration, the range of weights and / or activations may be determined by a range determiner 203. The output of the range determiner 203 is input to a first multiplier 204. The first multiplier 204 squares the output of the range determiner 203. That is, the output of the first multiplier 204 is the square of the range of weights and / or activations. The output of the first multiplier 204 is input to a second multiplier 205 and multiplied by a constant raised to a negative power of the current primary bit width value b. In one embodiment, the second multiplier 205 multiplies the output of the first multiplier 204 by 4. -b / 12. The output of the second multiplier 205 represents the variance value of the total error of the DNN based on the quantized weights / activations of the current layer / channel for the current primary bit width. In one embodiment, the output of the second multiplier may be input to a comparator 206, which tracks the variance values ​​of different primary bit widths evaluated for the current layer / channel. The comparator 206 may then select the smallest variance value as the bit width quantization of the current layer / channel. In an alternative embodiment, the output of the second multiplier 205 may be stored in a memory for subsequent evaluation by the comparator 206.

[0044] Each layer / channel of the deep neural network 201 may be evaluated by the one-time bit determination system 200, and the determined bit width of the layer / channel may be used to quantize the corresponding layer / channel of a pre-trained neural network so that the pre-trained neural network may run in a resource-constrained device (e.g., but not limited to, a smart phone).

[0045] here, Figure 2 The one-time bit determination system 200 may be included in an object recognition system (not shown). In addition, the object recognition system may further include a receiving unit and a recognition unit, wherein the receiving unit receives data to be recognized (e.g., multimedia data such as image data, voice data, text data, etc.), and the recognition unit recognizes an object from the data to be recognized using a quantized pre-trained neural network. Figure 2 The one-time bit determination system 200 quantizes the pre-trained neural network, so compared with using a neural network trained according to a typical quantization technique, the object recognition system according to the present invention can have a relatively higher recognition rate and use relatively lower computing resources. Here, data recognition should be understood in a broad sense, which includes but is not limited to object filtering, object detection, recognition, identification, rejection or discrimination. The method performed by the object recognition system can be referred to as an object recognition method.

[0046] Figure 3 is a graph showing the results of a one-shot bit-determined quantization technique performed on a commercial DNN compared to a typical quantization technique. Figure 3 In the figure, the horizontal axis is the bit width and the vertical axis is the maximum (Top-1) verification accuracy. Curve 301 represents the Top-1 verification accuracy as a function of bit width when quantization is performed in a typical manner, in which the same bit width quantization is used for each layer of the DNN. Curve 302 represents the Top-1 verification accuracy of the one-time bit determination quantization technique as a function of bit width, in which the bit width quantization value is selected layer by layer based on the standard deviation of the gradient of the weight. Curve 303 represents the Top-1 verification accuracy of the one-time bit determination quantization technique as a function of bit width, in which the bit width quantization value is selected layer by layer based on the range of weights. Curve 304 represents the Top-1 verification accuracy of the one-time bit determination quantization technique as a function of bit width, in which the bit width quantization value is selected channel by channel based on the range of weights. For the typical method, relatively accurate results are not obtained until the bit width is quantized to 7 bits, and each of the three one-time bit determination quantization techniques can provide better relatively accurate results for smaller bit width values. In particular, the one-shot bit determination technique (curve 304 ) of selecting a channel-by-channel bit-width quantization value based on a range of weights may provide relatively accurate results for 6-bit-width quantization values.

[0047] Figure 4 is a graph showing the results of a one-shot bit-determined quantization technique performed on another commercial DNN compared to a typical quantization technique. Figure 4 , the horizontal axis is bit width, and the vertical axis is Top-1 verification accuracy. Curve 401 represents the Top-1 verification accuracy as a function of bit width when quantization is performed in a typical manner, in which the same bit width quantization is used for each layer of the DNN. Curve 402 represents the Top-1 verification accuracy as a function of bit width for a one-shot bit determination quantization technique, in which the bit width quantization value is selected layer by layer based on the range of weights.

[0048] As those skilled in the art will recognize, the innovative concepts described herein can be modified and varied over a wide range of applications. Accordingly, the scope of the claimed subject matter should not be limited to any of the specific exemplary teachings discussed above, but rather is defined by the claims.

Claims

1. A system for quantizing a pre-trained neural network, the pre-trained neural network being used to recognize an object from image data, speech data, or text data, the system comprising: A range determiner that determines a range of weights of a selected layer or channel of a pre-trained neural network; The squarer squares the range of weights of the selected layer or channel; a multiplier that multiplies the square of the extent of the weights of the selected layer or channel by a constant raised to a negative power of a current primary bit width value selected from a set of primary bit width values; as well as A comparator compares the outputs of the multipliers of each current primary bitwidth value selected from the set of primary bitwidth values ​​for the selected layer or channel, and selects a primary bitwidth value corresponding to a minimum value output from the multiplier for the selected layer or channel.

2. The system according to claim 1, wherein: The constant is 4.

3. The system according to claim 2, wherein: The set of primary bit width values ​​includes values ​​in the interval [4, 10].

4. The system according to claim 2, wherein: The range of weights for each layer or channel also includes the range of weights and activations for each layer or channel of the pre-trained neural network.

5. The system according to claim 1, further comprising: A controller controls the rangers, squarers, multipliers, and comparators to select the bit width for each layer or channel of the pre-trained neural network.

6. The system according to claim 5, wherein: The verification data set is input into the input of the pre-trained neural network to determine the selected bit width value of each layer or channel of the pre-trained neural network.

7. A system for quantizing a pre-trained neural network, the pre-trained neural network being used to recognize an object from image data, speech data, or text data, the system comprising: A range determiner that determines a range of weights of a selected layer or channel of a pre-trained neural network; The first multiplier squares the range of the weights of the selected layer or channel; a second multiplier that multiplies the square of the extent of the weight of the selected layer or channel by a constant raised to a negative power of a current primary bit width value selected from the set of primary bit width values; as well as A bit width selector selects a main bit width value corresponding to the minimum value output from the second multiplier for the selected layer or the selected channel.

8. The system according to claim 7, wherein: The constant is 4.

9. The system according to claim 8, wherein: The set of primary bit width values ​​includes values ​​in the interval [4, 10].

10. The system according to claim 8, wherein: The range of weights for each layer or channel also includes the range of weights and activations for each layer or channel of the pre-trained neural network.

11. The system according to claim 7, further comprising: A controller controls the range determiner, the first multiplier, the second multiplier, and the bit width selector to select a bit width for each layer or channel of the pre-trained neural network.

12. The system according to claim 11, wherein: The validation dataset is fed into the input of the pre-trained neural network to determine the selected bit width value for each layer or channel.

13. A method for quantizing a pre-trained neural network, the pre-trained neural network being used to recognize an object from image data, speech data, or text data, the method comprising: Determining, by a layer / channel bit width determiner, for each layer or channel of the pre-trained neural network, for each main bit width value in a set of predetermined main bit width values, a minimum quantization noise of the layer or channel; as well as A master bit width value having minimum quantization noise for the layer or channel is selected for the layer or channel by a bit width selector.

14. The method according to claim 13, wherein: The minimum quantization noise of a layer or channel is based on the square of the range of the weights of the layer or channel multiplied by a constant raised to the negative power of the current dominant bit width value.

15. The method according to claim 14, wherein: The constant is 4.

16. The method according to claim 15, wherein: The set of primary bit width values ​​includes values ​​in the interval [4, 10].

17. The method according to claim 14, wherein: Determining the minimum quantization noise is also based on the square of the range of weights and activations of the layer or channel multiplied by a constant raised to the negative power of the current dominant bit width value.

18. The method according to claim 14, wherein: The step of determining the minimum quantization noise of a layer or channel for each layer or channel of the pre-trained neural network further comprises: for each main bit width value selected from the set of main bit width values, inputting a verification data set into the input of the pre-trained neural network.

19. The method according to claim 18, wherein: Determining the minimum quantization noise is also based on the square of the range of weights and activations of the layer or channel multiplied by a constant raised to the negative power of the current dominant bit width value.

20. An object recognition system, the object recognition system comprising: The system of any one of claims 1-12, quantizing a pre-trained neural network; a receiving unit, receiving data to be identified, wherein the data to be identified is image data, voice data or text data; and The recognition unit uses a quantized pre-trained neural network to recognize objects from the data to be recognized.

21. An object recognition method, the object recognition method comprising: quantizing a pre-trained neural network by the method of any one of claims 13-19; receiving data to be recognized, wherein the data to be recognized is image data, voice data or text data; and Identify objects from the data to be identified using a quantized pre-trained neural network.

Images