Method and apparatus for training a neural network for image recognition

Abstract

A method and apparatus for training a neural network for image recognition is provided. The neural network includes at least one layer having weights and output feature maps that have been jointly pruned and quantized. The weights of the layer are pruned using an analytical threshold function. Each weight remaining after pruning is quantized based on a weighted average of quantization and dequantization of the weights for all quantization levels to form a quantized weight of the layer. An output feature map of the layer is generated based on the quantized weight of the layer. Each output feature map of the layer is quantized based on a weighted average of quantization and dequantization of the output feature maps for all quantization levels. A cost function is used to update the parameters of the analytical threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer.

Description

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 819,484, filed on March 15, 2019, and U.S. Application No. 16 / 396,619, filed on April 26, 2019, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0002] The subject matter disclosed herein relates to neural networks. More specifically, the subject matter disclosed herein relates to methods and apparatus for training neural networks for image recognition. Background Art

[0003] During the inference phase of a neural network for image recognition, the output feature maps of the weights and activation functions take up a lot of memory. Neural networks also use a lot of operations to calculate the output inferences. Therefore, during the image recognition process, a lot of memory and a lot of operations are required. Summary of the invention

[0004] Example embodiments provide a method for training a neural network for image recognition, the method may include: acquiring training images; pre-training the neural network based on a first training image in the training images; pruning weights of a layer of the pre-trained neural network using an analytical threshold function; quantizing each weight of the layer remaining after pruning based on a weighted average of quantization and dequantization of the weights for all quantization levels to form a quantized weight of the layer; determining an output feature map of the layer based on the quantized weight of the layer and a second training image in the training images; quantizing each output feature map of the layer based on a weighted average of quantization and dequantization of the output feature maps for all quantization levels; and using a cost function to update parameters of the analytical threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer.

[0005] Example embodiments provide a device for training a neural network for image recognition, the device may include an interface and a processing device. The interface may receive a neural network. The received neural network may include a plurality of layers. The processing device may acquire a training image; pre-train the received neural network based on a first training image in the training image; and perform a predetermined process on the pre-trained neural network to generate a neural network including at least one layer having weights and output feature maps that have been jointly pruned and quantized. The predetermined process may include: pruning the weights of the at least one layer of the pre-trained neural network using an analytical threshold function, quantizing each weight of the at least one layer remaining after pruning based on a weighted average of quantization and dequantization of weights for all quantization levels to form a quantized weight of the at least one layer, determining an output feature map of the at least one layer based on the quantized weight of the at least one layer and a second training image in the training image, quantizing each output feature map of the at least one layer based on a weighted average of quantization and dequantization of the output feature maps for all quantization levels, and updating a parameter of the analytical threshold function, a weighted average of all quantization levels of the weights, and a weighted average of each output feature map of the at least one layer using a cost function.

[0006] Example embodiments provide a neural network that may include a plurality of layers, wherein at least one layer may include jointly pruned and quantized weights and an output feature map. The jointly pruned weights may be pruned using an analytical threshold function. Each weight remaining after pruning may also be quantized based on a weighted average of quantization and dequantization of the weights for all quantization levels. An output feature map may be formed based on the pruned and quantized weights of the layer. Each output feature map may be quantized based on a weighted average of quantization and dequantization of the output feature maps for all quantization levels and a parameter of the analytical threshold function. The weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer may be updated based on a cost function. In one embodiment, before the weights of the at least one layer and the output feature map are jointly pruned and quantized, the neural network is a full-precision trained neural network. In another embodiment, the cost function includes a pruning loss term, a weight quantization loss term, and a feature map quantization loss term.

[0007] An example embodiment provides a method for pruning weights and output feature maps of a layer of a neural network, the method may include: using an analytical threshold function to prune the weights of a layer of a neural network, the neural network being a trained neural network; quantizing each weight of the layer remaining after pruning based on a weighted average of quantization and dequantization of weights for all quantization levels to form a quantized weight of the layer; determining an output feature map of the layer based on the quantized weight of the layer; quantizing each output feature map of the layer based on a weighted average of quantization and dequantization of output feature maps for all quantization levels; and using a cost function to update parameters of the analytical threshold function, a weighted average of all quantization levels of the weights, and a weighted average of each output feature map of the layer. In one embodiment, the step of updating parameters of the analytical threshold function, a weighted average of all quantization levels of the weights, and a weighted average of each output feature map of the layer also includes: optimizing the cost function.

[0008] Example embodiments provide a neural network analyzer, which may include an interface and a processing device. The interface may receive a neural network that may include a plurality of layers. The processing device may generate a neural network that may include at least one layer having weights and output feature maps that have been pruned and quantized jointly. The processing device may use an analytical threshold function to prune the weights of the at least one layer of the neural network, quantize each weight of the at least one layer remaining after pruning based on a weighted average of quantization and dequantization of weights for all quantization levels to form a quantized weight of the at least one layer, determine an output feature map of the at least one layer based on the quantized weight of the at least one layer, quantize each output feature map of the at least one layer based on a weighted average of quantization and dequantization of output feature maps for all quantization levels, and use a cost function to update parameters of the analytical threshold function, weighted averages of all quantization levels of weights, and weighted averages of each output feature map of the at least one layer. In one embodiment, the interface also outputs a neural network that includes at least one layer having weights and output feature maps that have been pruned and quantized jointly. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the following section, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the accompanying drawings, in which:

[0010] Figure 1 Depicting a block diagram of a neural network analyzer that can perform joint pruning and quantization operations on layers of a deep neural network (DNN) in accordance with the subject matter disclosed herein;

[0011] Figure 2 Describes an example architecture of a DNN that can be input to a neural network trained at full precision. Figure 1 Neural network analyzer;

[0012] Figure 3 Depicting an example layer in a DNN according to the subject matter disclosed herein and application of an analytical threshold function to form an analytical weight function f(w) for optimally pruning weights of the example layer;

[0013] Figure 4A a graph depicting an example threshold function h(w) according to equation (1) of the subject matter disclosed herein;

[0014] Figure 4B a graph depicting a weight function f(w) according to equation (2) of the subject matter disclosed herein;

[0015] FIG. 5A to FIG. 5E depicts five example graphs of the threshold function h(w) of equation (1) with different values ​​of parameters α and β according to the subject matter disclosed herein to provide a visual of how the parameters α and β may affect the threshold function h(w) and the weight function f(w);

[0016] Figure 6 is a flow chart of an example embodiment of a method for performing joint pruning and quantization operations on a layer of a neural network (such as a DNN) according to the subject matter disclosed herein; and

[0017] Figure 7 Describing can be used to achieve Figure 1 A block diagram of an exemplary architecture of a data processing system of a neural network analyzer. DETAILED DESCRIPTION

[0018] In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the disclosure. However, those skilled in the art will appreciate that the disclosed aspects may be practiced without these specific details. In other cases, well-known methods, processes, components, and circuits are not described in detail to avoid obscuring the subject matter disclosed herein.

[0019] References throughout this specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment disclosed herein. Therefore, the phrases "in one embodiment" or "in an embodiment" or "according to an embodiment" (or other phrases with similar meanings) appearing in various places throughout this specification may not necessarily all represent the same embodiment. In addition, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "used as an example, instance, or illustration." Any embodiment described herein as "exemplary" will not be interpreted as necessarily preferred or superior to other embodiments. In addition, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner. In addition, depending on the context discussed herein, a singular term may include a corresponding plural form, and a plural term may include a corresponding singular form. Similarly, hyphenated terms (e.g., "two-dimensional", "predetermined", "specific-pixel", etc.) may occasionally be used interchangeably with corresponding non-hyphenated versions (e.g., "two-dimensional", "predetermined", "specific pixel", etc.), and capitalized items (e.g., "Counter Clock", "Row Select", "PIXOUT", etc.) may be used interchangeably with corresponding non-capitalized versions (e.g., "counter clock", "row select", "pixout", etc.). Such occasional interchangeable usage should not be considered inconsistent with each other.

[0020] In addition, depending on the context discussed herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. It should also be noted that the various drawings (including component diagrams) shown and discussed herein are for illustrative purposes only and are not drawn to scale. Similarly, various waveform diagrams and timing diagrams are shown for illustrative purposes only. For example, for clarity, the size of some elements may be exaggerated relative to other elements. In addition, if deemed appropriate, reference numerals are repeated in the accompanying drawings to indicate corresponding and / or similar elements.

[0021] The terms used herein are only for the purpose of describing some example embodiments, and are not intended to limit the subject matter claimed for protection. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that the terms "include" and / or "comprise" when used in this specification specify the existence of stated features, wholes, steps, operations, elements, components and / or their groups, but do not exclude the existence or increase of one or more other features, wholes, steps, operations, elements, components and / or their groups. As used herein, the terms "first", "second" etc. are used as labels for the nouns after them, and unless so clearly defined, do not indicate any type of order (e.g., space, time, logic, etc.). In addition, the same reference numerals may be used between two or more drawings to represent parts, components, blocks, circuits, units or modules with the same or similar functions. However, such usage is only for the purpose of simplifying the description and facilitating discussion, and does not indicate that the construction or structural details of such components or units are the same between all embodiments, or that such commonly referenced parts / modules are the only way to implement some of the example embodiments disclosed herein.

[0022] It will be understood that when an element or layer is referred to as being "on," "connected to," or "bound to" another element or layer, it may be directly on, directly connected to, or directly bound to the other element or layer, or there may be intervening elements or layers. Conversely, when an element or layer is referred to as being "directly on," "directly connected to," or "directly bound to" another element or layer, there are no intervening elements or layers. The same reference numerals always refer to the same elements. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0023] As used herein, the terms "first", "second", etc. are used as labels for the nouns that follow them, and do not indicate any type of order (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. In addition, the same reference numerals may be used between two or more figures to represent parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such usage is only for simplicity of illustration and ease of discussion, and does not indicate that the construction or structural details of such components or units are the same between all embodiments, or that such commonly referenced components / modules are the only way to implement some of the example embodiments disclosed herein.

[0024] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the subject matter belongs. It will also be understood that, unless expressly so defined herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense.

[0025] As used herein, the term "module" means any combination of software, firmware, and / or hardware configured to provide the functionality described herein in conjunction with the module. Software may be implemented as a software package, code, and / or instruction set or instructions, and, for example, the term "hardware" as used in any implementation described herein may include, individually or in any combination, hardwired circuits, programmable circuits, state machine circuits, and / or firmware storing instructions executed by programmable circuits. Modules may be implemented collectively or individually as circuits that form part of a larger system (e.g., but not limited to an integrated circuit (IC), a system on a chip (SoC), etc.). The various components and / or functional blocks disclosed herein may be implemented as modules that may include software, firmware, and / or hardware that provide the functionality described herein in conjunction with the various components and / or functional blocks.

[0026] The subject matter disclosed herein provides joint pruning and quantization, which allows the neural network to be optimized to select the pruning model that is most suitable for quantization, and at the same time, to select the quantization model that is most suitable for pruning. Therefore, the subject matter disclosed herein allows a larger set of allowable solutions to be considered, thereby providing better pruning and quantization results.

[0027] Figure 1A block diagram depicting a neural network analyzer 100 that can perform joint pruning and quantization operations on layers of a deep neural network (DNN) in accordance with the subject matter disclosed herein. In an example embodiment, the neural network analyzer 100 can be implemented as a data processing system executing suitable software. As depicted, the neural network analyzer 100 can receive a full-precision trained neural network 101 as input. The neural network analyzer 100 can operate on the layers of the neural network 101 by performing joint pruning and quantization operations 104 on the weights and output feature maps of the layers of the neural network 101. The neural network analyzer 100 outputs a neural network 102 in which the layers of the neural network have been jointly pruned and quantized. In another example embodiment, the neural network analyzer 100 may receive a neural network and a training image, first train the received neural network using a first training image in the training image (herein, referred to as "pre-training") to generate a pre-trained neural network 101, and then operate the layers of the pre-trained neural network 101 by performing a joint pruning and quantization operation 104 on the weights and output feature maps of each layer of the pre-trained neural network 101. Various training techniques may be used for pre-training. For example, the pre-trained neural network 101 may be a neural network 101 trained with full precision, but is not limited thereto.

[0028] The joint pruning and quantization operation 104 performed on each layer may include multiple forward and backward passes or iterations to fine-tune the pruned and quantized weights and output feature maps of each layer of the DNN. During the forward pass through the layers of the DNN, the weights are pruned and then quantized. The output feature map is then calculated using the quantized weights and a second training image in the training images. Here, the second training image may be the same or different from the first training image. The output feature map is then quantized. During the backward pass, the pruning parameters and the parameters for quantizing the pruned weights and output feature maps are updated based on the optimization of a cost function.

[0029] In one embodiment, the full precision trained neural network 101 and the pruned and quantized neural network 102 may be DNNs. Figure 2Depicts an example architecture 200 of a DNN that can be input to a neural network analyzer 100 as a neural network 101 trained at full precision. The example architecture 200 of VGG16 specifically includes 13 convolutional (CONV) layers (layers indicated by symbols containing CONV in the figure), three fully connected (FC) layers (layers indicated by symbols containing FC in the figure), and five pooling layers (PL). The input of the VGG 16 architecture 200 is applied to the convolution layer CONV1_1, and the output is provided at the classification layer (Classification).

[0030] In order to prune the weights of a layer of a DNN, each weight may be multiplied by an analytic threshold function h(w), which is generally characterized by having the following property: the values ​​of weights having a size less than a threshold are set to zero without affecting the values ​​of weights having a size greater than the threshold. Figure 3 Depicts an example layer 300 in a DNN according to the subject matter disclosed herein and the application of an analytical threshold function for forming an analytical weight function f(w) for optimally pruning the weights of the example layer 300. Figure 3 As indicated on the left side of , regardless of the level of complexity of the neural network that may include the example layer 300, the output feature map of layer 300 may be formed as the sum of the products of the input feature maps and the connection weights w to the output feature map.

[0031] At 301, an analytical threshold function h(w) may be applied to the weights w of layer 300 to form a weight function f(w) that replaces the weights w, thereby forming layer 300'. In one embodiment, each weight w may be multiplied by the analytical threshold function h(w). Figure 3 A graph of an example weight function f(w) is shown at the top center of FIG. In one embodiment, the analytical threshold function h(w) may be

[0032]

[0033] Wherein, α is a parameter that controls the sharpness of the threshold function h(w), and β is a parameter that controls the distance (or range) between the first edge and the second edge of the threshold function. The smaller the parameter α, the sharper the outline of h(w), and the smaller the parameter β, the wider the width of h(w). The parameters α and β are scaling values ​​and do not have units.

[0034] Figure 4AA graph depicting an example threshold function h(w) of equation (1). Parameter α controls the sharpness of the edges 401a and 401b of the threshold function, that is, parameter α controls the rate of change of edges 401a and 401b between h(w)=0 and h(w)=1. Parameter β in equation 1 controls the width 402 between the two edges 401a and 401b at h(w)=0.5.

[0035] like Figure 4A , for a first set 403 of continuous weight values ​​centered at 0, the threshold function h(w) has a value of 0. Additionally, for a second set 404 of continuous weight values ​​greater than the first set 403 of continuous weight values, the threshold function h(w) has a value of 1, and for a third set 405 of continuous weight values ​​less than the first set 403 of continuous weight values, the threshold function h(w) has a value of 1. A first edge 401a of the threshold function h(w) is between the first set 403 of continuous weight values ​​and the second set 404 of continuous weight values. A second edge 401b is between the first set 403 of continuous weight values ​​and the third set 405 of continuous weight values.

[0036] In the region of the first edge 401a and the second edge 401b, the value of the threshold function h(w) transitions between 0 and 1. When the threshold function h(w) transitions between 0 and 1, the sharpness of each of the first edge 401a and the second edge 401b (i.e., the sharpness of the edges 401a and 401b of the threshold function h(w)) is based on the value of the parameter α, and the distance 402 between the first edge 401a and the second edge 401b at h(w)=0.5 is based on the value of the parameter β.

[0037] In one embodiment, each weight in the weights w of the layer may be multiplied by a threshold function h(w) to form a weight function f(w) as

[0038]

[0039] Figure 4B A graph depicting the weight function f(w) of equation (2). For a first set 403 of continuous weight values, the weight function f(w) has a value of 0. For a second set 404 of continuous weight values ​​greater than the first set 403 of continuous weight values, the weight function f(w) has a value w, and for a third set 405 of continuous weight values ​​less than the first set 403 of continuous weight values, the weight function f(w) has a value w. The weight function f(w) includes transition regions 406a and 406b corresponding to the first edge 401a and the second edge 401b, respectively. As the value of parameter α becomes smaller, the first edge 401a and the second edge 401b become sharper, and the transition regions 406a and 406b become smaller in the w direction.

[0040] FIG. 5A to FIG. 5E Five example graphs of the threshold function h(w) of equation (1) with different values ​​of the parameters α and β are depicted to provide a visual of how the parameters α and β may affect the threshold function h(w) and the weight function f(w). FIG. 5A to FIG. 5C In the above example, the parameter α is changed from α = 1.0 × 10 -8 ( Figure 5A ) changes to α=1.0×10 -2 ( Figure 5C ), and the parameter β is constant at β = 4. FIG. 5A to FIG. 5C It can be seen that the smaller the value of α, the sharper the transition of the threshold function h(w) between 0 and 1. Figure 5D and Figure 5E In the above equation, the parameter β is changed from β=8( Figure 5D ) changes to β=16( Figure 5E ), and the parameter α is α=1.0×10 -8 Constant. Figure 5D and Figure 5E It can be seen that the smaller the parameter β is, the larger the width between the edges of the threshold function h(w) at h(w) = 0.5. Therefore, smaller values ​​of the parameters α and β provide a more robust threshold and a wider width for the threshold function h(w), thereby providing a relatively larger number of zero weights, or in other words, relatively fewer non-zero weights.

[0041] The parameters α and β in the threshold function h(w) can be trainable and optimized. As the parameters α and β become optimized, the shape of the threshold function h(w) converges to have optimal sharp edges and optimal spacing at h(w) = 0.5. The trainability of the parameters α and β provides a significant advantage over other pruning techniques that rely on iterative pruning and retraining, because the use of the threshold function h(w) causes the number of non-zero weights to be automatically optimized during backpropagation, rather than empirically selecting thresholds to ultimately reach an acceptable number of non-zero weights in different layers of the DNN. Optimally reduce the number of non-zero weights, thereby optimally reducing the computational burden on the device running the DNN.

[0042] Return to Figure 1 The combined pruning and quantization operation 104 in the layer can be used to quantize the pruned weights w of the layer l，p Determined as,

[0043]

[0044] Among them, w l is the weight of layer l, is a trainable parameter that controls the range of values ​​that will be pruned, and, is the weight w that controls pruning l，pA fixed constant for the sharpness of the function. and correspond to the parameters β and α of Equation 1, respectively.

[0045] Learning the parameters for quantizing the weights and the parameters for quantizing the output feature maps must find the number of bits that provides the best trade-off between accuracy and complexity. Instead of finding one optimal quantization level, a probability distribution that includes preferences for each available quantization level can be used. The most preferred quantization level receives the highest probability.

[0046] To learn the parameters for quantization, the range of allowable quantization levels of the weights can be set to q w =q w，min , ..., q w，max For example, q w The permissible value can be between 4 and 8 bits. The tendency of the quantization level for the weight can be expressed using a categorical distribution (i.e., a discrete probability distribution) as

[0047]

[0048] in, is a number between [0, 1] (with 1 representing the highest fraction achievable with the quantization level), w is a subscript indicating the probability associated with the weight, k indicates the quantization level (e.g., 5 bits), l indicates the layer of weights being quantized, Representation and quantization level q w The unnormalized probability of t corresponds to the trainable parameter, and t l，w is a trainable scaling parameter for categorical distributions.

[0049] The quantization of the layer weights is not performed at just one selected quantization level, but rather as a weighted average of all quantization levels as

[0050]

[0051] Among them, w l，q is the quantization-dequantization weight of layer l, quant k is the quantization of the weights in k bits, and dequant k is the dequantized value of the weight in k bits.

[0052] The pre-quantized output feature map for a layer is calculated using the quantized weights as determined by equation (5). Similar to the weights, the range of allowable quantization levels for the output feature map can be set to q f =q f，min , ..., q f，max , and the tendency of the quantization level for the output feature map can be expressed using a categorical distribution as

[0053]

[0054] in, is a number between [0, 1] (with 1 representing the highest score achievable with the quantization level), f is a subscript indicating the probability associated with the feature map, k indicates the quantization level, l represents the layer of the feature map being quantized, Representation and quantization level q f The unnormalized probability of t corresponds to the trainable parameter, and t l，f is a trainable scaling parameter for categorical distributions.

[0055] The output feature map is not quantized at only one selected quantization level, but is quantized as a weighted average of all quantization levels.

[0056]

[0057] Among them, f l，q is the quantization-dequantization output feature map for layer l, quant k is the quantization of the output feature map in k bits, and dequant k is the dequantization of the output feature map in k bits.

[0058] In addition to the trainable weights of the layer, the trainable parameters of each layer l include the pruning parameter β l，p , the unnormalized probability for each quantization level and And the distribution t l，w and t l，f The scaling parameter of the .

[0059] For a DNN pruned and quantized in a traditional way (i.e., pruning and quantization are separate independent operations), the total loss E(w) can be defined as

[0060]

[0061] Among them, E c is the cross-entropy loss, and E r is the L2 regularization of the weights.

[0062] for Figure 1 In the combined pruning and quantization operation 104 in , three losses can be included in the cost function. Pruning loss (where L is the maximum number of layers) can be added, which helps β become smaller and actually increases pruning. However, the pruning loss E p The cross entropy loss E of equation (8) can be chas a negative impact, so these two items should be balanced.

[0063] Weight quantization loss can be added, which helps with the optimization of putting all probability mass at the minimum quantization level. Doing so can also negatively affect the cross entropy loss E c .

[0064] Feature map quantization loss can be added, which is related to the weight quantization loss E w，q Operate in the same way.

[0065] During the backward pass of the joint pruning and quantization operation, the cost function that can be optimized can be

[0066]

[0067] Where N is the size of a minibatch that can be manually set in advance, and N can be a number (such as, but not limited to, 32, 64, 128, or 256), and the constant λ r , p , w，q and f，q It may be determined by grid-based parameter optimization or alternatively by random search.

[0068] Figure 6 is a flow chart of an example embodiment of a method 600 for performing a joint pruning and quantization operation on a layer of a neural network (such as a DNN) according to the subject matter disclosed herein. Figure 6Prior to the method 600 shown in , a neural network and a training image may be received, and the received neural network may be pre-trained using a first training image in the training images to generate a pre-trained neural network. For a layer of the pre-trained neural network, at 601, processing begins. At 602, a weight of the layer is pruned using an analytical threshold function (such as, but not limited to, the analytical threshold function of equation (1)). At 603, each weight of the layer remaining after pruning is quantized based on a weighted average of quantization and dequantization of the weights for all quantization levels to form a quantized weight of the layer. At 604, an output feature map of the layer is determined or calculated using the quantized weights formed in 603 and a second training image in the training images. Here, the second training image may be the same as or different from the first training image. At 605, each output feature map of the layer is quantized based on a weighted average of quantization and dequantization of the output feature maps for all quantization levels. At 606, a cost function is used to update the parameters of the analytical threshold function, the weighted average of all quantization levels of the weights, and the weighted average of all quantization levels of the output feature maps. At 607, the method 600 for the layer ends. The method 600 may be iteratively performed for a given layer of the neural network. Additionally, the method 600 may be performed on one or more of the multiple layers of the neural network.

[0069] Figure 7 A block diagram depicts an exemplary architecture 700 for a data processing system. In one embodiment, the architecture 700 may be used to implement Figure 1 A neural network analyzer 100.

[0070] The architecture 700 includes at least one processor, such as a central processing unit (CPU) 701, coupled to a memory element 702 via a system bus 703 or other suitable circuit. The architecture 700 may store program codes in the memory element 702. The processor 701 may execute program codes accessed from the memory element 702 via the system bus 703. Therefore, the processor 701 may be used as a dedicated processor. The memory element 702 may include one or more physical memory devices, such as, but not limited to, a local memory 704 and one or more mass storage devices 705. The local memory 704 may be a random access memory (RAM) or one or more other non-persistent memory devices typically used during the actual execution of the program code. The mass storage device 705 may be implemented as a hard disk drive (HDD), a solid state drive (SSD) and / or other persistent data storage devices. The architecture 700 may also include one or more cache memories (not shown), which provide temporary storage of at least some program codes so as to reduce the number of times that program codes must be obtained from the mass storage device 705 during execution.

[0071] The architecture 700 may also include input / output (I / O) devices, such as a keyboard 706, a display device 707, and / or a pointer device 708, which may be optionally coupled to the architecture 700. In some embodiments, one or more of the I / O devices may be combined in a touch screen used as a display device 707. Such a display device 707 may also include a keyboard 706 and a pointer device 708. One or more network adapters 709 may also be coupled to the architecture 700 to enable the architecture 700 to become coupled to other systems, computer systems, remote printers, and / or remote storage devices through an intervening private network or public network. Modems, cable modems, Ethernet cards, and wireless transceivers are non-limiting examples of different types of network adapters that can be used with the architecture 700. Depending on the specific device implemented with the architecture 700, the specific type of one or more network adapters may vary depending on the situation. The I / O device may be directly coupled to the architecture 700 or coupled to the architecture 700 through an intervening I / O controller.

[0072] like Figure 7 700. As depicted, memory element 702 may store an operating system 710 and one or more applications 711. In one aspect, operating system 710 and one or more applications 711 implemented in the form of executable program code may be executed by architecture 700. Thus, operating system 710 and one or more applications 711 may be considered an integral part of architecture 700. Operating system 710, one or more applications 711, and any data items used, generated, and / or operated by architecture 700 may be functional data structures that impart functionality when employed as part of a system implemented using architecture 700.

[0073] In one arrangement, application 711 may include one or more modules that, when executed by a system using architecture 700 or an architecture similar to architecture 700, may perform the functions described herein with reference to Figure 1 and Figures 3 to 6 Describes the various operations.

[0074] In another arrangement, the architecture 700 may be coupled to the platform 712 via a communication link 713. In one example, the architecture 700 may be coupled to the platform 712 via a network adapter 709. In another example, the architecture 700 may include one or more other I / O devices that may be used to couple the architecture 700 to the platform 712, such as a universal serial bus (USB) interface or other communication port. The platform 712 may be a circuit board and have a neural network accelerator 714 coupled thereto. In one arrangement, the neural network accelerator 714 may be implemented as an integrated circuit (IC) or multiple ICs. For example, the neural network accelerator 714 may be implemented as one or more programmable ICs (such as a field programmable gate array), one or more application specific ICs (ASICs), etc.

[0075] The architecture 700 may be configured to perform the operations described herein on a neural network or DNN to generate a pruned and quantized network, such as, Figure 1 That is, architecture 700 can receive trained neural network 101 via an I / O device, operate on one or more layers of the trained neural network, and output a trained neural network having one or more layers with weights and output feature maps that have been jointly pruned and quantized according to the subject matter disclosed herein via an I / O device. Architecture 700 can also provide a definition of pruned and quantized network 102 to neural network accelerator 714 for execution therein.

[0076] Figure 7 It is provided for illustration purposes only and, therefore, is not intended as a limitation on the inventive arrangements described herein. In some cases, a specific system implemented using architecture 700 may include fewer components or more components than shown. In addition, the specific operating system and / or one or more applications included as part of architecture 700 may vary.

[0077] As those skilled in the art will recognize, the innovative concepts described herein can be modified and varied over a wide range of applications. Therefore, the scope of the claimed subject matter should not be limited to any of the specific exemplary teachings discussed above, but rather defined by the claims.

Claims

1. A method for training a neural network for image recognition, the method comprising: Get training images; pre-training the neural network based on a first training image among the training images; Prune the weights of the layers of the pre-trained neural network using an analytical threshold function; quantizing each weight remaining after pruning of the layer based on a weighted average of quantization and dequantization of weights for all quantization levels to form a quantized weight of the layer; determining an output feature map of the layer based on the quantized weights of the layer and a second training image in the training images; quantizing each output feature map of the layer based on a weighted average of quantization and dequantization of the output feature maps for all quantization levels; and The cost function is used to update the parameters of the analytical threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer.

2. The method according to claim 1, wherein: The step of updating the parameters of the analytical threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer also includes: optimizing the cost function.

3. The method according to claim 2, wherein: The cost function includes pruning loss, weight quantization loss, and feature map quantization loss.

4. The method according to claim 1, further comprising: Iteratively prunes weights, quantizes each weight of the layer, determines an output feature map of the layer, quantizes each output feature map of the layer, and updates parameters of a resolution threshold function, a weighted average of all quantization levels of the weights, and a weighted average of each output feature map of the layer to optimize a cost function.

5. The method according to claim 4, wherein: The layer of the neural network is a first layer, The method further comprises: Using an analytical threshold function to prune the weights of the second layer of the pre-trained neural network, where the second layer comes after the first layer in the pre-trained neural network; quantizing each weight remaining after pruning of the second layer based on a weighted average of quantization and dequantization of the weights for all quantization levels to form quantized weights of the second layer; Determine an output feature map of the second layer based on the quantized weights of the second layer and the output feature map of the first layer; quantizing each output feature map of the second layer based on a weighted average of quantization and dequantization of the output feature maps for all quantization levels; and The cost function is used to update the parameters of the analytical threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the second layer.

6. The method according to claim 5, wherein: The step of updating the parameters of the analytical threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the second layer also includes: optimizing the cost function.

7. The method according to claim 1, wherein: The parameters of the analytical threshold function include a first parameter that controls the sharpness of the analytical threshold function, and a second parameter that controls the distance between a first edge and a second edge of the analytical threshold function.

8. An apparatus for training a neural network for image recognition, comprising: An interface receives a neural network, wherein the received neural network includes a plurality of layers; and A processing device for acquiring a training image; pre-training the received neural network based on a first training image in the training images; performing predetermined processing on the pre-trained neural network to generate a neural network including at least one layer having weights and output feature maps that have been jointly pruned and quantized, wherein the predetermined processing includes: using an analytical threshold function to prune the weights of the at least one layer of the pre-trained neural network, quantizing each weight remaining after pruning of the at least one layer based on a weighted average of quantization and dequantization of the weights for all quantization levels to form a quantized weight of the at least one layer, determining an output feature map of the at least one layer based on the quantized weights of the at least one layer and a second training image in the training images, quantizing each output feature map of the at least one layer based on a weighted average of quantization and dequantization of the output feature maps for all quantization levels, and using a cost function to update parameters of the analytical threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the at least one layer.

9. The device according to claim 8, wherein: The interface also outputs a neural network comprising the at least one layer having weights and output feature maps that have been jointly pruned and quantized.

10. The device according to claim 8, wherein: The pre-trained neural network is a neural network trained with full precision.

11. The device according to claim 8, wherein: The cost function includes pruning loss, weight quantization loss, and feature map quantization loss.

12. The device according to claim 11, wherein Parameters of the analytical threshold function, a weighted average of all quantization levels of the weights, and a weighted average of each output feature map of the at least one layer are updated based on the optimization of the cost function.

13. The device according to claim 11, wherein: Parameters of the analytical threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the at least one layer are iteratively updated based on the optimization of the cost function.

14. The device according to claim 8, wherein: The output feature map of the first layer is determined based on the quantized weight of the first layer and the second training image, and the output feature map of the i-th layer is determined based on the quantized weight of the i-th layer and the output feature map of the i-1-th layer, where i is an integer greater than 1 and less than or equal to the number of layers included in the neural network, The parameters of the analytical threshold function include a first parameter for controlling the sharpness of the analytical threshold function and a second parameter for controlling the distance between a first edge and a second edge of the analytical threshold function.

Images