Jointly prouning and quantizing deep neural neworks

Abstract

The system and method generate a neural network comprising at least one layer having jointly organized and quantized weights and output feature maps. The weights of the layer are organized using an analysis threshold function. Each weight remaining after organization is quantized based on the weighted average of the quantization and inverse quantization of the weights for all quantization levels to form the quantized weights for the layer. The output feature maps of the layer are generated based on the quantized weights of the layer. Each output feature map of the layer is quantized based on the weighted average of the quantization and inverse quantization of the output feature maps for all quantization levels. The parameters of the analysis threshold function, the weighted average of the weights for all quantization levels, and the weighted average of each output feature map of the layer are updated using a cost function.

Description

Technology Field

[0001] The subject matter disclosed herein relates to neural networks. More specifically, the subject matter disclosed herein relates to a system and method for jointly pruning weights and quantizing the output feature maps and weights of a layer of a neural network. Background Technology

[0002] The weights of the activation function and the output feature map occupy a massive amount of memory during the inference phase of a deep neural network (DNN). Neural networks also use a large amount of computation to calculate output inference. By pruning the weights and quantizing the activation function's output feature map and the weights, the memory requirements and computational load of the neural network can be reduced.

[0003] The conventional method of organizing and quantizing neural networks involves first organizing the network into individual, independent operations and then quantizing them. For example, refer to “Deep Compression: Compressing Deep Neural Networks with Organization, Learned Quantization, and Huffman Coding” by S. Han et al. (arxiv.org / 1510.00149). The effect of the separate operations introduced by Han is that, by receiving an organized network, optimizing the quantization parameters becomes more difficult. In effect, organization reduces the network’s acceptable state space by considering only one available organized model of the neural network. The problem to be solved

[0004] The technical problem that the present invention aims to solve is to provide a system and method for jointly organizing weights and quantizing the output feature map and weights of a neural network layer.

[0005] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0006] An exemplary embodiment provides a neural network comprising a plurality of layers, wherein at least one layer may include jointly organized and quantized weights and output feature maps. The jointly organized and quantized weights are organized using an analysis threshold function. Each remaining weight after organization is further quantized based on the weighted average of the quantization and inverse quantization of the weights for all quantization levels. An output feature map is formed based on the organized and quantized weights of the layer. Each output feature map may be quantized based on the weighted average of the quantization and inverse quantization of the output feature map for all weight levels. The parameters of the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer may be updated based on a cost function. In one embodiment, the neural network is a neural network that undergoes full precision learning before the weights and output feature maps of the at least one layer are jointly organized and quantized. In another embodiment, the cost function includes an organization loss term, a weight quantization loss term, and a feature map quantization loss term.

[0007] An exemplary embodiment provides a method for organizing weights and output feature maps of a layer of a neural network, the method comprising: a step of organizing weights of a layer of a neural network using an analysis threshold function—the neural network being a trained neural network—; a step of quantizing each weight of the layer remaining after organizing based on a weighted average of the quantization and inverse quantization of the weights for all quantization levels to form quantized weights for the layer; a step of determining an output feature map of the layer based on the quantized weights of the layer; a step of quantizing each output feature map of the layer based on a weighted average of the quantization and inverse quantization of the output feature map for all quantization levels; and a step of updating the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer using a cost function. In one embodiment, the step of updating parameters of the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer further comprises a step of optimizing the cost function.

[0008] An exemplary embodiment provides a neural network analyzer comprising an interface and a processing unit. The interface receives a neural network comprising a plurality of layers. The processing unit may generate a neural network comprising at least one layer having a jointly organized and quantized weight and an output feature map. The processing unit may organize the organized weights of the layers of the neural network using an analysis threshold function, and quantize each weight of the at least one layer remaining after organization based on the weighted average of the quantization and inverse quantization of the weights for all quantization levels to form quantized weights for the at least one layer, determine an output feature map of the at least one layer based on the quantized weights of the at least one layer, quantize each output feature map of the at least one layer based on the weighted average of the quantization and inverse quantization of the output feature maps for all quantization levels, and update the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the at least one layer using a cost function. In one embodiment, the interface further outputs a neural network comprising at least one layer having a joint theorem and quantized weights and output feature maps.

[0009] Specific details of other embodiments are included in the detailed description and drawings. Brief explanation of the drawing

[0010] In the following section, aspects of the subject matter disclosed in this specification will be described with reference to exemplary embodiments illustrated in the drawings. FIG. 1 illustrates a block diagram of a neural network analyzer capable of performing sorting and quantization joint operations on layers of a deep neural network (DNN) in accordance with the subject matter disclosed in this specification. Figure 2 illustrates an exemplary architecture of a DNN that can be input to the neural network analyzer of Figure 1 as a fully precise learned neural network. FIG. 3 illustrates the application of an exemplary layer and an analytical threshold function in a DNN to form an analytical weight function f(w) for optimally organizing the weights of an exemplary layer according to the subject matter disclosed in this specification. FIG. 4a illustrates a graph of an exemplary threshold function h(w) of Equation 1 according to the subject matter disclosed in this specification. FIG. 4b illustrates a graph of the weighting function f(w) of Equation 2 according to the subject matter disclosed in this specification. FIGS. 5a through 5e illustrate five exemplary graphs of the threshold function h(w) of Equation 1 having different values ​​for parameters α and β to provide a visual meaning of how parameters α and β can affect the threshold function h(w) and the weight function f(w) according to the subject matter disclosed herein. FIG. 6 is a flowchart of an exemplary embodiment of a method for performing a sorting and quantization joint operation for layers of a neural network, such as a DNN, in accordance with the subject matter disclosed in this specification. Figure 7 illustrates a block diagram of an exemplary architecture for a data processing system that can be used to implement the neural network analyzer of Figure 1. Specific details for implementing the invention

[0011] In the following detailed description, many specific details are described to provide a thorough understanding of the present disclosure. However, those skilled in the art will understand that certain disclosed embodiments may be practiced without these specific details. In other cases, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the essence disclosed herein.

[0012] Throughout this specification, references to “one embodiment” or “an embodiment” mean that specific features, structures, or characteristics described in relation to the embodiment may be included in at least one embodiment disclosed herein. Accordingly, the appearance of phrases “in one embodiment,” “in an embodiment,” or “according to one embodiment” (or other similar phrases) in various places throughout this specification does not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the word “exemplary” means “provided as an example, instance, or illustration.” Any embodiment described as “exemplary” in this specification should not be interpreted as necessarily being more preferable or advantageous than other embodiments. Furthermore, specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Also, depending on the context of the discussion herein, a singular form may include a corresponding plural form, and a plural form may include a corresponding singular form. Similarly, hyphenated terms (e.g., "2-dimensional," "predetermined," "pixel-specific," etc.) may occasionally be used interchangeably with their unhyphenated versions (e.g., "2-dimensional," "predetermined," "pixel-specific," etc.), and uppercase inputs (e.g., "Counter Clock," "Row Select," "PIXOUT," etc.) may be used interchangeably with their non-uppercase versions (e.g., "counter clock," "row select," "pixout," etc.). These interchangeable uses are not considered to be inconsistent with one another.

[0013] Additionally, depending on the context of the discussion in this specification, the singular form may include the corresponding plural form, and the plural form may include the corresponding singular form. It should be noted that the various drawings (including component diagrams) shown and discussed in this specification are for illustrative purposes only and are not drawn to scale. Similarly, various waveform and timing diagrams are shown for illustrative purposes only. For example, the dimensions of some components may be exaggerated relative to others for clarity. Also, where deemed appropriate, reference numbers in the drawings have been repeated to indicate corresponding and / or similar elements.

[0014] The terms used herein are merely for describing specific exemplary embodiments and are not intended to limit the claimed subject matter. Singular expressions include plural expressions unless the context clearly indicates otherwise. As used herein, the terms “comprising” and / or “comprising” are understood to specify the presence of the mentioned features, integers, steps, operations, elements and / or components, but not to exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. Terms used herein such as “first,” “second,” etc., are used as labels for nouns and do not imply any type of order (e.g., spatial, temporal, logical, etc.) unless explicitly defined. Therefore, the same reference number may be used across two or more drawings to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such use is for the sake of simplicity of explanation and ease of discussion only, and does not imply that the configuration or structural details of such components or units are identical across all embodiments or that a generally referenced part / module is the only method for implementing the teachings of a particular embodiment disclosed herein.

[0015] It will be understood that when a component or layer exists on another component or layer, or is referred to as "connected" or "connected," it may be directly connected to, connected to, or coupled to the other component or layer. Interposed elements or layers may exist. On the other hand, when a component is referred to as being "directly connected" to, "directly coupled" to, or "directly coupled" to another component, it should be understood that no other components exist in between. The same reference number refers to the same element throughout. As used herein, the term "and / or" includes any and all combinations of one or more related enumerated items.

[0016] Terms such as “first,” “second,” etc., as used herein are used as labels for nouns and do not imply any type of order (e.g., spatial, temporal, logical, etc.) unless explicitly defined. Additionally, the same reference number may be used across two or more drawings to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such use is solely for the sake of simplicity of description and ease of discussion. It does not imply that the configuration or structural details of such components or units are identical across all embodiments, or that these common reference parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

[0017] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains. Furthermore, it will be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with that in the context of the relevant technology, unless explicitly defined otherwise in this specification, and should not be interpreted in an unideal or overly formal sense.

[0018] As used herein, the term “module” refers to any combination of software, firmware, and / or hardware configured to provide the functionality described herein in relation to the module. Software may be implemented as a software package, code, and / or instruction set, or as instructions, and the term “hardware” as used in any implementation described herein may include, for example, wired circuits, programmable circuits, state machine circuits, and / or firmware that stores instructions executed by programmable circuits, either alone or in any combination. Modules may be implemented collectively or individually as circuits forming part of a larger system, for example, as circuits forming an integrated circuit (IC), a system-on-chip (SoC), etc. Various components and / or function blocks disclosed herein may be implemented as modules that may include software, firmware, and / or hardware providing the functionality described herein in relation to the various components and / or function blocks.

[0019] The subject matter disclosed herein provides a theorem and quantization co-operation that enables the optimization of a neural network to select the theorem model best suited for quantization and simultaneously select the quantized model best suited for the theorem. Accordingly, the subject matter disclosed herein allows a much larger set of acceptable solutions to be considered, thereby providing better theorem and quantization results.

[0020] FIG. 1 illustrates a block diagram of a neural network analyzer (100) capable of performing a sorting and quantization joint operation on the layers of a deep neural network (DNN) in accordance with the subject matter disclosed herein. In one embodiment, the neural network analyzer (100) may be implemented as a data processing system executing suitable software. As illustrated, the neural network analyzer (100) may receive a fully precise learned neural network (101) as input. The neural network analyzer (100) may operate on the layers of the neural network (101) by performing a sorting and quantization joint operation (104) on the weights and output feature maps of each layer of the neural network (101). The neural network analyzer (100) outputs a neural network (102) in which the layers of the neural network are sorted and quantized jointly.

[0021] The cleaning and quantization joint operation (104) performed at each layer may include a number of forward and backward passes or iterations for fine-tuning the cleaned and quantized weights and output feature maps for each layer of the DNN. During the forward pass through the layers of the DNN, the weights are removed and then quantized. Then, the output feature map is computed using the quantized weights. Then, the output feature map is quantized. During the backward pass, the cleaning parameters and the parameters for quantizing the cleaned weights and output feature maps are updated based on the optimization of the cost function.

[0022] In one embodiment, the fully precise learned neural network (101) and the clean and quantized neural network (102) may be a DNN. FIG. 2 illustrates an exemplary architecture (200) of a DNN that can be input to a neural network analyzer (100) as the fully precise learned neural network (101). The exemplary architecture (200) of the VGG (16) specifically includes 13 convolutional (CONV) layers, three fully connected (CONV) layers, and five pooling layers (PL). Input to the VGG (16) architecture (200) is applied to the convolutional layer (CONV1_1), and the output is provided in Classification.

[0023] To organize the weights of the layers of the DNN, an analysis threshold function h(w), characterized by the property that it does not affect weights with a size greater than the threshold and sets weights with a size smaller than the threshold to zero, may be multiplied to each weight. FIG. 3 illustrates the application of an exemplary layer (300) of the DNN and an analysis threshold function to form an analysis weight function f(w) for optimally organizing the weights of the exemplary layer (300) according to the subject matter disclosed herein. Regardless of the complexity of the neural network that may include the exemplary layer (300), the output feature map of the layer (300) may be formed as the sum of the products of the connection weight (w) and the input feature map for the output feature map, as shown on the left side of FIG. 3.

[0024] The analysis threshold function h(w) can form a layer (300') by applying it to the weight w of layer (300) in 301 to form a weight function f(w) that replaces the weight w. In one embodiment, each weight w may be multiplied by the analysis function h(w). A graph of an exemplary weight function f(w) is shown at the top center of FIG. 3. In one embodiment, the analysis threshold function h(w) is as follows:

[0025]

[0026] Here, α is a parameter that controls the sharpness of the threshold function h(w), and β is a parameter that controls the distance (or range) between the first and second edges of the threshold function. The smaller the parameter α, the sharper the profile of h(w) becomes, and the smaller the parameter β, the wider the width of h(w). Parameters α and β are scaling values ​​and have no units.

[0027] FIG. 4a illustrates a graph of an exemplary threshold function h(w) of Equation 1. The parameter α controls the sharpness of the edges (401a and 401b) of the threshold function. That is, the parameter α controls the rate of change of the edges (401a and 401b) between h(w) = 0 and h(w) = 1. The parameter β of Equation 1 controls the width (402) between the two edges (401a and 401b) at h(w) = 0.5.

[0028] As illustrated in FIG. 4a, the threshold function h(w) has a value of 0 for the first set (403) of continuous weight values ​​centered at 0. Additionally, the threshold function h(w) has a value of 1 for the second set (404) of continuous weight values ​​that is greater than the first set (403) of continuous weight values, and a value of 1 for the third set (405) of continuous weight values ​​that is smaller than the first set (403) of weight values. The first edge (401a) of the threshold function h(w) lies between the first set (403) of continuous weight values ​​and the second set (404) of continuous weight values. The second edge (401b) lies between the first set (403) of continuous weight values ​​and the third set (405) of continuous weight values.

[0029] The value of the threshold function h(w) varies between 0 and 1 in the region of the first and second edges (401a and 401b). When the threshold function h(w) varies between 0 and 1, the sharpness of each of the first and second edges (401a and 401b) is based on the value of the parameter α (i.e., the sharpness of the edges (401a and 401b) of the threshold function), and the distance (402) between the first and second edges (401a and 401b) at h(w) = 0.5 is based on the value of the parameter β.

[0030] In one embodiment, each weight w of the hierarchy can be multiplied by a threshold function h(w) to form a weight function f(w) as follows.

[0031]

[0032] FIG. 4b illustrates the graph of the weighting function f(w) of Equation 2. For the first set of weighting values ​​(403), the weighting function f(w) has a value of 0. The weighting function f(w) has a value of w for the second set of continuous weighting values ​​(404) which is greater than the first set of continuous weighting values ​​(403), and a value of w for the third set of continuous weighting values ​​(405) which is smaller than the first set of continuous weighting values ​​(403). The weighting function f(w) includes transition regions (406a and 406b) corresponding to the first edge (401a) and the second edge (401b), respectively. As the value of the parameter α decreases, the first and second edges (401a, 401b) become sharper, and the transition regions (406a, 406b) become smaller in the w direction.

[0033] FIGS. 5a through 5e illustrate five exemplary graphs of the threshold function h(w) of Equation 1, providing visual implications of how parameters α and β can affect the threshold function h(w) and the weight function f(w) with different values ​​for each parameter. In FIGS. 5a through 5c, the parameter α changes from α = 1.0 x 10⁻⁸ (Fig. 5a) to α = 1.0 x 10⁻² (Fig. 5c), while the parameter β remains constant at β = 4. In FIGS. 5a through 5c, as the value of α decreases, the threshold function h(w) varies more sharply between 0 and 1. In Figs. 5d and 5e, the parameter β changes from β = 8 (Fig. 5D) to β = 16 (Fig. 5e), while the parameter α remains constant at α = 1.0 x 10⁻⁸. In Figs. 5d and 5e, it can be seen that as the parameter β decreases, the width between the edges of the threshold function h(w) at h(w) = 0.5 becomes larger. Therefore, as the values ​​of parameters α and β decrease, a stronger threshold and a wider width are provided for the threshold function h(w), which provides a relatively large number of zero weights, or in other words, a relatively small number of non-zero weights.

[0034] The parameters α and β in the threshold function h(w) are learnable and optimizable. As the parameters α and β are optimized, the shape of the threshold function h(w) converges to have optimally sharp edges and an optimal gap at h(w) = 0.5. The learnability of the parameters α and β offers a significant advantage over other cleaning techniques that rely on iterative cleaning and retraining, because using the threshold function h(w) automatically optimizes the non-zero weights during backpropagation, instead of empirically selecting a threshold value to reach an acceptable number of non-zero weights in other layers of the DNN. In this way, optimally reducing the number of non-zero weights optimally reduces the computational burden on the device running the DNN.

[0035] Returning to the organization and quantization joint operation (104) of FIG. 1, the weights of the organized layer It can be determined as follows.

[0036]

[0037] Here is the weight of layer l, and is a learnable parameter that controls the range of values ​​to be cleaned up, and is the organized weight It is a fixed constant that controls the sharpness of the function. Parameter and correspond to the parameters β and α of Equation 1, respectively.

[0038] Learning the parameters for weight quantization and the parameters for output feature map quantization involves finding the number of bits that provides the best balance between accuracy and complexity. Instead of finding a single optimal quantization level, a probability distribution containing preferences for each available quantization level can be used. The most desirable quantization level receives the highest probability.

[0039] To learn the parameters for quantization, the range of acceptable quantization levels for the weights can be set to qw = qw, min, ..., qw, max. For example, qw can take acceptable values ​​between 4 bits and 8 bits. The preference for quantization levels for the weights can be expressed using the following categorical distribution (i.e., discrete probability distribution).

[0040]

[0041] Here is a number between [0,1] (1 represents the highest score achievable by the quantization level), w is an index indicating that the probability is associated with the weight, qw represents the quantization level (e.g., 5 bits), and l represents the hierarchy of weights being quantized, represents a learnable parameter corresponding to the unnormalized probability of quantization level q, and is a learnable scaling parameter of a categorical distribution.

[0042] The quantization of the layer's weights is not performed at only one selected quantization level, but instead is performed as a weighted average of all quantization levels as follows.

[0043]

[0044] Here is the quantization-inverse quantization weight for layer l, and is the quantization of weights at k bits, and is the inverse quantization of the weight at k bits.

[0045] The pre-quantized output feature map for the layer is calculated using quantized weights determined by Equation 5. Similar to the weights, the range of acceptable quantization levels for the output feature map is qf = qf, min, ..., qf, max, and the preference for quantization levels for the output feature map can be represented using the following categorical distribution.

[0046]

[0047] Here, is a number between [0,1] (1 represents the highest score achievable by the quantization level), and f is an index indicating that the probability is associated with the feature map, and represents the quantization level, and l represents the hierarchy of feature maps being quantized, represents a learnable parameter corresponding to the unnormalized probability of quantization level q, and is a learnable scaling parameter of a categorical distribution.

[0048] The output feature map is also not quantized at a single selected quantization level, but instead is quantized as a weighted average of all quantization levels as follows.

[0049]

[0050] Here, is a quantized-indequantized output feature map for layer l, and is the quantization of the output feature map at k bits, and is the inverse quantization of the output feature map at k bits.

[0051] The learnable parameters for each layer l are, in addition to the layer's learnable weights, the cleanup parameters , unnormalized probability for each quantization level and , and distribution and Includes the scaling parameters of.

[0052] In the case of a DNN that is organized and quantized in the conventional manner (i.e., organization and quantization are separate, independent operations), the total loss E(w) can be defined as follows.

[0053]

[0054] Here, is the cross-entropy loss. is L2 normalization for weights.

[0055] For the theorem and quantization joint operation (104) of FIG. 1, three theorem losses may be included in the cost function. Theorem losses It helps to minimize β, effectively increasing the theorem, where L is the maximum number of layers. However, the theorem loss is the cross-entropy loss in Equation 8 Since it can have a negative impact, the two terms must be balanced.

[0056] Weighted quantization loss The addition of helps optimize the placement of all probability masses at the smallest quantization level. Doing so may have a negative impact on the cross-entropy loss EC.

[0057] Feature map quantization loss It can be added to operate in the same way as the weight quantization loss Ew,q.

[0058] The cost function that can be optimized during the reverse pass of the theorem and quantization co-operation is as follows.

[0059]

[0060] Here, N is the size of the mini-batch, which can be manually set in advance, and can be a number such as 32, 64, 128, or 256, but is not limited to these. Constant , , , and It can be determined by grid-based parameter optimization or alternatively by random search.

[0061] FIG. 6 is a flowchart of an exemplary embodiment of a method (600) for performing a combined cleanup and quantization operation on layers of a neural network, such as a DNN, in accordance with the subject matter disclosed herein. For one layer, the process begins at 601. At 602, the weights of the layer are removed using an analysis threshold function, such as the analysis threshold function of Equation 1. At 603, each weight of the layer remaining after cleanup is quantized based on the weighted average of the quantization and inverse quantization of the weights for all quantization levels to form a quantized weight for the layer. At 604, an output feature map for the layer is determined or calculated using the quantized weights formed at 603. At 605, each output feature map of the layer is quantized based on the quantization and inverse quantization of the output feature map for all quantization levels. At 606, the parameters of the analysis function, the weighted average of all quantization levels of the weights, and the weighted average of all quantization levels of the output feature map are updated using a cost function. At 607, the method (600) for this layer is terminated. The method (600) can be performed repeatedly for a given layer of the neural network. Additionally, the method (600) can be performed on one or more layers of the neural network.

[0062] FIG. 7 illustrates a block diagram of an exemplary architecture (700) for a data processing system. In one embodiment, the architecture (700) can be used to implement the neural network analyzer (100) of FIG. 1.

[0063] The architecture (700) includes at least one processor, for example, a central processing unit (CPU) (701) connected to a memory element (702) via a system bus (703) or other suitable circuitry. The architecture (700) may store program code within the memory element (702). The processor (701) may execute program code accessed from the memory element (702) via the system bus (703). In this way, the processor (701) may function as a specialized processor. The memory element (702) may include, but is not limited to, one or more physical memory devices such as local memory (704) and one or more bulk storage devices (705). The local memory (704) may be random access memory (RAM) or other non-persistent memory devices typically used during the actual execution of program code. The bulk storage device (705) may be implemented as a hard disk drive (HDD), a solid-state drive (SSD), and / or other persistent data storage devices. The architecture (700) may also include one or more cache memories (not shown) that provide temporary storage of at least some program code to reduce the number of times the program code must be retrieved from the bulk storage device (705) during execution.

[0064] The architecture (700) may also include input / output (1 / O) devices, such as a keyboard (706), a display device (707), and / or a pointing device (708), which may be optionally coupled to the architecture (700). The display device (707) may also include the keyboard (706) and the pointing device (708). One or more network adapters (709) are also coupled to the architecture (700) so that the architecture (700) can be connected to other systems, computer systems, remote printers, and / or remote storage devices through the arbitration of a private or public network. Modems, cable modems, Ethernet cards, and wireless transceivers are non-limiting examples of different types of network adapters that may be used with the architecture (700). Depending on the specific device implemented in the architecture (700), the specific type of network adapter or network adapters may vary in the case. I / O devices may be connected to the architecture (700) directly or through the arbitration of an I / O controller.

[0065] As illustrated in FIG. 7, the memory element (702) may store an operating system (710) and one or more applications (711). In one aspect, the operating system (710) and application(s) (711), implemented in the form of executable program code, may be executed by the architecture (700). As such, the operating system (710) and application(s) (711) may be considered as an integrated part of the architecture (700). The operating system (710), application(s) (711), and any data item used, created, and / or operated by the architecture (700) may be a functional data structure that provides functionality when employed as part of a system implemented using the architecture (700).

[0066] In one configuration, the application (711) may include one or more modules capable of performing various operations described in the present specification with reference to FIGS. 1 and FIGS. 3 through 6 when executed by a system using architecture (700) or architecture similar to architecture (700).

[0067] In another configuration, the architecture (700) may be connected to the platform (712) via a communication link (713). In one example, the architecture (700) may be connected to the platform (712) via a network adapter (709). In another example, the architecture (700) may include one or more other 1 / O devices, such as a Universal Serial Bus (USB) interface or other communication port, that can be used to connect the architecture (700) to the platform (712). The platform (712) may be a circuit board and may have a neural network accelerator (714) connected thereto. In one configuration, the neural network accelerator (714) may be implemented as an integrated circuit (IC) or a plurality of ICs. For example, the neural network accelerator (714) may be implemented as one or more programmable ICs, such as a field programmable gate array, one or more application-specific ICs (ASICs), etc.

[0068] The architecture (700) may be configured to perform operations described herein in a neural network or DNN to generate a theorem and quantization network such as the theorem and quantization neural network (102) of FIG. 1. That is, the architecture (700) may receive a learned neural network (101) through an I / O device, operate on one or more layers of the learned neural network, and output a learned neural network through an I / O device having layers having jointly theorem and quantized weights and output feature maps according to the subject disclosed herein. The architecture (700) may also provide the definition of the theorem and quantization neural network (102) to a neural network accelerator (714) for internal execution.

[0069] FIG. 7 is provided for illustrative purposes only and is therefore not intended to be a limitation of the configuration of the invention described herein. In some cases, a specific system implemented using the architecture (700) may include fewer or more components than illustrated. Additionally, the specific operating system and / or application(s) included as part of the architecture (700) may vary.

[0070] As recognized by those skilled in the art, the innovative concepts described herein may be modified and changed over a wide range of applications. Accordingly, the scope of the claimed subject matter should not be limited to any of the specific exemplary teachings discussed above, but is instead defined by the following claims. Explanation of the symbols

[0071] 100 : Neural Network Analyzer 600: Method for performing theorem and quantization cooperation 700: An Exemplary Architecture for Data Processing Systems

Claims

Claim 1 A data processing system comprising a neural network, wherein the neural network comprises a plurality of layers, and at least one layer comprises pruned and quantized weights and an output feature map quantized based on said pruned and quantized weights, said pruned and quantized weights are pruned using an analysis threshold function and quantized based on a weighted average of the quantization and inverse quantization of said weights for all quantization levels, said output feature map is formed based on said pruned and quantized weights of said layer, and each output feature map is quantized based on the quantization of said output feature map for all weight levels, and the parameters of said analysis threshold function, said weighted average of said weights for all quantization levels, and said weighted average of each output feature map of said layer are updated based on a cost function. Claim 2 A data processing system according to claim 1, wherein the neural network is a neural network that is fully-precision trained before the weights of at least one layer and the output feature map are organized and quantized. Claim 3 A data processing system according to claim 1, wherein the cost function comprises a cleanup loss term, a weighted quantization loss term, and a feature map quantization loss term. Claim 4 A data processing system in which, in paragraph 3, the parameters of the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer are updated based on the optimization of the cost function. Claim 5 A data processing system according to paragraph 3, wherein the parameters of the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer are iteratively updated based on the optimization of the cost function. Claim 6 A data processing system according to claim 1, wherein the parameter of the analysis threshold function comprises a first parameter controlling the sharpness of the analysis threshold function and a second parameter controlling the distance between a first edge and a second edge of the analysis threshold function. Claim 7 A method for pruning weights and output feature maps of layers of a neural network, comprising: a step of pruning the weights of a layer of a neural network using an analysis threshold function by at least one processor, wherein the neural network is a learned neural network; a step of quantizing each weight of the layer remaining after pruning based on a weighted average of the quantization and inverse quantization of the weights for all quantization levels by the at least one processor to form quantized weights for the layer; a step of determining an output feature map of the layer based on the quantized weights of the layer by the at least one processor; a step of quantizing each output feature map of the layer based on a weighted average of the quantization and inverse quantization of the output feature map for all quantization levels by the at least one processor; and a step of updating the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer by the at least one processor using a cost function. Claim 8 In claim 7, the step of updating the parameters of the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer further comprises the step of optimizing the cost function. Claim 9 In claim 8, the above cost function comprises a cleanup loss term, a weighted quantization loss term and a feature map quantization loss term, a method. Claim 10 A method according to claim 7, further comprising the steps of iteratively organizing the weights, quantizing each weight of the layer, determining the output feature map of the layer, quantizing each output feature map of the layer, and updating the parameters of the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the layer to optimize the cost function. Claim 11 In claim 10, the layer of the neural network is a first layer, and the method comprises: a step of organizing the weights of a second layer of the neural network using an analysis threshold function by the at least one processor, wherein the second layer in the neural network follows the first layer; a step of quantizing each weight of the second layer remaining after organizing based on a weighted average of the quantization and inverse quantization of the weights for all quantization levels by the at least one processor to form quantized weights for the second layer; a step of determining the output feature map of the second layer based on the quantized weights of the second layer by the at least one processor; and a step of quantizing each output feature map of the second layer based on a weighted average of the quantization and inverse quantization of the output feature map for all quantization levels by the at least one processor. A method further comprising the step of updating the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the second layer using the cost function by at least one processor. Claim 12 In claim 11, the step of updating the parameters of the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of the second layer further comprises the step of optimizing the cost function. Claim 13 A method according to claim 7, wherein the parameters of the analysis threshold function include a first parameter controlling the sharpness of the analysis threshold function and a second parameter controlling the distance between the first edge and the second edge of the analysis threshold function. Claim 14 In a neural network analyzer, an interface for receiving a neural network, wherein the neural network is an interface comprising a plurality of layers; A neural network analyzer comprising a processing unit for generating a neural network having at least one layer having pruned and quantized weights and a quantized output feature map based on said pruned and quantized weights, wherein the processing unit prunes the pruned weights of the layer of the neural network using an analysis threshold function, quantizes each weight of said at least one layer remaining after pruning based on a weighted average of the quantization and inverse quantization of said weights for all quantization levels to form quantized weights for said at least one layer, determines the output feature map of said at least one layer based on said quantized weights of said at least one layer, quantizes each output feature map of said at least one layer based on a weighted average of the quantization and inverse quantization of said output feature maps for all quantization levels, and updates said analysis threshold function, said weighted average of said weights for all quantization levels, and said weighted average of each output feature map of said at least one layer using a cost function. Claim 15 In claim 14, the above interface further outputs a neural network having at least one layer having organized and quantized weights and a quantized output feature map based on the organized and quantized weights, a neural network analyzer. Claim 16 In paragraph 14, the neural network analyzer is a neural network in which the weights of at least one layer are organized and quantized, and the output feature map is fully precise learned before being quantized. Claim 17 In paragraph 14, the above cost function comprises a cleanup loss term, a weighted quantization loss term and a feature map quantization loss term, a neural network analyzer. Claim 18 A neural network analyzer according to claim 17, wherein the parameters of the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of at least one layer are updated based on the optimization of the cost function. Claim 19 A neural network analyzer according to claim 17, wherein the parameters of the analysis threshold function, the weighted average of all quantization levels of the weights, and the weighted average of each output feature map of at least one layer are iteratively updated based on the optimization of the cost function. Claim 20 A neural network analyzer according to claim 14, wherein the parameters of the analysis threshold function include a first parameter controlling the sharpness of the analysis threshold function and a second parameter controlling the distance between a first edge and a second edge of the analysis threshold function.

Images