System-aware selective quantization for distributed deep learning for performance optimization

Abstract

A convolutional neural network includes a front layer, a back layer, and a plurality of other layers connected between the front and back layers. One of the other layers is a transition layer. Activations of neurons from the front layer back to the transition layer are assigned a first precision, and activations of neurons from the transition layer back to the back layer are assigned a second precision. Weights of inputs to neurons from the front layer back to the transition layer are assigned a third precision, and weights of inputs to neurons from the transition layer back to the back layer are assigned a fourth precision. In some embodiments, layers preceding the transition layer have different convolutional kernels than layers following the transition layer.

Description

Background Technology

[0001] This invention relates to the fields of electrical, electronic and computer science, and more particularly to machine learning structures such as neural networks.

[0002] A neural network is an execution model for training or classifying machine learning algorithms. Typically, the computational operations of this model are performed on multiple processors configured to work together.

[0003] In neural networks, the processor simulates thousands or millions of neurons connected by axons and synapses. Each connection is either enforced, inhibited, or neutralized in its effect on the activation state of the connected neural unit. Each individual neural unit has a summation function that multiplies each of its inputs by a corresponding weight, and then sums the weighted values ​​to obtain the total signal. In some implementations, a threshold or constraint function exists on at least some connections and / or at least some neural units such that the total signal must exceed the constraint before the constrained neural unit will be activated. Once a neural unit is activated, it sends the activation signal as input to one or more other neural units. Neural networks can enable supervised, unsupervised, or semi-supervised machine learning.

[0004] The typical output of a deep neural network (DNN) is to label the input as belonging to one of several categories. Typically, the labeled output is based on patterns learned from a training set of inputs and labels. DNNs use very large models with high computational complexity to achieve high accuracy in applications; these very large models require 100 megabytes (MB) of data storage and high bandwidth for data movement. This high computational complexity poses a technical challenge for widespread deployment, especially in resource-constrained edge environments.

[0005] Despite the above, it is known that neural networks can make mistakes in assigning labels because some labels assigned by neural networks are inconsistent with those assigned by human experts who review the same input. Summary of the Invention

[0006] The principles of this invention provide a system-aware selective quantization technique for distributed deep learning for performance optimization.

[0007] In one aspect, an exemplary method includes constructing a convolutional neural network. The convolutional neural network includes: a front layer having a first number of neurons and a first convolutional kernel; a rear layer having a second number of neurons and a second convolutional kernel, wherein the second number of neurons is less than the first number of neurons; and a plurality of other layers connecting the front layer and the rear layer. The method further includes selecting a transition layer from the plurality of other layers having a third number of neurons and a third convolutional kernel; assigning a first activation precision to the activations of neurons returning from the front layer to the transition layer, and assigning a second activation precision to the activations of neurons returning from the transition layer to the rear layer; and assigning a first weight precision to the weights of the inputs of neurons returning from the front layer to the transition layer, and assigning a second weight precision to the weights of the inputs of neurons returning from the transition layer to the rear layer. The first activation precision differs from the second activation precision. The first weight precision differs from the second weight precision. The transition layer is selected to correspond to a transition from activation-heavy layers to weight-heavy layers.

[0008] In another aspect, an exemplary method includes constructing a convolutional neural network. The convolutional neural network includes: a front layer having a first number of neurons and a first convolutional kernel; a rear layer having a second number of neurons and a second convolutional kernel, wherein the second number of neurons is less than the first number of neurons; and a plurality of other layers connecting the front layer and the rear layer. The method includes selecting a transition layer from the plurality of other layers having a third number of neurons and a third convolutional kernel; assigning the first convolutional kernel to neurons returning from the front layer to the transition layer; and assigning the second convolutional kernel to neurons returning from the transition layer to the rear layer. The first convolutional kernel has a first compression by reducing a first fixed percentage of values ​​less than the maximum value in the kernel window to zero. The second convolutional kernel has a second compression by reducing a second fixed percentage of values ​​less than the maximum value in the kernel window to zero. The second fixed percentage is less than the first fixed percentage. The transition layer is selected to correspond to a transition from an activation-heavy layer to a weight-heavy layer.

[0009] In another aspect, an exemplary method includes constructing a convolutional neural network. The convolutional neural network includes: a front layer having a first number of neurons and a first convolutional kernel; a rear layer having a second number of neurons and a second convolutional kernel, wherein the second number of neurons is less than the first number of neurons; and a plurality of other layers connecting the front layer and the rear layer. The method includes selecting a transition layer from the plurality of other layers having a third number of neurons and a third convolutional kernel; assigning a first activation precision to activations of neurons returning from the front layer to the transition layer, and assigning a first weight precision to weights of inputs of neurons returning from the front layer to the transition layer; and assigning a second activation precision to activations of neurons returning from the transition layer to the rear layer, and assigning a second weight precision to weights of inputs of neurons returning from the transition layer to the rear layer. The first activation precision is less than the first weight precision. The second activation precision is greater than the second weight precision. The transition layer is selected to correspond to a transition from an activation-heavy layer to a weight-heavy layer.

[0010] One or more embodiments of the present invention or elements thereof may be implemented in the form of a computer program product comprising a computer-readable storage medium having computer-usable program code for performing or facilitating indicated method steps. Furthermore, one or more embodiments of the present invention or elements thereof may be implemented in the form of a system (or apparatus) comprising a memory and at least one processor containing computer-executable instructions coupled to the memory and operable by the instructions to perform or facilitate exemplary method steps. Additionally, in another aspect, one or more embodiments of the present invention or elements thereof may be implemented in the form of an apparatus for performing one or more method steps described herein; such apparatus may include: (ii) a hardware module, (iii) a software module stored in a tangible computer-readable storage medium (or multiple such media) and implemented on a hardware processor, or (iii) a combination of (i) and (ii); any one of (i)-(iii) implements the specific techniques set forth herein.

[0011] As used herein, a “facilitating” action includes performing an action, making an action easier, assisting in performing an action, or causing an action to be performed. Therefore, by way of example and not limitation, instructions executing on one processor can facilitate an action performed by instructions executing on a remote processor by sending appropriate data or commands to prompt or assist in performing the action to be performed. To avoid confusion, in cases where an actor facilitates an action by performing actions other than the action itself, the action is still performed by an entity or combination of entities.

[0012] In view of the above, the technology of the present invention can provide substantially beneficial technical effects. For example, one or more embodiments provide one or more of the following: reducing the memory footprint (and bandwidth requirements) of the neural network while maintaining model accuracy; increasing the computational throughput of the convolutional neural network without reducing classification accuracy; improving the classification accuracy of the convolutional neural network without increasing the computational resource load; and optimizing the quantization of activations and weights in the neural network in response to the hardware constraints of the system executing the neural network.

[0013] These and other features and advantages of the invention will become apparent from the following detailed description of illustrative embodiments of the invention, which are read in conjunction with the accompanying drawings. Attached Figure Description

[0014] Figure 1 An exemplary convolutional neural network is schematically depicted;

[0015] Figure 2 The table depicts the percentage of processing engine utilization that can be improved through various aspects of the present invention;

[0016] Figure 3 The prediction accuracy of the optimized convolutional neural network according to an exemplary embodiment of the present invention is depicted in the form of a graph.

[0017] Figure 4 A method for selecting the location of a transition layer in a convolutional neural network, according to an exemplary embodiment, is schematically depicted.

[0018] Figure 5 A method for optimizing a convolutional neural network according to an exemplary embodiment is described in the form of a flowchart;

[0019] Figure 6 Another method for optimizing a convolutional neural network according to an exemplary embodiment is depicted in the form of a flowchart; and

[0020] Figure 7 A computer system is described that can be used to implement one or more aspects and / or elements of the present invention. Detailed Implementation

[0021] Figure 1An exemplary convolutional neural network (CNN) 100 is depicted. CNN 100 receives an input matrix 102 (typically a matrix of pixel values, which can be of any dimension) and produces classification labels 104. CNN 100 includes a front layer 106, which is a convolutional layer that receives the input matrix 102. CNN 100 also includes a classification layer (fully connected layer) 108 that produces the classification labels 104. Between the front layer 106 and the classification layer 108 are several other convolutional layers, including a back layer 110 and a transition layer. The back layer 110 is the layer closest to the fully connected layer 108 and has the fewest number of neurons, each tuned to recognize high-level features of the input matrix 102. The transition layer is connected somewhere between the front layer 106 and the back layer 110.

[0022] CNN is a type of deep neural network (DNN). DNNs use very large models that require 100 megabytes (MB) of data storage and high bandwidth for data movement to achieve high accuracy in computationally complex applications. This model complexity poses a significant challenge for widespread deployment, especially in resource-constrained edge environments. One or more embodiments provide a method and apparatus to reduce memory footprint (and bandwidth requirements) while maintaining as much model accuracy as possible.

[0023] In the operation of CNN 100, each neuron in each convolutional layer scans activations from neurons selected from the previous layer, assigns weights to the selected neurons, and outputs activations based on the sum of weighted activations received from the selected neurons. In fully connected layers 108, each neuron from the previous layer is connected to every neuron in that layer. Typically, the output activations of fully connected layers are used to generate classification labels 104.

[0024] Generating activations for each neuron is computationally intensive, and the computational workload increases with the precision of the activations. Generally, higher precision activations (closer to the simulation) lead to more accurate classifications, although there are limitations in the neural network architecture that stem from the precision gain of higher precision. Computational performance (reducing workload) can be enhanced by decreasing the precision of the selected weights or activations. Aspects of the present invention relate to how to select weights and activations that will have reduced precision (“quantized”).

[0025] Generally, when a CNN is trained with more data, it becomes more robust to quantization (accuracy changes less with precision). For example, a CNN provided with one thousand training images is more robust than the same CNN provided with only twelve training images. Similarly, a CNN provided with one thousand images of 4096×3072 pixels will be more robust than the same CNN provided with one thousand images of 512×384 pixels. Therefore, one aspect of the invention is the discovery that when there are a large number of activations working together (i.e., in the convolutional layers near the previous layer 106, many neurons provide many outputs), the precision of the activations can be reduced more significantly, and when there are a large number of weights working together (i.e., in the convolutional layers near the subsequent layer 110, fewer neurons receive many inputs), the precision of the weights can be reduced more significantly. In one or more embodiments, the transition layer 112 distinguishes the boundary between quantized activations (between the previous layer 106 and the transition layer) and quantized weights (between the transition layer and the subsequent layer 110).

[0026] Based on the above explanation, it will be understood that the choice of transition layers has a significant impact on both the accuracy and computational efficiency of CNN 100. Figure 2 Table 200 illustrates how to improve the utilization of the Example Processing Engine (PE) of the example CNN (ResNet-18) by selectively quantizing the activations, weights, or both at different layers. The first column shows the results of half-precision quantization at all layers. The second column shows the results of ternary (2-bit) quantization of the weights at all layers. The third column shows the results of ternary quantization of the activations at all layers. The fourth column shows the results of ternary quantization of the weights and activations at all layers. The fifth column shows the results of ternary quantization of the activations from the previous layer 106 to the transition layer 112, and the results of ternary quantization of the weights from the transition layer 112 to the next layer 110. For example, if the precision of the weights and activations is halved at all layers, the PE utilization at layer CONV2_4 is 46.0%. If only the weights of all layers are quantized to two-bit precision, the PE utilization is 46.3%. If only the activations at all layers are quantized to two-bit precision, the PE utilization is 87.1%. Empirically, quantizing the activations yields the greatest benefit in terms of computational efficiency at layer CONV2_4. Further down, at layer CONV5_1, quantization weights result in a PE utilization rate of 79.1%, while quantization activation results in a PE utilization rate of only 73.5%.

[0027] Figure 3 The curve 300 depicts when Figure 1The convolutional neural network 100 has been optimized for its prediction accuracy according to an exemplary embodiment. The prediction accuracy at different training epochs is shown as the number of training steps increases from left to right. At 302, accuracy is optimal for high-precision values ​​of activations and weights. At 304, suboptimal accuracy is achieved only in the convolutional layers of group 3 for two-bit quantized weight values. (Note that group 3 on the graph corresponds to Table 200 and...) Figure 1 CONV4_1 to CONV4_4 in the diagram.) At 306, the third best accuracy is obtained by using two-bit quantized weight values ​​in the group 3 convolutional layers and two-bit quantized activation values ​​in the group 0-group 2 convolutional layers. (Note that group 0 on the graph corresponds to Table 200 and...) Figure 1 In CONV1_1, and group 2 corresponds to CONV3_1 to CONV3_4). In 308, the least accurate method is to quantize the weights in all layers.

[0028] Therefore, one aspect of the present invention is a method 400 for selecting a transition layer 112, wherein activations are quantized up to the transition layer 112, and weights are quantized after the transition layer 112, such as... Figure 4 As depicted in the flowchart. At 401, hardware system parameters / constraints 402 and a description 403 of the neural network 100 are fed into an analytical performance estimation model 404, which iteratively modifies the neural network description 403 to approximate the optimal value of a selected hardware system metric. For example, for illustrative purposes, execution latency can be selected as the metric of interest. Other metrics such as throughput, power, or energy efficiency can also be used as metrics of interest. Based on system constraints (bandwidth, on-chip memory capacity, or computing power), the performance estimator generates weights and activation precisions for each data structure of each layer of the neural network that are expected to produce optimal values ​​for the metric of interest. For example, the performance estimator can scan possible precision configurations, and for each precision selection, it checks data reuse and memory footprint to estimate latency for a given system constraint. Therefore, at 404, method 400 recommends which layers of the neural network 100 should be transition layers 112. Typically, activations are quantized before transition layers 112 when weights are of high precision. Weights are then quantized after layer 112 while activations are of high precision. In layer 112, in some embodiments, both weights and activations are high-precision. In other embodiments, both weights and activations are quantized at layer 112. In some embodiments, at layer 112, weights are quantized while activations are high-precision. In other embodiments, at layer 112, activations are quantized while weights are high-precision. The quantization of weights or activations in transition layers will depend on the estimated performance; for example, if weights are more dominant than activations in that layer, the performance estimator will indicate the use of lower-precision weights.

[0029] At 406, a neural network 100 (neural network simulation) is performed on the training data using the recommended quantization. As an example, the ImageNet dataset (Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015) can be used for image classification tasks. At 408, it is determined whether the desired classification accuracy is achieved through selective quantization of the weights and / or activations of each layer. As a non-limiting example, in one embodiment, good accuracy is an error rate of less than 3%; in another embodiment, good accuracy is an error rate of less than 2%. In yet another embodiment, acceptable accuracy is an error rate that degrades from the full-precision baseline accuracy by no more than 1%. For any given application, the exact value of good accuracy will be determined empirically by a person skilled in the art and is not weighted within the scope of the invention. However, one aspect of the invention is the step of determining whether good accuracy has been achieved. If not, then the recommendation of layer 112 is adjusted.

[0030] Once the accuracy of the given recommendation for transition layer 112 has been confirmed, at 410, according to description 403, neural network 100 is deployed on test data, where each layer has quantized weights and activations. As previously discussed, before transition layer 112, activations are quantized while weights are at high precision; after transition layer 112, weights are quantized while activations are at high precision. The processing of weights and activations at layer 112 varies according to various embodiments.

[0031] Figure 5 A method 500 for optimizing a convolutional neural network is depicted in the form of a flowchart. In 502, a neural network is constructed comprising a front layer, a back layer, and other layers connecting the front and back layers. In 504, a transition layer is selected from multiple other layers based on determining the transition from the activation layer to the weight layer. In 506, a first activation precision is assigned to the activations of neurons returning from the front layer to the transition layer. In 508, a second activation precision is assigned to the activations of neurons returning from the transition layer to the back layer. In 510, a first weight precision is assigned to the weights of the inputs of neurons returning from the front layer to the transition layer, and in 512, a second weight precision is assigned to the weights of the inputs of neurons returning from the transition layer to the back layer.

[0032] Figure 6Another method 600 for optimizing convolutional neural networks is described in flowchart form. In 602, a neural network is constructed comprising a front layer, a back layer, and other layers connecting the front and back layers. In 604, a transition layer is selected from several other layers based on determining the transition from the activation layer to the weight layer. In 606, a first convolutional kernel is assigned to the layer returning from the front layer to the transition layer. In 608, a second convolutional kernel is assigned to the layer returning from the transition layer to the back layer. The first convolutional kernel compresses its input by reducing a first fixed percentage of values ​​less than the maximum value in the kernel window to zero. The second convolutional kernel compresses its input by reducing a second fixed percentage of values ​​less than the maximum value in the kernel window to zero. The second fixed percentage is less than the first fixed percentage.

[0033] In the discussion up to this point, it will be understood that, generally speaking, according to one aspect of the invention, the exemplary method 500 includes establishing a convolutional neural network 100 at 502. The convolutional neural network includes a front layer 106 having a first number of neurons and a first convolutional kernel; a rear layer 110 having a second number of neurons and a second convolutional kernel; and a plurality of other layers connecting the front and rear layers. The second number of neurons is less than the first number of neurons. The method further includes, at 504, selecting a transition layer 112 having a third number of neurons and a third convolutional kernel from the plurality of other layers; at 506, assigning a first activation precision to the activations of neurons returning from the front layer to the transition layer; at 508, assigning a second activation precision to the activations of neurons returning from the transition layer to the rear layer; at 510, assigning a first weight precision to the weights of the inputs of neurons returning from the front layer to the transition layer; and at 512, assigning a second weight precision to the weights of the inputs of neurons returning from the transition layer to the rear layer. The first activation precision differs from the second activation precision. The first weight precision differs from the second weight precision. The transition layer is selected to correspond to a transition from an activation layer to a weight layer.

[0034] In one or more embodiments, at least one of the second and third convolution kernels is different from the first convolution kernel.

[0035] In one or more embodiments, the first activation precision is less than the second activation precision. In one or more embodiments, the first weight precision is greater than the second weight precision. In one or more embodiments, the first activation precision is greater than the second weight precision.

[0036] In one or more embodiments, the transition is determined by a performance estimator. For example, the performance estimator iteratively changes the description of the convolutional neural network to approximate the optimal value of a selected hardware system metric. As another example, the performance estimator scans multiple possible precision configurations, and for each precision configuration, the performance estimator checks data reuse and memory footprint to estimate the latency for a given system constraint. In one or more embodiments, each precision configuration defines the allocation of activation precision and weight precision for each layer of the convolutional neural network.

[0037] In another aspect, exemplary method 600 includes establishing a convolutional neural network 100 at 602. The convolutional neural network includes a front layer 106 having a first number of neurons and a first convolutional kernel; a rear layer 110 having a second number of neurons and a second convolutional kernel, wherein the second number of neurons is less than the first number of neurons; and a plurality of other layers connecting the front and rear layers. The method includes selecting a transition layer 112 having a third number of neurons and a third convolutional kernel from the plurality of other layers at 604; at 606, assigning a first convolutional kernel to neurons returning from the front layer to the transition layer; and at 608, assigning a second convolutional kernel to neurons returning from the transition layer to the rear layer. The first convolutional kernel has a first compression by reducing a first fixed percentage of values ​​less than the maximum value in the kernel window to zero. The second convolutional kernel has a second compression by reducing a second fixed percentage of values ​​less than the maximum value in the kernel window to zero. The second fixed percentage is less than the first fixed percentage. The transition layer is selected to correspond to a transition from an activation-heavy layer to a weight-heavy layer.

[0038] In one or more embodiments, the third convolution kernel may be the same as the first convolution kernel or may be the same as the second convolution kernel.

[0039] In another aspect, exemplary method 500 includes establishing a convolutional neural network 100 at 502. The convolutional neural network includes a front layer 106 having a first number of neurons and a first convolutional kernel; a rear layer 110 having a second number of neurons and a second convolutional kernel, wherein the second number of neurons is less than the first number of neurons; and a plurality of other layers connecting the front and rear layers. The method includes selecting a transition layer 112 having a third number of neurons and a third convolutional kernel from the plurality of other layers at 504; at 506, assigning a first activation precision to the activations of neurons returning from the front layer to the transition layer, and assigning a first weight precision to the weights of the inputs of neurons returning from the front layer to the transition layer; and at 508, assigning a second activation precision to the activations of neurons returning from the transition layer to the rear layer, and assigning a second weight precision to the weights of the inputs of neurons returning from the transition layer to the rear layer. The first activation precision is less than the first weight precision. The second activation precision is greater than the second weight precision. The transition layer is selected to correspond to a transition from an activation-heavy layer to a weight-heavy layer.

[0040] One or more embodiments of the present invention or elements thereof may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and operable to perform exemplary method steps, or in the form of a non-transitory computer-readable medium containing computer-executable instructions that, when executed by a computer, cause the computer to perform the exemplary method steps. Figure 7 A computer system that can be used to implement one or more aspects and / or elements of the present invention is described.

[0041] In computer system 10, there is a computer server 12 that can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the aforementioned systems or devices.

[0042] Computer server 12 can be described in the general context of executable instructions in a computer system, such as program modules executed by the computer system. Typically, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer server 12 can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0043] like Figure 7 As shown, computer server 12 in computer system 10 is illustrated as a general-purpose computing device. Components of computer server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that couples various system components, including system memory 28, to processor 16.

[0044] Bus 18 represents one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. By way of example and not limitation, these architectures include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0045] Computer server 12 typically includes various computer system readable media. Such media can be any available media accessible by computer server 12, and it includes volatile and non-volatile media, removable and non-removable media.

[0046] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, and generally referred to as "hard disk drives"). Although not shown, disk drives for reading from and writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks such as CD-ROMs, DVD-ROMs, or other optical media may be provided. In this case, each may be connected to bus 18 via one or more data media interfaces. As will be further described below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0047] A program / utility 40 having a set (at least one) of program modules 42, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 28 as an example and not as a limitation. Each of the operating system, one or more applications, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program modules 42 typically perform the functions and / or methods of the embodiments of the invention described herein.

[0048] Computer server 12 can also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc.; one or more devices that enable a user to interact with computer server 12; and / or any device that enables computer server 12 to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can occur via input / output (I / O) interface 22; however, computer server 12 can communicate with one or more networks via network adapter 20, such as local area networks (LANs), general area networks (WANs), and / or public networks (e.g., the Internet). As depicted, network adapter 20 communicates with other components of computer server 12 via bus 18. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with computer server 12. Examples include, but are not limited to: microcode, device drivers, redundant processing units and external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0049] Therefore, one or more embodiments can utilize software running on a general-purpose computer or workstation. (Reference) Figure 7 Such an implementation may employ, for example, a processor 16, a memory 28, and an input / output interface 22 to a display 24 and external devices 14 (such as a keyboard, pointing device, etc.). The term "processor" as used herein is intended to include any processing device, such as a processing device including a CPU (Central Processing Unit) and / or other forms of processing circuitry. Furthermore, the term "processor" may refer to more than one individual processor. The term "memory" is intended to include memory associated with a processor or CPU, such as RAM (Random Access Memory) 30, ROM (Read-Only Memory), fixed memory devices (e.g., hard disk drive 34), removable memory devices (e.g., disk), flash memory, etc. Furthermore, the phrase "input / output interface" as used herein is intended to refer to an interface, for example, for inputting data into the processing unit (e.g., a mouse), and one or more mechanisms (e.g., a printer) for providing results associated with the processing unit. The processor 16, memory 28, and input / output interface 22 may be interconnected, for example, via a bus 18 that is part of a data processing unit 12. Appropriate interconnections, such as via bus 18, can also be provided to network interface 20, such as a network card, which can be provided to interface with a computer network, and to media interface, such as a disk or CD-ROM drive, which can be provided to interface with appropriate media.

[0050] Therefore, computer software including instructions or code for performing the methods of the present invention as described herein can be stored in one or more associated memory devices (e.g., ROM, fixed or removable memory) and, when ready for use, partially or entirely loaded (e.g., loaded into RAM) and implemented by the CPU. Such software may include, but is not limited to, firmware, resident software, microcode, etc.

[0051] A data processing system suitable for storing and / or executing program code will include at least one processor 16 directly or indirectly coupled to memory element 28 via system bus 18. Memory element 28 may include local memory used during the actual implementation of the program code, mass storage, and cache memory 32 that provides temporary storage for at least some of the program code to reduce the number of times code must be retrieved from mass storage during implementation.

[0052] Input / output or I / O devices (including but not limited to keyboards, displays, indicators, etc.) can be coupled to the system directly or through an intermediate I / O controller.

[0053] Network adapter 20 can also be coupled to the system to enable the data processing system to couple to other data processing systems or remote printers or storage devices via an intermediate private or public network. Modems, cable modems, and Ethernet cards are just a few types of network adapters currently available.

[0054] As used herein (including the claims), "server" includes a physical data processing system that runs server programs (e.g., such as...). Figure 7 (System 12 shown). It is understood that such a physical server may or may not include a monitor and keyboard.

[0055] One or more embodiments may be implemented, at least in part, in the context of a cloud or virtual machine environment, although this is exemplary and not limiting. Return to References Figures 1-2 And the attached text.

[0056] It should be noted that any method described herein may include additional steps of providing a system comprising different software modules implemented on a computer-readable storage medium; modules may include any or all of the appropriate elements depicted in the block diagram and / or described herein; any one, some, or all of the modules / blocks and / or submodules / subblocks are described by way of example and not limitation. The method steps may then be performed using the different software modules and / or submodules of a system executed on one or more hardware processors (e.g., 16) as described above. Furthermore, the computer program product may include a computer-readable storage medium having code adapted to be implemented to perform one or more method steps described herein, including providing different software modules to the system.

[0057] Exemplary System and Article Details

[0058] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0059] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0060] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0061] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing the status information of the computer-readable program instructions.

[0062] Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0063] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0064] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0065] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a non-consecutive order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0066] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0067] In a preferred embodiment of the invention described herein, a method is provided comprising: establishing a convolutional neural network comprising: a front layer having a first number of neurons and a first convolutional kernel; a back layer having a second number of neurons and a second convolutional kernel, wherein the second number of neurons is less than the first number of neurons; and a plurality of other layers connected between the front layer and the back layer; selecting a transition layer from the plurality of other layers having a third number of neurons and a third convolutional kernel; assigning a first convolutional kernel to neurons returning from the front layer to the transition layer, wherein the first convolutional kernel has a first compression by reducing a first fixed proportion of values ​​less than the maximum value in the kernel window to zero; and assigning a second convolutional kernel to neurons returning from the transition layer to the back layer, wherein the second convolutional kernel has a second compression by reducing a second fixed proportion of values ​​less than the maximum value in the kernel window to zero, wherein the second fixed proportion is less than the first fixed proportion, wherein the transition layer is selected to correspond to a transition from an activation layer to a weight layer.

Claims

1. A method comprising: establishing a convolutional neural network, the convolutional neural network comprising: a front layer having a first number of neurons and a first convolutional kernel; a back layer having a second number of neurons and a second convolutional kernel; and a plurality of other layers connected between the front layer and the back layer; selecting a transition layer from the plurality of other layers having a third number of neurons and a third convolutional kernel; assigning a first activation precision to activations of neurons from the front layer back to the transition layer and a second activation precision to activations of neurons from the transition layer back to the back layer, wherein the first activation precision is different from the second activation precision; and assigning a first weight precision to weights of inputs of neurons from the front layer back to the transition layer and a second weight precision to weights of inputs of neurons from the transition layer back to the back layer, wherein the first weight precision is different from the second weight precision, wherein the transition layer is selected to correspond to a transition from activation heavy layers to weight heavy layers; wherein the transition is determined by a performance estimator that iteratively changes a description of the convolutional neural network to approximate an optimal value of a selected hardware system metric and that scans a plurality of possible precision configurations and, for each precision configuration, checks data reuse and memory footprint to estimate a latency for a given system constraint.

2. The method of claim 1, wherein, at least one of the second convolutional kernel and the third convolutional kernel is different from the first convolutional kernel.

3. The method of claim 1, wherein, the first activation precision is less than the second activation precision.

4. The method of claim 1, wherein, the first weight precision is greater than the second weight precision.

5. The method of claim 1, wherein, the first activation precision is greater than the second weight precision.

6. The method of claim 1, wherein, the performance estimator estimates a performance metric according to potentially different transition layers.

7. The method of claim 1, wherein, the second number of neurons is less than the first number of neurons.

8. The method of claim 1, wherein, each precision configuration defines an assignment of activation precision and weight precision to each layer of the convolutional neural network.

9. The method of claim 1, comprising: assigning a first convolutional kernel to neurons from the front layer back to the transition layer, wherein the first convolutional kernel has a first compression by reducing to zero a first fixed proportion of values in a kernel window that are less than a maximum value; and assigning a second convolutional kernel to neurons from the transition layer back to the back layer, wherein the second convolutional kernel has a second compression by reducing to zero a second fixed proportion of values in the kernel window that are less than the maximum value, wherein the second fixed proportion is less than the first fixed proportion.

10. The method of claim 1, wherein, the possible precision configurations define an assignment of the first convolutional kernel and the second convolutional kernel to different layers of the convolutional neural network.

Images