Neural processing element with improved accuracy

Abstract

A neural processing element is configured with a hardware AND gate configured to perform a logical AND operation between a sign extension signal and a most significant bit ("MSB") of an operand. A state of the sign extension signal can be based on a type of layer of a deep neural network ("DNN") that generated the operand. If the sign extension signal is a logical false, no signal extension is performed. If the sign extension signal is a logical true, a coupler couples an output of the hardware AND gate and the operand, thereby extending the operand from an N-bit unsigned binary value to an N+1-bit signed binary value. The neural processing element can also include another hardware AND gate and another coupler for similarly processing another operand. An output of the couplers for both operands is provided to a hardware binary multiplier.

Description

BACKGROUND

[0001] Deep neural networks ("DNNs") are loosely modeled after information processing and communication patterns in biological neural systems, such as the human brain. DNNs can be used to solve complex classification problems, such as but not limited to object detection, semantic labeling, and feature extraction. As a result, DNNs form the basis of many artificial intelligence ("AI") applications, such as computer vision, speech recognition, and machine translation. In many of these areas, DNNs can match or exceed human accuracy.

[0002] The high-level performance of DNNs stems from their ability to extract high-level features from input data after using statistical learning over large datasets to acquire an efficient representation of the input space. However, the superior performance of DNNs comes at the cost of high computational complexity. High-performance general-purpose processors, such as graphics processing units ("GPUs"), are often used to provide the high-level of computational performance needed for many DNN applications.

[0003] While general-purpose processors, like GPUs, can provide high levels of computational performance for implementing DNNs, these types of processors are often not suitable for performing long-running DNN operations in computing devices where low power consumption is critical. For example, general-purpose processors, such as GPUs, can not be suitable for performing long-running DNN tasks in battery-powered portable devices, like smartphones or augmented / virtual reality ("AR / VR") devices, where reducing power consumption is needed to extend battery life.

[0004] Reducing power consumption can also be important for non-battery-powered devices, such as power-over-Ethernet ("POE") security cameras, for example, when performing sustained DNN tasks, such as detecting human motion. In this particular example, POE switches can only provide limited power; reducing the power consumption of POE devices, like security cameras, allows the use of POE switches that provide less power.

[0005] Application-specific integrated circuits ("ASICs") have been developed that can provide high-performance DNN processing while reducing power consumption compared to general-purpose processors. However, despite the progress made in this area, there remains a need for improved performance and reduced power consumption for ASICs that perform DNN processing, particularly for computing devices where low power consumption is critical.

[0006] The present disclosure presented herein is presented in response to these and other technical challenges. SUMMARY

[0007] A neural processing element capable of performing computations with improved precision is disclosed herein. The architecture of the neural processing element disclosed herein achieves higher precision when processing operands generated by certain types of DNN layers. For example, but not limited to, a neural processing element implementing the disclosed technology can take as input a signed or unsigned operand having N bits. In certain cases, the neural processing element can perform operations on the operand using N+1 bits, thereby providing improved precision compared to architectures in which the neural processing element would perform its internal computations using only N bits. Other technical advantages not specifically mentioned herein can also be achieved by implementations of the disclosed subject matter.

[0008] To achieve the technical advantages briefly mentioned above, a DNN processor including a neural processing element is disclosed. The neural processing element can take as input a signed or unsigned operand having N bits. In one particular implementation, the operand is a signed or unsigned 8-bit binary number. In other implementations, the operand can utilize a different number of bits.

[0009] The neural processing element also includes circuitry for performing computations with improved precision on certain types of operands. For example, the neural processing element can perform extended precision computations on operands that are limited to positive numbers, such as operands generated by a rectified linear unit (“ReLU”) layer of a DNN. Typically, the signed N-bit values output by a ReLU layer use only N-1 bits (representing values from -127 to +128), as one bit is reserved for the sign. However, using the disclosed technology, the neural processing element can utilize all N bits (representing values from 0 to 255) of the operands generated by the ReLU layer in its internal computations.

[0010] To provide improved precision, the neural processing element is configured with a hardware AND gate. The hardware AND gate is configured to perform a logical AND operation between a binary signal (which can be referred to herein as a “sign extension signal”) and the most significant bit (“MSB”) of the operand. In the example where the operand is an 8-bit binary number, the MSB is the 7th bit of the operand.

[0011] The state of the sign extension signal can be based on the type of layer of the DNN previously processed by the DNN processor. Specifically, if the operand is unsigned (such as when the operand is generated by a ReLU layer of a DNN), the sign extension signal can be set to logical FALSE. In this case, the sign of the operand is not extended, but rather a zero bit is prepended to the operand, making it a 9-bit signed number.

[0012] If the sign extension signal is TRUE (i.e., when the input is signed, such as the output of a Tanh layer), the sign extension is performed on the operand. To perform the sign extension operation, the neural processing element also includes a concatenation circuit (“concatenator”) configured to concatenate two binary values. Specifically, the concatenator concatenates the output of the hardware AND gate and the operand, thereby extending the operand from an N-bit unsigned value to an N+1-bit signed value. In a configuration where the operand is an 8-bit binary number, for example, the output of the concatenator is a 9-bit signed binary number.

[0013] The neural processing element can also include another hardware AND gate and another concatenator for processing another operand in the manner described above. In this example, the output of the concatenator for both operands is provided to a hardware binary multiplier. In a configuration where the operands are 8-bit binary numbers, the output of the hardware binary multiplier is an 18-bit signed binary number. The neural processing element can continue processing the output of the hardware binary multiplier.

[0014] It should be appreciated that the above-described subject matter can be implemented as a computer-controlled apparatus, a computer-implemented method, a computing device, or as an article of manufacture such as a computer program product. These and various other features will be apparent from a reading of the following DETAILED DESCRIPTION and a review of the associated drawings.

[0015] This Summary is provided to introduce a brief description of some aspects of the disclosed technology, which is further described below in the DETAILED DESCRIPTION. This Summary is neither intended to identify key or essential features of the claimed subject matter nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the problems mentioned in any part of this disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a computing architecture diagram illustrating aspects of a configuration and operation of a processing system including a DNN processor implementing aspects of the technology disclosed herein, according to one embodiment;

[0017] Figure 2 is a computing architecture diagram illustrating aspects of a configuration of a neural processing element with improved precision, according to one embodiment;

[0018] Figure 3A is a data diagram illustrating an illustrative example of an operation of a neural processing element configured using the disclosed technology;

[0019] Figure 3Bis a dataflow graph illustrating another illustrative example of the operation of a neural processing element configured using the disclosed technology;

[0020] Figure 4 is a flow diagram illustrating a routine that explains aspects of the operation of the neural processing element shown in Figure 2

[0021] Figure 5 is a computer architecture diagram illustrating an illustrative computer hardware and software architecture of a computing device that can act as an application host for a DNN processor that includes a neural processing element with improved precision, according to one embodiment; and

[0022] Figure 6 is a network diagram illustrating a distributed computing environment in which aspects of the disclosed technology can be implemented, according to various embodiments presented herein. DETAILED DESCRIPTION

[0023] The following DETAILED DESCRIPTION relates to a neural processing element with improved precision. As discussed briefly above, implementations of the disclosed technology can enable a neural processing element to perform its internal calculations with higher precision when processing operands generated by certain types of DNN layers. Other technical advantages not specifically mentioned herein can also be realized through implementations of the disclosed subject matter.

[0024] While the subject matter described herein is presented in the general context of a hardware neural processing element implemented in conjunction with a hardware DNN processor, those skilled in the art will recognize that other implementations can be

[0025] In the following DETAILED DESCRIPTION, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration various embodiments for implementing the subject matter described herein. Referring now to the drawings, in which like numerals represent like elements throughout the several figures, aspects of a neural processing element with improved precision will be described.

[0026] Figure 1 ​This is a computational architecture diagram illustrating aspects of the configuration and operation of a processing system according to one embodiment, which implements the techniques disclosed herein. In some embodiments, the processing system disclosed herein is configured to solve classification problems (and related problems), such as, but not limited to, eye tracking, manual tracking, object detection, semantic tagging, and feature extraction.

[0027] To provide this functionality, a DNN processor 112 is provided, which can implement recall-only neural networks and programmatically support various network architectures. Training of the network implemented by the DNN processor 112 can be performed offline in a server cluster, data center, or another suitable computing environment. The result of training the DNN is a set of parameters, which may be referred to as "weights" or "kernels." These parameters represent transformation parameters that can be applied to the input, resulting in a categorically or semantically labeled output.

[0028] The DNN processor 112 disclosed herein can be considered a superscalar processor. The DNN processor 112 can dispatch one or more instructions to multiple execution units (referred to as neural processing elements 105F). The execution units can be "simultaneously dispatched and completed," where each execution unit is synchronized with each of the other execution units. The DNN processor 112 can be classified as a single-instruction, multiple-data ("SIMD") architecture.

[0029] The neural processing element 105F is a fundamental unit in neural networks, used for coarse modeling of biological neural processing elements in the brain. Specifically, the neural processing element 105F can compute the inner product of the input vector and the weight vector added to the bias using the applied activation function.

[0030] Each neural processing element 105F in the DNN processor 112 is capable of performing weighted sums, max pooling, bypassing, and potentially other types of operations. Neural processing element 105F processes the input and weight data for each clock cycle. Each neural processing element 105F is synchronized with all other neural processing elements 105F in terms of in-kernel processing to minimize kernel data flow within the DNN processor 112.

[0031] Each neural processing element 105F can include a multiplier, an adder, a comparator, and multiple accumulators. Figure 1 (Not shown in the image). By having multiple accumulators, the neural processing element 105F is able to maintain the context of multiple different active kernels simultaneously. Each accumulator can be loaded from a read of SRAM 150 (described below). An accumulator can add itself to the contents of other accumulators from other neural processing elements 105F.

[0032] The DNN processor 112 accepts planar data as input, such as image data. However, the input to the DNN processor 112 is not limited to image data. Rather, the DNN processor 112 can operate on any input data presented to the DNN processor 112 in a uniform planar format. In one particular embodiment, the DNN processor 112 can accept multi-planar one or two byte data frames as input.

[0033] Each input frame can be convolved with a set of NxKxHxW kernels, where N is the number of kernels, K is the number of channels per kernel, H is the height and W is the width. The convolution is performed across overlapping intervals of the input data, where the interval is defined by the step size in the X and Y directions. These functions are performed by the neural processing element 105F and managed by the DNN processor 112 via software-visible control registers. Other types of operations can also be performed, including but not limited to fully connected, batch normalization, and other types of operations.

[0034] The DNN processor 112 supports multiple data types: weights, input data / feature maps, activation data, biases, and scalars. In most cases, input data / feature maps and activation data are two names for the same data, the distinction being that the term “activation data” is used when referring to the output of a layer. The term “input data / feature map” is used when referring to the input of a layer.

[0035] The neural processing element 105F in the DNN processor 112 computes a weighted sum of its inputs and passes the weighted sum through an “activation function” and a “transfer function”. The transfer function typically has a sigmoid shape, but can also take the form of a piecewise linear function, a step function, or other types of functions. The activation function allows the neural processing element 105F to train to a larger set of input and desired outputs, where the classification boundary is non-linear.

[0036] The DNN processor 112 operates on a list of layer descriptors corresponding to the layers of a neural network. The list of layer descriptors can be handled as instructions by the DNN processor 112. These descriptors can be pre-fetched from memory into the DNN processor 112 and executed in order. The descriptor list acts as a set of instructions to the DNN processor 112. In some configurations, two types of instructions are utilized: layer descriptors and program instructions that get executed on a sequence controller. Software tools and / or compilers can be executed on a device external to the DNN processor 112 to create the descriptor list that is executed on the DNN processor 112.

[0037] Typically, there are two main categories of descriptors: memory-to-memory (“M2M”) descriptors and operation descriptors. M2M descriptors can be used to move data from main memory to a local buffer (i.e., buffer 125 hereinafter) / from the local buffer to main memory for use by the operation descriptor. M2M descriptors follow a different execution pipeline than operation descriptors. The target pipeline for M2M descriptors can be the internal DMA engine 105B or the configuration register 105G, while the target pipeline for operation descriptors can be the neural processing element 105F.

[0038] An operation descriptor specifies the particular operation that the neural processing element 105F should perform on a data structure located in local static random access memory (“SRAM”). Operation descriptors are processed sequentially and are capable of performing many different layer operations, at least some of which are described herein.

[0039] like Figure 1 As shown, the DNN processor 112 has a memory subsystem with unique L1 and L2 buffer structures. Figure 1 The L1 and L2 buffers shown are specifically designed for neural network processing. For example, L2 buffer 150 can maintain a selected memory capacity using a high-speed dedicated interface operating at a selected frequency. L1 buffer 125 can maintain a selected memory capacity that can be partitioned between the kernel and active data. L1 buffer 125 may be referred to herein as "buffer 125", and L2 buffer 150 may be referred to herein as SRAM 150.

[0040] In some embodiments, computational data (i.e., input data, weights, and activation data) is stored in SRAM 150 in row-major order. The computational data may be organized into two buffers, one containing the input data, which may be referred to herein as the "input buffer," and the other containing the kernel weights, which may be referred to herein as the "weight buffer." The buffers are filled from SRAM 150 by load / memory unit 105C. Data is accumulated in each buffer until it reaches its predetermined capacity. The buffer data is then copied to a shadow buffer in some embodiments and presented to neural processing element 105F.

[0041] The DNN processor 112 may also include a number of other components, including but not limited to a register interface 105G, a prefetch unit 105A, a save / restore unit 105E, a layer controller 105D, and a register interface 105G. In some embodiments, the DNN processor 112 may include additional or alternative components.

[0042] In some configurations, the DNN processor 112 operates in conjunction with other external computing components. For example, in some embodiments, the DNN processor 112 is connected to a host application processor system on a chip (“host SoC”) 130. For example, the DNN processor 112 can be connected to the host SoC 130 through a PCIe interface. Appropriate PCIe components, such as a PCIe endpoint 135, can be used to implement these connections.

[0043] The host SoC 130 acts as an application processor for the DNN processor 112. The main operating system, applications, and secondary sensor processing are performed by the host SoC 130. The host SoC 130 can also be connected to an input data source 102, such as an external camera, that provides input data, such as image data, to the DNN processor 112. An additional central processing unit (“CPU” or “processor”), such as a TENSILICA node, 140, can also be used to coordinate the operation of the DNN processor 112, aspects of which will be described below.

[0044] The DDR DRAM 155 can also be connected to the host SoC 130, which can be used as the main system memory. This memory is accessible from the host SoC 130 through a memory controller 145 across the high bandwidth fabric 120, such as a PCIe bus. The high bandwidth fabric 120 provides bidirectional direct memory access (“DMA”) small message passing transactions and larger DMA transactions. The bridge 115 and low bandwidth fabric 110 can connect the DNN processor 112 to the host SoC 130 for sub-module configuration and other functions.

[0045] The DNN processor 112 can include a DMA engine 105B configured to move data to and from the main memory 155. In some embodiments, the DMA engine 105B has two channels. One channel is dedicated to fetch operation descriptors, while the other channel is dedicated to M2M operations. DMA descriptors can be embedded in M2M descriptors. Descriptors in this context are DMA descriptors for moving the contents of memory, and should not be confused with the operation descriptors described above.

[0046] To offload the local SRAM memory 150, and to provide more space for input data and weight data, the activation output can be optionally streamed directly to the DDR memory 155. When streaming to the DDR memory 155, the DNN processor 112 will accumulate enough data on the high bandwidth fabric 120 for a burst transaction, and will buffer enough transactions to minimize back pressure on the neural processing elements 105F. Additional details regarding the operation of the DNN processor 112 will be provided below.

[0047] Additional details regarding the operation of the DNN processor 112 will be provided below. In particular, details regarding the configuration and operation of the neural processing element 105F (a neural processing element with improved precision) will be provided below with respect to Figure 2 and FIG. 3. The configuration and operation of the neural processing element 105F (a neural processing element with improved precision) will be provided below with respect to

[0048] Figure 2 is a computational architecture diagram illustrating aspects of the configuration and operation of the neural processing element 105F configured for improved precision computations. As briefly discussed above, the architecture of the neural processing element 105F disclosed herein enables higher precision computations when processing operands generated by certain types of DNN layers.

[0049] As briefly described above, the neural processing element 105F takes operands 202A and 202B (which can be referred to as "operands 202") as input. The operands 202 can be N-bit signed or unsigned binary numbers, where N is a power of two. In one particular implementation, for example, the operands 202 are 8-bit signed or unsigned binary numbers. In other implementations, the operands 202 can be expressed using a different number of bits.

[0050] As also briefly described above, in certain cases, such as when the operands 202 are generated by a ReLU layer of a DNN, the neural processing element 105F can expand the operands 202 from N bits to N+1 bits. Typically, the signed N-bit values output by a ReLU layer use only N-1 bits (representing values of -127 to +128 when using 8 bits) because one bit is reserved for the sign. However, using the disclosed techniques, the neural processing element 105F can utilize all N bits (representing values of 0 to 255 when using 8 bits) of the operands 202 generated by a ReLU layer in its internal computations.

[0051] Once the operands have been expanded from N bits to N+1 bits in the manner described above, the neural processing element 105F can operate on the N+1 bits of the expanded operands, thus providing improved precision compared to architectures in which the neural processing element 105F performs its internal computations using only N bits. Other technical advantages not specifically mentioned herein can also be realized by implementations of the disclosed subject matter.

[0052] To provide the functionality disclosed herein, among other things, the neural processing element 105F of the DNN processor 112 includes hardware AND gates 206A and 206B. The hardware AND gate 206A is configured to perform a logical AND operation between a binary signal 208A (which can be referred to herein as a "sign extension signal 208A") and the MSB of the operand 202A. The hardware AND gate 206B is configured to perform a logical AND operation between a binary signal 208B (which can be referred to herein as a "sign extension signal 208B") and the MSB of the operand 202B. In some configurations, the sign extension signals 208 are 1-bit binary numbers.

[0053] In the example shown in FIG. 2, the operands 202A and 202B are 8-bit binary numbers. Thus, the hardware AND gate 206A is configured to perform a logical AND operation between the sign extension signal 208A and the 7th bit of the operand 202A. Similarly, the hardware AND gate 206B is configured to perform a logical AND operation between the sign extension signal 208B and the 7th bit of the operand 202B. Figure 2

[0054] As briefly discussed above, the state of the sign extension signal 208 can be set based on the type of layer of the DNN that was previously processed by the DNN processor 112. Specifically, if the operand 202A is unsigned, such as when it was produced by a ReLU layer of the DNN, then the sign extension signal 208A can be set to a logical false. Similarly, if the operand 202B is unsigned, such as when it was produced by a ReLU layer of the DNN, then the sign extension signal 208B can be set to a logical false. The neural processing element 105F can also set the sign extension signal 208 to a logical false when the operand 202 can only take on positive values, such as when it was generated by a layer of the DNN that produces only positive values.

[0055] In general, the sign extension signal 208 can be set based on the type of activation function (i.e., nonlinearity) that was applied by the DNN directly in the layer that was previously processed. When processing the current layer, information is provided that identifies the nonlinearity that was applied in the previous layer of the DNN. In some configurations, a user can specify particular activation functions for which the sign extension signal 208 will be a logical true or a logical false.

[0056] When the sign extension signal 208 is true, the neural processing element 105F performs a sign extension operation. This can occur when the previous layer of the DNN was a Tanh ("tanh") layer that produces signed outputs.

[0057] ​To perform the sign extension operation, the neural processing element also includes concatenation circuits 204A and 204B (which can be referred to herein as “concatenators 204A or 204B”). The concatenators 204A or 204B are circuits configured to concatenate two binary values.

[0058] The concatenator 204A concatenates the 1-bit output of the hardware AND gate 206A and the operand 202A, thereby extending the operand from an N-bit value to an N+1-bit value. Similarly, the concatenator 204B concatenates the 1-bit output of the hardware AND gate 206B and the operand 202B, thereby extending the operand from an N-bit value to an N+1-bit value. In a configuration in which the operand 202 is an 8-bit unsigned binary number, as shown in Figure 2 for example, the output of the concatenator 204 is a 9-bit signed binary number.

[0059] The signed output 210A and the signed output 210B of the concatenators 204A and 204B, respectively, are passed to a hardware binary multiplier 212. The hardware binary multiplier 212 is a circuit configured to multiply two binary numbers, in this case the outputs 210A and 210B. In the example shown in Figure 2 in which the outputs 210 are 9-bit signed binary numbers, the output 214 of the hardware binary multiplier 212 is an 18-bit signed binary number. The output 214 of the multiplier 212 can be provided to other components in the neural processing element 105F for additional processing.

[0060] Figure 3A is a data diagram illustrating an illustrative example of the operation of a neural processing element 105F configured using the disclosed technology. In this example, the operand 202A is an 8-bit unsigned binary number, such as a binary number output by a ReLU layer of a DNN. In this example, the sign extension signal 208A is set to a logical false (i.e., zero (0) in the example shown) because the operand is unsigned. Thus, regardless of the value of the 7th bit of the operand 202A, the output 210A of the AND gate 206A is zero.

[0061] As described above, the output of the AND gate 206A (i.e., zero) is concatenated with (i.e., prepended to) the operand 202A. In the example shown, for example, the output 210A of the AND gate 206A is concatenated with the operand 202A, thereby converting the 8-bit operand 202A to a 9-bit signed binary number. In this way, the bits 0 through 7 of the operand 204A can be used to represent binary numbers (e.g., positive numbers having values from 0 to 255). The 9-bit signed output 210A is provided to the multiplier 212.

[0062] Figure 3Bis a data graph illustrating another illustrative example of operation of a neural processing element 105F configured using the disclosed technology. Figure 3B The operand 202A shown in Figure 3A is the same as the operand shown in the example presented in

[0063] As described above, the output of the AND gate 206A is coupled with (i.e., added in front of) the operand 202A. In the illustrated example, for instance, an 8th bit is added to the operand 202A. If the value of the 7th bit of the operand 202A is one, the value of the 8th bit is set to one (in Figure 3B is shown as output 210A'). If the value of the 7th bit of the operand 202A is zero, the value of the 8th bit is set to zero (in Figure 3B is shown as output 210A"). In this way, the value of the sign bit can be preserved for the signed operand 202, which can be positive or negative. The 9-bit signed output 210A is provided to the multiplier 212 for use in the manner described above. It should be appreciated that while Figure 3A and Figure 3B are illustrated with respect to 8-bit operands, the same operations can be applied to operands having other numbers of bits.

[0064] Figure 4 is a flowchart illustrating a routine 400 that illustrates aspects of operation of the neural processing element 105F described above with respect to Figures 1 to 3B configured for increased precision. It should be appreciated that the logical operations described herein with respect to Figure 4 and other figures can be implemented (1) as a sequence of computer implemented acts or program modules running on a computing device and / or (2) as interconnected machine logic circuits or circuit modules within a computing device.

[0065] Particular implementations of the technology disclosed herein are matters of choice, dependent on the performance and other requirements of the computing device. Accordingly, the logical operations described herein are referred to variously as states, operations, structural devices, acts, or modules. These states, operations, structural devices, acts and modules can be implemented in hardware, software, special-purpose digital logic, and any combination thereof. It should be appreciated that more or fewer operations can be performed than shown in the figures and described herein. These operations can also be performed in a different order than those described herein.

[0066] The routine 400 begins at operation 402 where a signed or unsigned binary operand 202 is received. As described above, the operand 202 is 8 bits in one particular implementation, but another number of bits can be used in other configurations.

[0067] From operation 402, the routine 400 continues to operation 404 where the AND gate 206 performs a logical AND operation on the MSB of the operand 202 and the sign extension signal 208. As described above, in one embodiment, the sign extension signal 208 is a single bit signal. The sign extension signal 208 indicates whether a sign extension operation is performed on the operand 202 received at operation 402.

[0068] From operation 404, the routine 400 continues to operation 406 where a new MSB is added to the N-bit operand 202 to create an N+1 bit signed binary value 210. The routine 400 then continues to operation 408 where the result of the logical AND operation performed at operation 404 is stored in the new MSB of the N+1 bit signed value 210. As described above, in some embodiments, these operations are performed by concatenation.

[0069] From operation 408, the routine 400 continues to operation 410 where the concatenator 204 passes the 9 bit signed value to the multiplier 212. The routine 400 then continues to return to operation 402 where additional operands 202 can be processed in the manner described above.

[0070] Figure 5 is a computer architecture diagram illustrating an illustrative computer hardware and software architecture of a computing device that can serve as an application host for the DNN processor 112 presented herein. Specifically, Figure 5 The architecture illustrated in FIG. 1 can be used to implement a server computer, a mobile phone, an e-reader, a smartphone, a desktop computer, an AR / VR device, a tablet computer, a laptop computer, or another type of computing device suitable for use with the DNN processor 112.

[0071] Figure 5The computer 500 illustrated in FIG. 5 includes a central processing unit 502 ("CPU"), a system memory 504, including a random access memory 506 ("RAM") and a read-only memory ("ROM") 508, and a system bus 510 that couples the memory 504 to the CPU 502. A basic input / output system ("BIOS" or "firmware") containing the basic routines that help to transfer information between elements within the computer 500, such as during startup, can be stored in the ROM 508. The computer 500 further includes a mass storage device 512 for storing an operating system 522, application programs, and other types of programs. The mass storage device 512 can also be configured to store other types of programs and data.

[0072] The mass storage device 512 is connected to the CPU 502 through a mass storage controller (not shown), which is connected to the bus 510. The mass storage device 512, and its associated computer-readable media, provide nonvolatile storage for the computer 500. Although the description of computer-readable media contained herein refers to a mass storage device such as a hard disk, CD-ROM drive, DVD-ROM drive, or USB storage key, those skilled in the art will appreciate that computer-readable media can be any available computer storage media or communication media that can be accessed by the computer 500.

[0073] Communication media includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics changed or set by a method or process that is used to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, video, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer-readable media.

[0074] For example, but not by way of limitation, computer storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, digital versatile disks (DVD), HD-DVD, BLU-RAY, or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer 500. For purposes of the claims, the phrase "computer storage medium," and variations thereof, does not include waves or signals per se or communication media.

[0075] According to various configurations, the computer 500 can operate in a networked environment using logical connections to remote computers through a network such as the network 520. The computer 500 can connect to the network 516 through a network interface unit 520 connected to the bus 516. It should be appreciated that the network interface unit 916 can also be utilized to connect to other types of networks and remote computer systems. The computer 500 can also include an input / output controller 518 for receiving and processing input from a number of other devices, including a keyboard, mouse, touch input, electronic stylus Figure 5 (not shown in FIG. 8) or physical sensors such as cameras). Similarly, the input / output controller 518 can provide output to a display screen or other type of output device (also not shown in FIG. 8). Figure 5

[0076] It will be appreciated that the software components described herein may, when loaded into the CPU 502 and executed, transform the CPU 502 and the overall computer 500 from a general-purpose computing device into a special-purpose computing device that is customized to facilitate the functionality presented herein. The CPU 502 can be constructed from any number of transistors or other discrete circuit elements, which can individually or collectively assume any number of states. More specifically, the CPU 502 can operate as a finite-state machine, in response to executable instructions contained

[0077] ​Encoding the software modules presented herein can also transform the physical structure of computer-readable media presented herein. The specific transformation of physical structure can depend on various factors, in different implementations of this description. Examples of such factors can include, but are not limited to, the technology used to implement the computer-readable media, whether the computer-readable media is characterized as primary or secondary storage, and the like. For example, if the computer-readable media is implemented using semiconductor-based memory, the software disclosed herein can be encoded on the computer-readable media by transforming the physical state of the semiconductor memory. For example, the software can transform the state of transistors, capacitors or other discrete circuit elements constituting the semiconductor memory to store data in the form of two or more states, or changes between those states, that simply are physical modifications or changes including physical transformations.

[0078] As another example, computer-readable media disclosed herein can be implemented using magnetic or optical technology. In such implementations, the software presented herein can transform the physical state of magnetic or optical media, when this software is encoded or stored in the magnetic or optical media. These transformations can include altering the magnetic characteristics of particular locations within given magnetic media. These transformations can also include altering the physical features or characteristics of particular locations within given optical media, to change the optical characteristics of those locations. Other transformations of physical media are possible without departing from the scope and spirit of the present description, with the foregoing examples provided only to facilitate such discussion.

[0079] In light of the above, it should be appreciated that many types of physical transformations take place in the computer 900 in order to store and execute the software components presented herein. It also should be appreciated that Figure 5 The architecture of the computer 500 illustrated in FIG. 5, or a similar architecture, can be used to implement other types of computing devices, including handheld computers, video game devices, embedded computer systems, mobile devices such as smartphones, tablet computers, and AR / VR devices, and other types of computing devices known to those skilled in the art. It is also contemplated that the computer 500 can not include Figure 5 All of the components illustrated in FIG. 5, or a subset thereof, can be included in the computer 500. Figure 5 Other components not explicitly described above can be included in the computer 500, or equivalents thereto, or can be utilized in conjunction with the computer 500, without departing from the scope of the present description. Figure 5 A completely different architecture can be used in the computer 500, without departing from the scope of the present description.

[0080] Figure 6 is a network diagram illustrating a distributed network computing environment 600 in which aspects of the disclosed technology can be implemented, in accordance with various embodiments presented herein. As Figure 6As shown in FIG. 6, one or more server computers 600A can be interconnected via a communication network 520 (which can be any or a combination of a fixed wired or wireless LAN, WAN, intranet, extranet, peer-to-peer network, virtual private network, the Internet, Bluetooth communication network, dedicated low voltage communication network, or other communication network) with a plurality of client computing devices such as, but not limited to, tablet computer 600B, game console 600C, smart watch 600D, telephone 600E (such as a smart phone), personal computer 600F, and AR / VR device 600G.

[0081] In a networked environment in which the communication network 520 is the Internet, for example, the server computer 600A can be a dedicated server computer operable to process and communicate data to and from the devices 600B-600G via any of a number of known protocols, such as the hypertext transfer protocol ("HTTP"), file transfer protocol ("FTP"), or simple object access protocol ("SOAP"). Additionally, the networked computing environment 600 can utilize various data security protocols, such as the secure sockets layer ("SSL") or the pretty good privacy ("PGP"). Each of the client computing devices 600B-600G can be equipped with an operating system that is operable to support one or more computing applications or terminal sessions (such as a web browser Figure 6 (not shown in FIG. 6) or other graphical user interface Figure 6 (not shown in FIG. 6) or mobile phone desktop environment Figure 6 (not shown in FIG. 6)) to gain access to the server computer 600A.

[0082] The server computer 600A can be communicatively coupled to other computing environments Figure 6 (not shown in FIG. 6) and receive data regarding the interactive / resource network of participating users. In illustrative operations, a user Figure 6 (not shown in FIG. 6) can interact with computing applications running on the client computing devices 600B-600G to obtain desired data and / or perform other computing applications.

[0083] Data and / or computing applications can be stored on one or more servers 600A and communicated to cooperating users through the client computing devices 600B-600G over the exemplary communication network 620. Participating users Figure 6 (not shown in FIG. 6) can request access to particular data and applications, in whole or in part, that are housed on the server computer 600A. This data can be communicated between the client computing devices 600B-600G and the server computer 600A for processing and storage.

[0084] The server computer 600A can host computing applications, processes, and applets for generating, authenticating, encrypting, and communicating data and applications, and can cooperate with other server computing environments (not shown in Figure 6 , third-party service providers (not shown in Figure 6 ), network attached storage (NAS), and storage area networks (SAN) to implement application / data transactions.

[0085] It should be appreciated that the computing architectures shown in Figure 5 and the distributed network computing environments shown in Figure 6 have been simplified for a clear understanding of the present disclosure. It should further be appreciated that the computing architectures and distributed computing networks can include and utilize far more computing components, devices, software programs, networking devices, and other components not specifically described herein.

[0086] The present disclosure presented herein also includes the subject matter described in the following clauses:

[0087] Clause 1. A deep neural network (DNN) processor comprising a plurality of neural processing elements, each neural processing element of the plurality of neural processing elements comprising: a first hardware AND gate configured to perform a logical AND operation between a first sign extension signal and a most significant bit (MSB) of a first operand 202A; a first coupler configured to couple an output of the first hardware AND gate and the first operand, and provide a first output to a hardware binary multiplier; a second hardware AND gate configured to perform a logical AND operation between a second sign extension signal and a MSB of a second operand; and a second coupler configured to couple an output of the second hardware AND gate and the second operand, and provide a second output to the hardware binary multiplier.

[0088] Clause 2. The deep neural network processor of clause 1, wherein the first operand and the second operand comprise signed or unsigned binary 8-bit numbers.

[0089] Clause 3. The deep neural network processor of any one of clauses 1 or 2, wherein the first output and the second output comprise 9-bit signed binary numbers.

[0090] Clause 4. The deep neural network processor of any one of clauses 1-3, wherein an output of the hardware binary multiplier comprises a signed 18-bit binary number.

[0091] Clause 5. The deep neural network processor of any one of clauses 1 or 4, wherein the MSB of the first operand comprises a 7th bit of an 8-bit signed or unsigned binary number.

[0092] Clause 6. The deep neural network processor of any of clauses 1 or 5, wherein the MSB of the second operand comprises the 7th bit of an 8-bit signed or unsigned binary number.

[0093] Clause 7. The deep neural network processor of any of clauses 1-6, wherein the state of the first sign extension signal or the second sign extension signal is based on a type of layer of a DNN previously processed by the DNN processor.

[0094] Clause 8. The deep neural network processor of any of clauses 1-7, wherein the layer of the DNN previously processed by the DNN processor comprises a ReLU layer.

[0095] Clause 9. A computer-implemented method comprising: performing a logical AND operation between a first sign extension signal and a most significant bit (MSB) of a first operand; coupling an output of a first hardware AND gate and the first operand; and passing a result of the coupling of the output of the first hardware AND gate and the first operand to a hardware binary multiplier.

[0096] Clause 10. The computer-implemented method of clause 9, further comprising: performing a logical AND operation between a second sign extension signal and a MSB of a second operand; coupling an output of a second hardware AND gate and the second operand; and passing a result of the coupling of the output of the second hardware AND gate and the second operand to the hardware binary multiplier.

[0097] Clause 11. The computer-implemented method of any of clauses 9 or 10, wherein the first operand and the second operand comprise signed or unsigned binary 8-bit numbers.

[0098] Clause 12. The computer-implemented method of any of clauses 9-11, wherein the result of the coupling of the output of the first hardware AND gate and the first operand and the result of the coupling of the second hardware AND gate and the second operand comprise a 9-bit signed binary number.

[0099] Clause 13. The computer-implemented method of any of clauses 9-12, wherein an output of the hardware binary multiplier comprises a signed 18-bit binary number.

[0100] Clause 14. The computer-implemented method of any of clauses 9-13, wherein the state of the first sign extension signal or the second sign extension signal is based on a type of layer of a deep neural network (DNN) previously processed by the DNN processor.

[0101] Clause 15. The computer-implemented method of any of clauses 9-14, wherein the layer of the DNN previously processed by the DNN processor comprises a ReLU layer.

[0102] Clause 16. A deep neural network (DNN) processor comprising a plurality of neural processing elements, each of the neural processing elements configured to: receive an 8-bit operand, the 8-bit operand comprising a signed binary 8-bit operand or an unsigned 8-bit operand; perform a logical AND operation on a most significant bit (MSB) of the 8-bit operand and a binary signal, the binary signal indicating whether a sign extension operation is to be performed on the 8-bit operand; add a new MSB to the 8-bit operand to create a 9-bit signed binary value; store a result of the logical AND operation in the new MSB of the 9-bit signed value; and pass the 9-bit signed value to a multiplier.

[0103] Clause 17. The DNN processor of clause 16, wherein the plurality of neural processing elements are further configured to: receive a second 8-bit operand, the second 8-bit operand comprising a signed binary 8-bit operand or an unsigned 8-bit operand; perform a logical AND operation on a most significant bit (MSB) of the second 8-bit operand and a second binary signal, the second binary signal indicating whether a sign extension operation is to be performed on the second 8-bit operand; add a new MSB to the second 8-bit operand to create a second 9-bit signed binary value; store a result of the logical AND operation in the new MSB of the second 9-bit signed value; and pass the second 9-bit signed value to the multiplier.

[0104] Clause 18. The DNN processor of any one of clauses 16-17, wherein an output of the hardware binary multiplier comprises a signed 18-bit binary number.

[0105] Clause 19. The DNN processor of any one of clauses 16-18, wherein a state of the binary signal is based on a type of a layer of a DNN previously processed by the DNN processor.

[0106] Clause 20. The DNN processor of any one of clauses 16-19, wherein the binary signal indicating whether a sign extension operation is to be performed on the 8-bit operand is set based on a non-linearity applied in a layer of a DNN previously processed by the DNN processor.

[0107] Based on the above, it should be appreciated that a neural processing element has been disclosed herein that can provide improved accuracy. While the subject matter introduced in this document has been described in the context of specific features, methods, and transformation actions specific to computer structural features, methods, and transformation actions, specific computer machines, and computer readable media, it is to be understood that the subject matter described in the following claims is not necessarily limited to the specific features, methods, and transformation actions, specific computer machines, and computer readable media described herein. Rather, the specific features and actions are disclosed as illustrative forms of implementing the claimed subject matter.

[0108] The above-described topics are provided by way of illustration only, and should not be construed as limiting. Various modifications and changes can be made to the subject matter described herein without following the illustration of the examples described and without departing from the scope of the disclosure, which is set forth in the following claims.

Claims

1. A deep neural network (DNN) processor, comprising a plurality of neural processing elements, each of the plurality of neural processing elements comprising: The first hardware AND gate is configured to perform a logical AND operation between the first sign extension signal and the most significant bit (MSB) of the first operand. A first connector is configured to connect the output of the first hardware AND gate and the first operand, and to provide the first output to a hardware binary multiplier. The second hardware AND gate is configured to perform a logical AND operation between the second sign extension signal and the MSB of the second operand, and A second connector is configured to connect the output of the second hardware AND gate and the second operand, and to provide the second output to the hardware binary multiplier.

2. The deep neural network processor of claim 1, wherein the first operand and the second operand comprise signed or unsigned 8-bit binary numbers.

3. The deep neural network processor according to claim 1, wherein the first output and the second output comprise 9-bit signed binary numbers.

4. The deep neural network processor of claim 1, wherein the output of the hardware binary multiplier comprises a signed 18-bit binary number.

5. The deep neural network processor of claim 1, wherein the MSB of the first operand comprises the 7th bit of an 8-bit signed or unsigned binary number.

6. The deep neural network processor of claim 1, wherein the MSB of the second operand comprises the 7th bit of an 8-bit signed or unsigned binary number.

7. The deep neural network processor of claim 1, wherein the state of the first symbolic extension signal or the state of the second symbolic extension signal is based on the type of layer of the DNN previously processed by the DNN processor.

8. The deep neural network processor of claim 7, wherein the layers of the DNN previously processed by the DNN processor include ReLU layers.

9. A computer-implemented method, comprising: The first hardware AND gate performs a logical AND operation between the first symbol extension signal and the most significant bit (MSB) of the first operand. Connect the output of the first hardware AND gate to the first operand; The second hardware AND gate performs a logical AND operation between the second symbol extension signal and the MSB of the second operand; Connect the output of the second hardware AND gate to the second operand; as well as The result of concatenating the output of the first hardware AND gate and the first operand, and the result of concatenating the second hardware AND gate and the second operand, are passed to the hardware binary multiplier.

10. The computer-implemented method of claim 9, wherein the first operand and the second operand comprise signed or unsigned 8-bit binary numbers.

11. The computer-implemented method of claim 9, wherein the result of the connection between the output of the first hardware AND gate and the first operand, and the result of the connection between the output of the second hardware AND gate and the second operand, comprise 9-bit signed binary numbers.

12. The computer-implemented method of claim 11, wherein the output of the hardware binary multiplier comprises a signed 18-bit binary number.

13. The computer-implemented method of claim 12, wherein the state of the first symbolic extension signal or the state of the second symbolic extension signal is based on the type of layer of the DNN previously processed by the deep neural network (DNN) processor.

14. The computer-implemented method of claim 13, wherein the layer of the DNN previously processed by the DNN processor includes a ReLU layer.

15. A deep neural network (DNN) processor, comprising a plurality of neural processing elements, each of the neural processing elements being configured to: Receive 8-bit operands, including signed binary 8-bit operands or unsigned 8-bit operands; Perform a logical AND operation on the most significant bit (MSB) of the 8-bit operand and a binary signal, wherein the binary signal indicates whether to perform a sign extension operation on the 8-bit operand; Add the new MSB to the 8-bit operand to create the first 9-bit signed binary value; The result of the logical AND operation is stored in the new MSB of the first 9-bit signed value; Receive a second 8-bit operand, which may be a signed 8-bit binary operand or an unsigned 8-bit operand; Perform a logical AND operation on the most significant bit (MSB) and the second binary signal of the second 8-bit operand, whereby the second binary signal indicates whether to perform a sign extension operation on the second 8-bit operand. Add the new MSB to the second 8-bit operand to create the second 9-bit signed binary value; The result of the logical AND operation is stored in the new MSB of the second 9-bit signed value; as well as The first 9-bit signed value and the second 9-bit signed value are passed to the multiplier.

16. The DNN processor of claim 15, wherein the output of the hardware binary multiplier comprises a signed 18-bit binary number.

17. The DNN processor of claim 16, wherein the state of the binary signal is based on the type of the layer of the DNN previously processed by the DNN processor.

18. The DNN processor of claim 15, wherein the binary signal indicating whether to perform a sign extension operation on the 8-bit operand is set based on nonlinearities applied in the layers of the DNN previously processed by the DNN processor.

Images