Neural Network Processor

Abstract

In one example, the neural network processor includes a memory interface (534), an instruction buffer (520), a weight buffer (526), ​​an input data register (528a), a weight register (528b), an output data register (528a), a computing engine (524), and a controller (522). The controller is configured to receive a first instruction from the instruction buffer and, in response to the first instruction, fetch input data elements from the memory interface to the input data registers and fetch weight elements from the weight buffer to the weight registers, receive a second instruction from the instruction buffer and, in response to the second instruction, fetch the input data elements and weight elements from the input data registers and the weight registers, respectively, to the computing engine and perform an arithmetic operation between the input data elements and the weight elements using the computing engine to generate output data elements.

Description

[Technical Field]

[0001] An artificial neural network is a computing system with an architecture based on biological neural networks. An artificial neural network can be trained in a training process using training data to learn how to perform certain computational tasks. An artificial neural network can be implemented on a neural network processor, which can include memory and computing resources to support the computational operations of the artificial neural network. Some applications impose limitations on the amount of memory and computing resources available on a neural network hardware accelerator. Summary of the Invention

[0002] In one example, a neural network processor is provided. The neural network processor includes a memory interface, an instruction buffer, a weight buffer, input data registers, weight registers, output data registers, a computing engine, and a controller. The controller is configured to receive a first instruction from the instruction buffer and, in response to the first instruction, fetch input data elements from the memory interface to the input data registers and fetch weight elements from the weight buffer to the weight registers. The controller is also configured to receive a second instruction from the instruction buffer and, in response to the second instruction, fetch the input data elements and weight elements from the input data registers and the weight registers to the computing engine, respectively, perform an arithmetic operation between the input data elements and the weight elements using the computing engine to generate output data elements, and store the output data elements in the output data registers.

[0003] In one example, a method is provided that includes receiving a first instruction from an instruction buffer of a neural network processor, and in response to the first instruction, fetching input data elements from a memory external to the neural network processor into input data registers of the neural network processor and fetching weight elements from a weight buffer of the neural network processor into weight registers of the neural network processor, the method further includes receiving a second instruction from the instruction buffer, and in response to the second instruction, fetching the input data elements and weight elements from the input data registers and weight registers, respectively, into a computing engine of the neural network processor, performing an arithmetic operation between the input data elements and the weight elements using the computing engine to generate output data elements, and storing the output data elements in output data registers of the neural network processor.

[0004] In one example, a neural network processor includes a memory interface, an instruction buffer, a weight buffer, input data registers, weight registers, output data registers, address registers, an address generation engine, a data load / store engine, a computing engine, and a controller. The address generation engine is configurable to set an input data address, an output data address, and a weight address in the address register. The data load / store engine is configurable to fetch input data from the memory interface to the input data register based on the input data address, fetch output data from the output data register to the memory interface based on the output data address, and fetch weights from the weight buffer to the weight register based on the weight address. The computing engine is configurable to perform an operation based on the input data and the weights to generate output data. The controller is configured to, in response to one or more instructions from the instruction buffer, extract a first subinstruction directed to the address generation engine to set one of the input data address or the output data address, extract a second subinstruction directed to the address generation engine to set a weight address, extract a third subinstruction directed to the arithmetic engine to perform an operation, extract a fourth subinstruction directed to the data load / store engine to fetch a weight at a weight address in the weight buffer, and extract a fifth subinstruction directed to the data load / store engine to fetch input data at the input data address via the memory interface, and configure, in parallel, the address generation engine using the first and second subinstructions, the arithmetic engine using the third subinstruction, and the data load / store engine using the fourth and fifth instructions. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a schematic diagram illustrating a system in which inference operations may be performed, according to some examples.

[0006] [Figure 2] 2 is a schematic diagram illustrating example data processing operations performed by electronic devices of the system of FIG. 1, according to some examples. [Figure 3] 2 is a schematic diagram illustrating example data processing operations performed by electronic devices of the system of FIG. 1, according to some examples. [Figure 4] 2 is a chart illustrating example data processing operations performed by electronic devices of the system of FIG. 1, according to some examples.

[0007] [Figure 5] FIG. 1 is a schematic diagram illustrating a neural network processor, in accordance with some examples.

[0008] [Figure 6A] 6 is a chart illustrating example instructions executable by the neural network processor of FIG. 5, according to some examples. [Figure 6B] 6 is a chart illustrating example instructions executable by the neural network processor of FIG. 5, according to some examples. [Figure 6C] 6 is a chart illustrating example instructions executable by the neural network processor of FIG. 5, according to some examples. [Figure 6D] 6 is a chart illustrating example instructions executable by the neural network processor of FIG. 5, according to some examples.

[0009] [Figure 7] 6 includes a chart illustrating a set of instructions executable by the neural network processor of FIG. 5, according to some examples.

[0010] [Figure 8A] 6 is a chart illustrating a memory management scheme provided by the neural network processor of FIG. 5, according to some examples. [Figure 8B]6 is a chart illustrating a memory management scheme provided by the neural network processor of FIG. 5, according to some examples. [Figure 9] 6 is a chart illustrating a memory management scheme provided by the neural network processor of FIG. 5, according to some examples.

[0011] [Figure 10A] FIG. 6 is a schematic diagram illustrating a circular buffer circuit of the neural network processor of FIG. 5, in accordance with some examples.

[0012] [Figure 10B] 10B is a chart illustrating operation of the circular buffer circuit of FIG. 10A, in accordance with some examples.

[0013] [Figure 11] 6C is a chart illustrating flow control elements of the instruction syntax of FIGS. 6A and 6B, according to some examples. [Figure 12] 6C is a chart illustrating flow control elements of the instruction syntax of FIGS. 6A and 6B, according to some examples.

[0014] [Figure 13] 6 is a schematic diagram illustrating internal components of a computation engine of the neural network processor of FIG. 5, according to some examples.

[0015] [Figure 14] 14 includes a chart illustrating the convolution operation performed by the computation engine of FIG. 13, according to some examples.

[0016] [Figure 15A] 14 includes a chart illustrating post-processing operations performed by the computation engine of FIG. 13 for different weight precisions, according to some examples. [Figure 15B] 14 includes a chart illustrating post-processing operations performed by the computation engine of FIG. 13 for different weight precisions, according to some examples. [Figure 15C] 14 includes a chart illustrating post-processing operations performed by the computation engine of FIG. 13 for different weight precisions, according to some examples. [Figure 15D] 14 includes a chart illustrating post-processing operations performed by the computation engine of FIG. 13 for different weight precisions, according to some examples.

[0017] [Figure 16] 14 is a schematic diagram illustrating arithmetic operations performed by the computation engine of FIG. 13, in accordance with some examples.

[0018] [Figure 17] 14 is a schematic diagram illustrating a multiplication circuit of the arithmetic engine of FIG. 13, in accordance with some examples.

[0019] [Figure 18] 14 is a schematic diagram illustrating a computing unit of the computing engine of FIG. 13, in accordance with some examples.

[0020] [Figure 19] 19 is a schematic diagram illustrating an accumulator of the arithmetic unit of FIG. 18, in accordance with some examples.

[0021] [Figure 20A] 19 is a schematic diagram illustrating internal components of the computation engine of FIG. 13, including the computation unit of FIG. 18, in accordance with some examples. [Figure 20B] 19 is a schematic diagram illustrating internal components of the computation engine of FIG. 13, including the computation unit of FIG. 18, in accordance with some examples.

[0022] [Figure 21A-1] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21A-2] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21B-1]14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21B-2] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21C-1] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21C-2] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21D-1] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21D-2] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21E-1] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21E-2] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21F-1] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21F-2] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21G-1] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21G-2] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21H-1]14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples. [Figure 21H-2] 14A-14C are schematic diagrams illustrating different configurations of the computation engine of FIG. 13 for different input precision and weight precision, according to some examples.

[0023] [Figure 22A] 14 is a schematic diagram illustrating an internal multiplexer of the computation engine of FIG. 13, in accordance with some examples. [Figure 22B] 14 is a schematic diagram illustrating an internal multiplexer of the computation engine of FIG. 13, in accordance with some examples. [Figure 22C] 14 is a schematic diagram illustrating an internal multiplexer of the computation engine of FIG. 13, in accordance with some examples.

[0024] [Figure 23] 14 is a schematic diagram illustrating internal components of the post-processing engine of FIG. 13, in accordance with some examples.

[0025] [Figure 24] 14 is a schematic diagram illustrating internal components of the post-processing engine of FIG. 13, in accordance with some examples.

[0026] [Figure 25A] 14 is a schematic diagram illustrating a configuration of the post-processing engine of FIG. 13, in accordance with some examples. [Figure 25B] 14 is a schematic diagram illustrating a configuration of the post-processing engine of FIG. 13, in accordance with some examples. [Figure 25C] 14 is a schematic diagram illustrating a configuration of the post-processing engine of FIG. 13, in accordance with some examples.

[0027] [Figure 26A-1] FIG. 14 is a schematic diagram illustrating the internal components and their operation of the max pooling engine of FIG. 13, in accordance with some examples. [Figure 26A-2] FIG. 14 is a schematic diagram illustrating the internal components and their operation of the max pooling engine of FIG. 13, in accordance with some examples. [Figure 26B] 14 is a chart illustrating the internal components and their operation of the max pooling engine of FIG. 13, according to some examples.

[0028] [Figure 27] 1 is a flowchart illustrating the operation of a neural network processor, in accordance with some examples. [Figure 28] 1 is a flowchart illustrating the operation of a neural network processor, in accordance with some examples. [Figure 29A] 1 is a flowchart illustrating the operation of a neural network processor, in accordance with some examples. [Figure 29B] 1 is a flowchart illustrating the operation of a neural network processor, in accordance with some examples.

[0029] In the drawings, the same reference numbers are used to denote the same (or similar) features. DETAILED DESCRIPTION OF THE INVENTION

[0030] FIG. 1 is a schematic diagram illustrating a system 100. The system 100 may include multiple electronic devices 102, including electronic devices 102a, 102b, and 102c, and a cloud network 103. Each electronic device 102 may include a sensor 104 and a data processor 106. For example, electronic device 102a includes sensor 104a and data processor 106a, electronic device 102b includes sensor 104b and data processor 106b, and electronic device 102c includes sensor 104c and data processor 106c. The sensors 104 may be of various types, such as audio / acoustic sensors, motion sensors, image sensors, etc. In some examples, each electronic device 102 may include multiple sensors of different types (e.g., acoustic sensors and motion sensors) or multiple instances of the same type of sensor (e.g., multiple microphones). Each sensor system can receive a stimulus 108 (e.g., an acoustic signal, an optical signal, motion, etc.) and generate a decision 110 based on the received stimulus. The decisions may indicate, for example, whether an event of interest has been detected. For example, electronic device 102a may generate decision 110a based on stimulus 108a, electronic device 102b may generate decision 110b based on stimulus 108b, and electronic device 102c may generate decision 110c based on stimulus 108c. In some examples, each electronic device 102 may be an end node of the Internet of Things (IoT) and may be an edge device of network 103. Each electronic device 102 may transmit its respective decision 110 to cloud network 103, which may take action based on the decision (e.g., send an alert about the abnormal event, contact law enforcement, etc.).

[0031] The data processor 106 of a particular electronic device 102 may perform data processing operations on data collected by the sensors 104 on the particular electronic device to generate decisions 110. For example, in examples where the sensors 104 include audio / acoustic sensors, the data processor 106 may perform data processing operations such as keyword spotting, voice activity detection, and detection of particular acoustic signatures (e.g., broken glass, shots fired). Also, in examples where the sensors 104 include motion sensors, the data processor 106 may perform data processing operations such as vibration detection, activity recognition, and anomaly detection (e.g., whether a window / door is knocked or opened when no one is home or at night). Also, in examples where the sensors 104 include image sensors, the data processor 106 may perform data processing operations such as face recognition, gesture recognition, and visual wake word detection (e.g., determining whether a person is present in an environment). The data processor 106 may also generate and output decisions 110 based on results of data processing operations including, for example, detecting keywords in speech, detecting particular acoustic signatures, particular activities, particular gestures, etc. If the electronic device 102 includes multiple sensors, the data processor 106 may perform sensor fusion operations on different types of sensor data to generate the decisions 110.

[0032] The data processor 106 may include various circuit elements for processing the sensor signals generated by the sensor 104. For example, the data processor 106 may include a sample-and-hold (S / H) circuit for sampling the sensor signals. The data processor 106 may include an analog-to-digital converter (ADC) for quantizing the samples into a digital signal. The data processor 106 may also include a neural network processor for implementing an artificial neural network for processing the samples. The artificial neural network (hereinafter referred to as "neural network") may include multiple processing nodes. The neural network may perform inference or classification operations on the sensor data to generate the aforementioned decision. The inference operation may be performed to generate a decision by combining the sensor data with a set of weight elements obtained from a neural network training operation. Examples of neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), etc.

[0033] The processing nodes of a neural network may be divided into various layers, including, for example, an input layer, multiple hidden layers (e.g., hidden layers), and an output layer. The input layer and hidden layer may each be a convolutional layer forming a CNN, while the output layer may be a fully connected layer, with the input layer, hidden layer, and output layer together forming a DNN. Each processing node in the input layer receives an element of an input set and scales the element with a weight factor to indicate the element's influence on the output. The input set may include, for example, acoustic data, motion data, image data, a combination of different types of data, or a set of input features extracted from such data. Additionally, the processing nodes in the hidden layer may combine the scaled elements received from each processing node in the input layer to compute a set of intermediate outputs. For example, each processing node in the hidden layer may generate an intermediate output by computing the sum of element-weight products and then applying an activation function to the sum. The intermediate outputs from each processing node in one hidden layer may be considered as activated votes (or non-votes) to determine the intermediate outputs of the next hidden layer, and may be associated with weights indicating the influence of the vote. The intermediate outputs may represent output features of a particular immediately preceding layer. The output layer may generate a sum of scaled intermediate outputs from the final intermediate layer. In some examples, the output layer may generate a binary output (e.g., "yes" or "no") based on whether the sum of the scaled intermediate outputs exceeds a threshold, which may indicate a decision from a data processing operation (e.g., detection of a keyword in speech, a particular acoustic signature, a particular activity, or a particular gesture).

[0034] The neural network processor of the data processor 106 can be programmed to perform operations based on an artificial neural network model. The neural network processor can be programmed based on an instruction sequence including computational operations (e.g., addition, multiplication, activation function processing, etc.) associated with the model. The instructions can also access internal and external memory devices to retrieve and store data. A compiler can receive information about the neural network model, input data, and available memory and computing resources and generate a set of instructions indicating, for example, when to access internal and external memory devices for such data, which components of the neural network processor perform operations on the data based on the neural network model, etc., to perform neural network processing. In the example of FIG. 1, the data processor 106 of each electronic device 102 can receive data 120 (e.g., 120a, 120b, and 120c) from the cloud network 103, including, for example, a set of weight elements, compiled instructions representing each layer of the neural network model, etc. In some examples, the neural network processor of each electronic device 102 can perform training operations to adjust initial weight elements. In some examples, the training operation may also be performed in the cloud network 103, and each electronic device 102 may receive updated weight factors from the cloud network 103 and perform the inference / classification operation using the updated weights.

[0035] 2, 3, and 4 are schematic diagrams illustrating example data processing operations performed by the electronic device 102 when processing an acoustic signal. FIG. 2 illustrates an example data processing operation 200 for identifying words in speech. The example data processing operation includes a feature extraction operation 202, a DNN layer processing operation 204, and a post-processing operation 206. As part of the feature extraction operation 202, the data processor 106 can receive samples of an acoustic signal 212 from the sensor 104 and extract features 214 from the samples. The acoustic signal can represent a speech signal having different magnitude / power levels over time. Additionally, the features 214 can include distinguishing characteristics of the acoustic signal, and based on the distinguishing characteristics of the acoustic signal, subsequent processing operations (e.g., the DNN layer processing operation 204 and the post-processing operation 206) can identify words in speech. In the example of FIG. 2, the features 214 can include a time-frequency distribution of signal power. In some examples, the feature extraction operation 202 may include a convolution operation, which may be performed by a neural network processor implementing a CNN of the data processor 106, a general-purpose processor (e.g., a microcontroller), a compute-in-memory (CIM) circuit, or the like.

[0036] Additionally, as part of the DNN layer processing operations 204, the data processor 106 may process the features 214 using a multi-layer DNN and a set of weight elements 216 to compute a set of outputs 218. The post-processing operations 206 may post-process and quantize the outputs 218 to generate inference outputs 220. The post-processing operations may include, for example, activation function processing to map the outputs 218 to a set of inference outputs, as well as other post-processing operations such as batch normalization (Batch Norm, or BNorm) and residual layer processing to facilitate convergence during training. The inference outputs 220 may include a set of probabilities for a set of candidate words. In the example of FIG. 2, the inference outputs 220 may map the candidate word "yes" with a probability of 0.91, "no" with a probability of 0.02, and other words with a probability of 0.01. In some examples, a determination that the speech signal 212 contains the word "yes" may be made as part of the post-processing operations 206 based on the candidate word "yes" being mapped to the highest probability among the candidate words. In some examples, as shown in FIG. 2, the output of the post-processing operation 206, which may represent the output of a neural network layer, may be fed back to the DNN layer processing operation 204 as an input to another neural network layer.

[0037] 3 illustrates an example of an exemplary processing operation 300 that may be part of the DNN layer processing operation 204 of FIG. 2. In FIG. 3, the processing operation 300 may be a convolution operation performed by a neural network layer on a set of input data elements 302 to generate a set of output data elements 304. The input data elements 302 and the output data elements 304 may each be in the form of a multi-dimensional array including a first dimension (input height), a second dimension (input width), and a third dimension (input channels). The array of input data elements 302 may be in the form of a multi-dimensional array including a first dimension (input height), a second dimension (input width), and a third dimension (input channels). h , width F wand may have Nin input channels. For example, an input data element 302 may represent a feature 214, with a first dimension representing time, a second dimension representing frequency, and a third dimension representing a data channel (or channels). A channel may represent a data source. For example, if an input data element 302 is audio data generated by multiple microphones, data generated from a particular microphone may be associated with a particular data channel. As another example, if an input data element 302 is image data generated by an image sensor for different color channels (e.g., red, green, and blue), data generated for a particular color channel may be associated with a particular data channel.

[0038] To perform the convolution operation to compute one output data element (e.g., output data element 304a), the data processor 106 may perform a dot product between the set of weight elements 306 and a subset of the input data elements 302. In the example of FIG. 3, the weight elements 306 also have a weight height K h , weight width K w , and N in input channels. The set of weight elements 306 may be shifted by a stride D relative to the set of input data elements 302 so as to overlap with different subsets of the input data elements 302. Different dot products (and different output data elements 304) may be computed with different strides between the set of weight elements 306 and different overlapping subsets of the input data elements 302, as follows:

number

[0039] In Equation 1, r represents the index along the width dimension, s represents the index along the height dimension, and c represents the index along the input channel dimension. c eD+r,fD+srepresents an input data element 302 of input channel c, has subscripts eD+r along the width dimension and fD+s along the height dimension, where e and f are multiples of the stride D, and F c r,s represents a weight data element 306 with subscript r along the width dimension and subscript s along the height dimension, and Y e,f represents an output data element 304 with subscript e along the width dimension and subscript f along the height dimension. In some examples, the data processor 106 performs a convolution operation between the input data element 302 and multiple sets of weight elements 306 to produce N out The input data elements 304 may generate output data elements 304 having output channels, and a convolution between the input data elements and one set of weight elements 306 may generate the output data elements 304 for a particular output channel.

[0040] 4 shows an example set of loop instructions 400. In some examples, the loop instructions 400 may represent a portion of the processing operations 300 of FIG. 3 or other processing operations performed by one neural network layer. For example, the set of loop instructions 400 may represent a portion of a convolution operation with a particular stride D. The neural network processor of the data processor 106 executes the set of loop instructions to select N input data elements 302 and weight elements 306 from multiple sets. out The loop instruction 400 can operate on a multidimensional array of output data elements 304 having output channels (0 to N). out Dimensions and input channels (0 to N) in The outermost loops 402 and 404 sweep the range of indices in the dimension m (between 0 and -1) and the input height (0 to F h Dimension and input width (i between -1) (0 to F w Intermediate loops 406 and 408 sweep the range of indices in the j dimension (between 0 and -1) and the weight filter / kernel height (between 0 and K h Dimension and weight range (h between 0 and K) wand innermost loops 410 and 412 that sweep a range of indices in the w dimension (between -1 and -1), and compute an output data element Y as a dot product between an input data element associated with an input height, input width, and input channel index range and a weight element associated with a weight height, weight width, input channel, and output channel index range.

[0041] The parameters of the set of loop instructions 400 may be configured based on the particular type / topology of the neural network layer represented by the set of loop instructions. For example, to implement a depthwise convolutional layer in which each output data element is generated from a dot product of an input data element and a weight element for the same input channel, the outermost loops 402 and 404 may be merged into one loop, or the variable m may be set to a constant. Also, to implement an average pooling layer, all of the weight elements provided to the neural network layer may be multiplied by 1 / (K W ×K h ) and each output data element can be set to the number of input data elements and input channels N in To implement a pointwise convolutional layer generated from a dot product with a 1 × 1 kernel with depth equal to w and K. h (width and height of the weight element array) can be set to 1. Also, to implement a fully connected layer, K w , K. h , F w , and F h can each be set to 1.

[0042] Referring again to FIG. 1 , having the electronic device 102 perform inference operations on sensor data locally, rather than transmitting the sensor data to the cloud network 103 for inference, can provide various advantages. For example, by storing and processing the sensor data locally, transmission of sensor data from the electronic device 102 to the cloud network 103 can be reduced or entirely eliminated. Such a configuration can reduce the power and bandwidth associated with transmitting sensor data, especially when the electronic device 102 continuously generates large amounts of sensor data (e.g., is part of an always-on system). It can also reduce security and privacy risks associated with transmitting sensor data (e.g., images, audio data, etc.). It can also improve the speed and reliability of providing inference decisions, or at least reduce the impact of factors external to the electronic device 102, such as network bottlenecks in transmitting sensor data, unavailability of the cloud network 103, etc.

[0043] Although it is advantageous to have the electronic device 102 perform inference operations on sensor data locally, various challenges exist. Specifically, inference operations can be power-intensive and use large amounts of memory and computing resources, even when performed by dedicated hardware such as a neural network processor. On the other hand, the electronic device 102 may be a low-power device and have a small form factor, especially if it is an IoT device. Therefore, the neural network processor on the electronic device 102 may have very limited memory and computing resources available for performing the inference operation. Also, different applications may have different and conflicting requirements for the inference operation. For example, some applications may require high accuracy in the inference operation, while some other applications may not require such high accuracy. Also, the neural network processor may support a wide range of neural network topologies, layer types, kernel / filter sizes, and filter, input data, and output data dimensions to support different applications. All of these present challenges for having a neural network processor that can perform various inference operations with limited memory and computing resources to support a wide range of applications.

[0044] FIG. 5 is a schematic diagram illustrating example internal components of an electronic device 102 having a neural network processor 502 that can address at least some of the challenges described above. Referring to FIG. 5 , the electronic device 102 may include the neural network processor 502 coupled to a memory 512, a direct memory access (DMA) controller 516, a processor 514, and a sensor interface circuit 517 via interconnect 518. The sensor interface circuit 517 may include interface circuitry (e.g., an analog-to-digital converter (ADC)) that interfaces with the sensor 104. In some examples, the neural network processor 502, the memory 512, the DMA controller 516, the interconnect 518, and the sensor 104 may be part of an integrated circuit (IC), such as a system-on-chip (SoC). In some examples, the neural network processor 502 may be a standalone application-specific integrated circuit (ASIC). The interconnect 518 may be an SoC bus interconnect or another type of interconnect. Alternatively, the processor 514 may be a general-purpose processor, including, for example, a central processing unit (CPU), one or more processor cores, etc., capable of executing instructions to perform various tasks.

[0045] Memory 512 is shared by neural network processor 502 and processor 514. Memory 512 may store instructions, input data, and weights for each neural network layer provided to neural network processor 502 to perform inference operations, as well as output data generated by neural network processor 502 from the inference operations. The input data may include, for example, sensor data provided by sensors 104, feature data extracted from the sensor data, or intermediate output data generated by previous neural network layers. Memory 512 may also store other data, such as program instructions and data for processor 514. Memory 512 may include any suitable on-chip memory, such as a flash memory device, static random access memory (SRAM), resistive random access memory (ReRAM), etc.

[0046] By having processor 514 and neural network processor 502 share memory 512, rather than providing separate dedicated memory devices for neural network processor 502 and processor 514, the overall memory size of electronic device 102 can be reduced, which may reduce the power and footprint of electronic device 102. As described in more detail below, to reduce memory resource usage and memory data movement, neural network processor 502 can implement various memory management schemes, such as in-place operations where output data overwrites input data using circular addressing of output data, and circular addressing of input data to support always-on applications, which may allow neural network processor 502 to operate with limited memory resources. Neural network processor 502 is also configured to handle variable latency for accessing data from memory 512, thereby enabling concurrent operation of processor 514 with minimal performance impact on neural network processor 502.

[0047] The processor 514 can execute software programs that use the neural network processor 502 to perform inference operations and then perform additional operations based on the results of the inference operations. For example, the processor 514 can execute a software program for a home security system. The software program can include instructions (e.g., application programming interfaces (APIs)) for the neural network processor 502. One API can be associated with operations at one neural network layer, and the software program can include multiple APIs for multipole neural network layers. When invoking / executing an API, the processor 514 can provide the neural network processor 502 with neural network layer instructions executable by the neural network processor 502, as well as memory addresses of weights, parameters, and input data, which the neural network processor can then fetch into memory addresses in the memory 512. The processor 514 can also send control signals to the neural network processor 502 to initiate operations at that neural network layer. Once a neural network layer operation is complete and the output data is stored in memory 512, neural network processor 502 can send the memory address of the output data and control signals back to processor 514 to signal completion, and processor 514 can invoke another API for the next neural network layer. Processor 514 can also execute software programs to perform other functions, such as sending inference decisions to cloud network 103 (or other devices / systems), providing a graphical user interface, among others. Processor 514 can perform such other functions concurrently with neural network processor 502.

[0048] DMA controller 516 can be configured to perform DMA operations to transfer certain data between memory 512 and neural network processor 502. For example, when calling an API, processor 514 can provide memory addresses for stored instructions, weights, and parameters to neural network processor 502 (e.g., in the form of a memory descriptor). Neural network processor 502 can then retrieve the stored instructions, weights, and parameters based on the memory addresses provided by processor 514. As described below, in some examples, neural network processor 502 can fetch input data directly from memory 512 as needed, instead of fetching the input data in bulk using DMA controller 516. Also, neural network processor 502 can store newly generated output data directly in memory 512 in relatively small chunks (e.g., 32-bit chunk size) as the output data is generated and reaches the chunk size, instead of fetching output data in bulk using DMA controller 516. Such a configuration can avoid having input / output data buffers on the neural network processor 502 (or at least reduce the size of the input / output data buffers), reducing the power and area footprint of the neural network processor 502.

[0049] The neural network processor 502 may be a neural network hardware accelerator and may provide hardware resources, including computational and memory resources, for neural network layer operations to support inference operations. The neural network processor 502 may include an instruction buffer 520, an arithmetic controller 522, and an arithmetic engine 524 with configurable arithmetic precision. The neural network processor 502 also includes a weight and parameter buffer 526, registers 528, a load / store controller 530, and an address generator 532. The load / store controller 530 further includes a memory interface 534. Each component of the neural network processor 502 may include combinational logic circuits (e.g., logic gates), sequential logic circuits (e.g., latches, flip-flops, etc.), and / or memory devices (e.g., SRAM) to support various operations of the neural network processor 502, as described below.

[0050] Instruction buffer 520 can fetch and store operation instructions for a neural network layer from memory 512 in response to control signals from processor 514. For example, in response to an API being called to initiate a neural network layer operation, processor 514 can control instruction buffer 520 to transfer instructions (e.g., microcode) for the neural network layer operation from memory 512 or receive instructions from another source (e.g., another processor) and store the instructions in instruction buffer 520.

[0051] The computation controller 522 can decode each computation instruction stored in the instruction buffer 520 and control the computation engine 524, the load / store controller 530, and the address generator 532 to perform operations based on the instruction. For example, in response to an instruction to perform a convolution operation, the computation controller 522 can control the computation engine 524 to perform the operation for the convolution operation after data and weights for the convolution operation are fetched and stored in the data register 528a and the weight / parameter register 528b, respectively. The computation controller 522 can maintain a program counter (PC) that tracks / indicates the next instruction to be executed. In some examples, the computation instruction can include a flow control element, such as a loop element, a macro element, etc., that can be extracted by the computation controller 522. The computation controller 522 can then modify the PC value in response to the flow control element and modify the flow / sequence of execution of the instructions based on the flow control element. Such a configuration may be made more compact by allowing neural network layer instructions to include loop instructions that reflect convolutional layer operations, such as those shown in FIG. 4, thereby reducing microcode size and facilitating the translation of neural network layer topologies into instructions.

[0052] The computation engine 524 may include circuit elements for performing convolution operations to support CNN network layer operations and circuit elements for performing post-processing operations (e.g., BNorm and residual layer processing) on ​​neural network output data. As described below, the computation engine 524 is configurable to perform MAC (e.g., convolution) and post-processing operations on weights and input data over a range of bit precisions (e.g., binary precision, ternary precision, 4-bit precision, 8-bit precision, etc.) based on parameters provided by the processor 514 and stored in configuration registers 528c. The parameters may be provided for a particular neural network layer and may be updated during execution of different neural network layers. This allows the processor 514 to dynamically configure the bit precision of the convolution and post-processing operations in the computation engine 524 based on, for example, the need to perform inference operations, the application that will use the inference results, the available power of the electronic device 102, etc. In some examples, the computation engine 524 allows for different bit precisions for the weights and input data for different neural network layers, which can enable a wide range of precision-computation tradeoffs. This also allows the neural network processor 502 to operate as a domain-specific or application-specific instruction set processor, which can be configured / customized to efficiently perform certain applications (e.g., machine learning based on deep neural networks).

[0053] The weight and parameter buffer 526 can store weights for multiple neural network layers and parameters for different internal components of the neural network processor 502 to support post-processing operations. The neural network processor 502 can fetch the weights and parameters from memory 512, for example, via DMA controller 516. In some examples, the weight and parameter buffer 526 can include an SRAM device. Having the weight and parameter buffer 526 store weights and parameters that are static for a particular neural network layer can reduce the movement of such static data between the neural network processor 502 and memory 512 during operations on the neural network layer. Also, the size of the weight and parameter data can be relatively small compared to the size of the input and output data for the neural network layer operations, thereby allowing the weight and parameter buffer 526 to have a small footprint.

[0054] Additionally, registers 528 may include data registers 528a, weight and parameter registers 528b, address registers 528c, and configuration registers 528d. Data registers 528a may store a subset of input data and a subset of output data for neural network layer operations, and weight and parameter registers 528b may store a subset of weights for neural network layer operations and parameters for post-processing operations. Address register 528c may store memory addresses accessed by load / store controller 530 in memory 512 for weights and input / output data. Address register 528c may store addresses within weight and parameter buffer 526 accessed by load / store controller 530 to fetch weights and parameters. Configuration register 528d may store configuration parameters that are common to various components for neural network processor 502. For example, configuration register 528d may store parameters for setting the bit precision of input data elements and weight elements for convolution and post-processing operations in computation engine 524, the particular memory management scheme of neural network processor 502 (e.g., in-place operations, circular addressing of input data, etc.), etc. As mentioned above, some of these parameters may be provided / updated by processor 514 during execution of the neural network or between executions of two neural network layers.

[0055] Read / write operations of data register 528 a and weight register 528 b may be performed by load / store controller 530 based on instructions executed by arithmetic controller 522. For example, in response to an instruction indicating fetching input data into data register 528 a, load / store controller 530 may fetch the input data from memory 512 directly (e.g., without going through DMA controller 516) via memory interface 534 and store the fetched data in data register 528 a. Also, in response to an instruction indicating storing output data back in memory 512, load / store controller 530 may fetch output data from data register 528 a and store the output data in memory 512 directly (e.g., without going through DMA controller 516) via memory interface 534. As explained above, such a configuration may avoid having input / output data buffers on neural network processor 502 (or reduce the size of the input / output data buffers), thereby reducing the power and footprint of neural network processor 502.

[0056] As will be described later, load / store controller 530 can implement various memory management schemes, such as in-place operation, in which output data overwrites input data, and circular addressing of input data to support always-on applications, by setting the memory addresses stored in address register 528c. Such a configuration can reduce the footprint of input / output data in memory 512, reduce movement of input / output data within memory 512, and facilitate shared access to memory 512 between neural network processor 502 and processor 514. Meanwhile, load / store controller 530 can also fetch weights and parameters from weight and parameter buffer 526 to weight register 528b based on instructions executed by operation controller 522. As described above, processor 514 can control neural network processor 502 to fetch weights and parameters from memory 512 or from another processor via a separate memory interface (not shown in FIG. 5) using DMA controller 516 and store the weights and parameters in weight and parameter buffer 526.

[0057] 6A, 6B, and 6C are charts illustrating example instructions executable by neural network processor 502. Referring to FIGS. 6A and 6B, instruction 600 includes multiple subinstructions, which may include subinstructions 602, 604, 606, 608, 610, 612, and 614. Each subinstruction may target a specific component of neural network processor 502, and different components may be programmed by the subinstructions and may perform the operations represented by the subinstructions in parallel and / or independently. For example, subinstruction 602 targets computation engine 524. Subinstruction 604 targets a portion of address generator 532, which generates memory addresses for accessing weights, input data, output data, and post-processing parameters such as bias, scale, and shift. Subinstruction 604 may also include a start loop indicator and may be targeted to computation controller 522. Sub-instruction 606 is directed to the portion of address generator 532 that generates memory addresses for accessing input and output data. Sub-instruction 608 is directed to load / store controller 530 to fetch weights and post-processing parameters (e.g., bias, scale, and shift) from weight and parameter buffer 526 to weight / parameter registers 528b. Sub-instruction 610 is directed to load / store controller 530 to fetch input data from memory 512 to data registers 528a and store output data from data registers 528a to memory 512. Sub-instruction 612 is directed to operation controller 522 and provides an end indicator for a loop or macro instruction.

[0058] Additionally, subinstruction 614 indicates the type of instruction 600. Neural network processor 502 can support instructions of different types and bit lengths. For example, in FIGS. 6A and 6B, instruction 600 is a 48-bit (48b) instruction having five subinstructions (602, 604, 606, 608, and 610) targeted to arithmetic engine 524, address generator 532, and load / store controller 530. Neural network processor 502 can also support 24-bit (24b) instructions having a subset of subinstructions 602-610, in addition to other subinstructions. Based on subinstruction 614, arithmetic controller 522 can determine the type and bit length of the instruction and decode and extract the subinstructions from the instruction based on the type and bit length.

[0059] The arithmetic controller 522 can extract a subinstruction from a predetermined bit position of the instruction and generate control signals for target components of the neural network processor 502 based on the extracted subinstruction. The arithmetic controller 522 can control the arithmetic engine 524, the address generator 532, and the load / store controller 530 to execute each subinstruction in parallel, thereby enabling the neural network processor 502 to provide N-way parallel programmability (e.g., 5-way execution of subinstructions 602-610).

[0060] Each of the subinstructions 602-610 includes fields that identify an operation to be performed by the target component and / or a register to be accessed. For example, subinstruction 602 includes fields 602a, 602b, 602c, and 602d. Field 602a may identify an operation to be performed by the computation engine 524, such as a multiply-accumulate (MAC) operation, a BNorm operation, or a max-pooling operation. Field 602b may identify a destination data register (labeled MACreg0-MACreg7) for storing the output of the operation. Fields 602c and 602d identify source input data registers (labeled Din0 and Din1 in FIGS. 6A and 6B) and weight registers (labeled weight-0 and weight-1 in FIGS. 6A and 6B), respectively, from which input data elements and weight elements are fetched for the operation. As described below, in some examples, field 602c may also indicate whether a max pooling operation is performed by the computation engine 524 or whether the max pooling operation provides a zero output.

[0061] Subinstruction 604 also includes fields 604a, 604b, and 604c. Field 604a may identify an operation to be performed to update the address stored in a source address register identified by field 604c, and the updated address is stored in a destination address register identified by field 604b. The operations may include, for example, an increment by one (ADD), a decrement by one (SUB), and a move operation (MOV) to replace the address in the destination address register with the address in the source address register. The address may be a memory address in memory 512 (for input data / output data) or an alias / reference / address to a location in weight and parameter buffer 526 (for weights or parameters). The source / destination address registers may include address registers for input and output data (labeled ARin0, ARin1, ARout0, and ARout1 in FIGS. 6A and 6B), address registers for weights (labeled ARwt0 and ARwt1 in FIGS. 6A and 6B), address registers for shifts and scales (labeled ARss0 and ARss1 in FIGS. 6A and 6B), and address registers for bias values ​​(labeled ARbias0 and ARbias1 in FIGS. 6A and 6B). In some examples, field 604a may also identify a setup loop operation (SETUP_LP), which may be a start indicator for a loop instruction, and field 604c may identify a loop count register LC-reg (which may be part of configuration registers 528d) that stores a loop count value for the loop, so that the subinstruction 604 can be processed by the operation controller 522.

[0062] Subinstruction 606 also includes fields 606a, 606b, and 606c. Field 606a can identify an operation to be performed to update an address stored in a source address register identified by field 606c, and the updated address is stored in a destination address register identified by field 606b. The operations can include, for example, an increment by one (ADD), a decrement by one (SUB), and a move operation (MOV) to replace the address in the destination address register with the address in the source address register. The addresses are memory addresses in memory 512 for input or output data, and the source / destination address registers can include address registers for input and output data (labeled ARin0, ARin1, ARout0, and ARout1 in FIGS. 6A and 6B). As described below, when in-place operation or circular addressing of input data is performed, address generator 532 can process the adjusted addresses using a circular buffer to provide circular addressing of the input / output data.

[0063] Sub-instruction 608 also includes fields 608a, 608b, and 608c. Field 608a may indicate a load instruction to fetch weight / bias / scale / shift from an address stored in an address register identified by field 608c (e.g., ARwt0, ARwt1, ARss0, ARss1, ARbias0, ARbias1) and store the fetched weight / bias / scale / shift in a register identified by field 608b (weight-0, weight-1, scale-shift-0, scale-shift-1, MACreg0 through MACreg7).

[0064] Sub-instruction 610 also includes fields 610a, 610b, and 610c. Field 610a may indicate whether sub-instruction 610 is a load or store instruction. In the case of a load instruction, field 610c may identify an address register (ARin0, ARin1) that stores a memory address from which input data is loaded, and field 610b may identify an input data register (Din0 or Din1) that stores the input data. In the case of a store instruction, field 610c may identify an address register (ARout0, ARout1) that stores a memory address for storing output data, and field 610b may identify an output data register (Dout) from which output data is fetched.

[0065] In some examples, the Din0, Din1, and Dout registers each have 32 bits, the weight register and scale shift register each have 64 bits, and each MACreg register has 72 bits.

[0066] 6C and 6D show examples of 24-bit instructions 630, 632, 634, and 636. Instruction 630 may be a 24b Type 1 instruction and includes subinstructions 602, 606, and 610. Instruction 632 may be a 24b Type 2 instruction and includes subinstruction 602 and subinstruction 608′, which includes the functionality of subinstruction 608 (to fetch weights and post-processing parameters from weight and parameter buffer 526 to weight / parameter register 528b) and auto-increment of the buffer address. Instruction 634 may be a 24b Type 3 instruction and includes subinstructions 640 for flow control (macro, halt) and debugging (breakpoint). Instruction 636 may be a 24b Type 4 instruction and includes subinstruction 604 and subinstruction 606. Each of the .24b instructions 630, 632, 634, and 634 includes a sub-instruction 614 (to indicate the type / bit length of the instruction) and a sub-instruction 612 (to support flow control).

[0067] Having neural network processor 502 configured to execute instructions of various bit lengths and various numbers of subinstructions provides flexibility for maximizing parallelism in the execution of subinstructions while reducing code size and power. Specifically, while neural network processor 502 does not always execute five subinstructions 602, 604, 606, 608, and 610 to support neural network operations, some combinations of subinstructions (e.g., subinstructions that read from memory and increment an address register) execute together more frequently than other combinations. Therefore, supporting 24-bit instructions with subinstructions that execute together more frequently can reduce code size. Power consumed in fetching and decoding shortened instructions can also be reduced. Meanwhile, supporting 48-bit instructions that include five subinstructions 602-610 can maximize parallelism in the execution of subinstructions. All of these can improve the flexibility of neural network processor 502 in supporting different applications with different requirements for power, code size, and execution parallelism.

[0068] FIG. 7 includes a chart 700 illustrating a set of instructions / microcode stored in instruction buffer 520. In FIG. 7, values ​​in column 702 may represent addresses of each instruction in instruction buffer 520. In the example of FIG. 7, neural network processor 502 may execute instructions sequentially according to the values ​​in column 702, and each value in column 702 may represent a time slot of execution. Also, column 706 represents subinstructions 610 executed by load / store controller 530 when fetching weights, column 704 represents subinstructions 608 executed by load / store controller 530 when fetching input data, column 708 represents subinstructions 606 executed by address generator 532 when updating memory addresses from which input data is fetched, column 710 represents subinstructions 604 executed by address generator 532 when updating memory addresses from which weights are fetched, and column 712 represents subinstructions 602 executed by arithmetic engine 524 when performing a MAC operation.

[0069] At time slot 0, the load / store controller 530 begins fetching input data at the first memory address of the memory 512 specified in the ARin0 register. The input data is stored in the input data register Din0. The load / store controller 530 also fetches a first set of weight elements at the first buffer address (of the weight and parameter buffer 526) specified in the ARwt0 register and stores the weights in the weight register Wt0. The address generator 532 also increments the first memory address in the ARin0 register to generate a second memory address and increments the first buffer address in the Wt0 register to generate the second buffer address. Fetching input data from the memory 512 can continue at time slot 1.

[0070] In time slot 2, the fetching of input data from memory 512 to input data register Din0 is completed. Load / store controller 530 also fetches a second set of weight elements at the second buffer address specified in ARwt0 register and stores the weights in weight register Wt1. Address generator 532 also increments the second buffer address in Wt0 register to generate a third buffer address. Arithmetic engine 524 also executes subinstruction 602 to perform a MAC operation between the input data stored in Din0 and a subset (Wt0L) of the weight elements in weight register Wt0, while load / store controller 530 fetches the weights into weight register Wt1.

[0071] At time slot 3, load / store controller 530 begins fetching input data at a second memory address in memory 512 specified in the ARin0 register. The input data is stored in input data register Din1. Address generator 532 also increments the second memory address in the ARin0 register to generate a third memory address. Computation engine 524 also executes subinstruction 602 to perform a MAC operation between the input data stored in Din0 and the remainder of the weight elements (Wt0H) in weight register Wt0. Load / store controller 530 suspends fetching weights from the third buffer address at time slot 3 because the weights in weight registers Wt0 and Wt1 are either in use (at time slot 3) or have not yet been used.

[0072] At time slot 4, fetching of input data at a second memory address in memory 512 continues. A MAC operation is completed between the input data stored in Din0 and the remainder of the weight elements in weight register Wt0. Thus, load / store controller 530 fetches a third set of weight elements from a third buffer address into weight register Wt0. Address generator 532 also increments the third buffer address to a fourth buffer address. Computation engine 524 also executes subinstruction 602 to perform a MAC operation between the input data stored in Din0 and a subset (Wt1L) of the weight elements in weight register Wt1.

[0073] The fetching of input data into Din1 is completed in time slot 5. However, load / store controller 530 suspends the fetching of input data from the third memory address into input data register Din0 because arithmetic engine 524 is still operating on the input data in Din0. In time slot 5, arithmetic engine 524 also executes subinstruction 602 to perform a MAC operation between the input data stored in Din0 and the remainder of the weight elements (Wt1H) in weight register Wt1.

[0074] In time slot 6, load / store controller 530 fetches the fourth set of weight elements from the fourth buffer address into weight register Wt1, while arithmetic engine 524 executes sub-instruction 602 to perform a MAC operation between the input data stored in Din0 and the first set of weights (Wt0L) stored in weight register Wt0 that load / store controller 530 updated in time slot 4. Address generator 532 also increments the fourth buffer address to a fifth buffer address. In time slot 7, arithmetic engine 524 also executes sub-instruction 602 to perform a MAC operation between the input data stored in Din0 and the remainder of the weight elements in weight register Wt0 (Wt0H).

[0075] In time slot 8, the arithmetic engine 524 executes subinstruction 602 to perform a MAC operation between the input data stored in Din0 and the weight elements (Wt1L and Wt1H) in the weight register Wt1, while the load / store controller 530 fetches the weight at the fifth buffer address into the weight register Wt0. The weight fetched into the weight register Wt0 in time slot 8 is used by the arithmetic engine 524 for the MAC operation with the input data stored in Din1. The address generator 532 also increments the fifth buffer address to generate a sixth buffer address.

[0076] In time slot 9, the arithmetic engine 524 also executes sub-instruction 602 to perform a MAC operation between the input data stored in Din0 and the remaining weight elements (Wt1H) in the weight register Wt1, completing the first set of MAC operations between Din0 and the weights. The results of the MAC operations are stored in the data registers MACreg0-MACreg7.

[0077] In time slot 10, the load / store controller 530 fetches the weight at the sixth buffer address into weight register Wt1, while the arithmetic engine 524 executes subinstruction 602 to perform a MAC operation between the input data stored in Din1 and a subset (Wt0L) of weight elements in weight register Wt0.

[0078] In time slot 11, load / store controller 530 fetches input data at a third memory address in memory 512 into register Din0. Address generator 532 also increments the third memory address to generate a fourth memory address, and arithmetic engine 524 executes subinstruction 602 to perform a MAC operation between the input data stored in Din1 and the remainder of the weight elements (Wt0H) in weight register Wt0.

[0079] 7, neural network processor 502 can execute instructions to maximize (or at least increase) parallelism between the fetching of weights and input data by load / store controller 530, the updating of memory and buffer addresses for fetching weights and input data by address generator 532, and computational operations in computation engine 524. Stalls in computation engine 524 due to delays in fetching weights and input data by load / store controller 530 can also be minimized (or at least reduced). For example, in timeslots 2 and 3, computation engine 524 performs operations on the weights in register Wt0 while load / store controller 530 updates the weights in register Wt1, allowing computation engine 524 to perform operations on the updated weights in register Wt1 in subsequent timeslots 4 and 5 without further delay and eliminating the need for computation engine 524 to stall waiting for the weights in register Wt1 to be updated. Also, input data is fetched from memory 512 into one register (e.g., Din1) while computation engine 524 operates on the input data stored in another register (e.g., Din0), so that computation engine 524 does not have to stall to wait for updated input data. Also, by spacing accesses to memory 512 by a relatively long time (e.g., by eight time slots in FIG. 7), memory access conflicts between processor 514 and neural network processor 502 can be avoided or at least reduced.

[0080] 8A and 8B are charts illustrating example memory management schemes that may be implemented by address generator 532. The example memory management schemes of Figures 8A and 8B may represent in-place operations in which output data overwrites input data and circular addressing of output data is implemented. In such a configuration, input and output data do not need to be stored in separate spaces within memory 512, thereby reducing memory resource usage by neural network processor 502.

[0081] Referring to chart 800, memory 512 can store 16 sets of input data elements (fm0), with each set of input data elements including four input data elements for four different channels (c0, c1, c2, and c3). Each row of chart 800 can be associated with a memory address, and the rows can be associated with consecutive addresses. However, space can be allocated to store 17 sets of input data elements, with row 16 (memory space 802) being empty and available to store one set of output data elements (fm1).

[0082] At time T0, the initial read address is at row 0, as indicated by pointer 804. A first set of input data elements in row 0 are fetched into an input data register (e.g., Din0) and then fetched to the arithmetic engine 524 for the arithmetic operation. An initial write address for the first set of output data elements of the arithmetic operation is associated with memory space 802, as indicated by pointer 806. Such a configuration can prevent the output data from overwriting the first set of input data elements, which may still be in use. The initial read and write addresses can be set using sub-instruction 606. After fetching the first set of input data elements, address generator 532 can increment the read address by 1 in response to another sub-instruction 606, so that pointer 804 can then point to row 1.

[0083] At time T1, a second set of input data elements in row 1 (fm0_01c0 through fm0_01c3) are fetched into the input data register, as indicated by pointer 804. After fetching the second set of input data elements, address generator 532 may increment the read address by one in response to sub-instruction 606, so that pointer 804 may then point to row 2. A first set of output data elements (fm1_00c0 through fm1_00c3) are stored in row 16, as indicated by pointer 806. After storing the first set of output data elements in row 16, address generator 532 may increment the write address by one in response to sub-instruction 606. However, to support in-place operations, circular addressing may be provided for the output data, so that the incremented write address may wrap around and point to row 0, so that the next set of output data elements can overwrite the first set of input data elements.

[0084] 8B , at time T2, the third set of input data elements in row 2 (fm0_02c0 through fm0_02c3) are fetched into the input data register, as indicated by pointer 804. After fetching the third set of input data elements, address generator 532 may increment the read address by one in response to sub-instruction 606, so that pointer 804 may then point to row 3. The second set of output data elements (fm1_00c0 through fm1_00c3) are stored in row 0, as indicated by pointer 806. After storing the second set of output data elements in row 0, address generator 532 may increment the write address by one in response to sub-instruction 606, so that pointer 806 may then point to row 1. Finally, at time T3, all output data elements are stored in rows 0 to 14 and 16, and pointer 806 points to row 14, which stores output data elements fm1_15_c0 to fm1_15_c3.

[0085] 9 includes a chart illustrating another example memory management scheme that may be implemented by address generator 532. The example memory management scheme of FIG. 9 may represent circular addressing of input data to support always-on applications, such as voice command recognition applications. For such applications, inference operations may be performed more frequently than the command length. Thus, there may be significant overlap in input data between two inference operations.

[0086] For example, for voice command recognition, input voice samples can be grouped into 40 millisecond (ms) frames, and eight frequency-domain features can be extracted per frame. Thus, for a command length of 1 second, features are extracted for 25 frames, resulting in input features with 25 positions and 8 channels per position. If an inference operation is performed every 120 ms, a new input feature processed by the inference operation can have three new feature sets (at three positions, with 8 channels per position) relative to the previous set of input features. The 22 overlapping sets of input features can be moved within memory to fit the input features within the same set of allocated memory addresses, but such data movement increases memory access latency and power consumption.

[0087] Instead of moving duplicate input data within memory 512, new input data may be stored in memory 512 (e.g., by sensor 104, by processor 514, etc.) according to a circular addressing scheme, and address generator 532 may also update the input data addresses to fetch the input data based on the same circular addressing scheme. Chart 900 in FIG. 9 illustrates an example of a circular addressing scheme. In chart 900, memory 512 may store 25 sets of input data elements, with each set of input data elements including eight input data elements for eight different channels (c0-c7). Each row of chart 900 may be associated with a memory address, and the rows may be associated with consecutive addresses.

[0088] At time T0, memory 512 stores an initial set of 25 input data elements (fm0_00×fm0_24 * ) The first address for fetching the 25 sets of input data elements to perform the first speculation operation is at row 0 indicated by pointer 902. Address generator 532 can provide the initial input data address as the address associated with row 0 and increment the input data address in response to subinstruction 606, and load / store controller 530 can fetch the 25 sets of input data elements from memory 512 from the input data address.

[0089] At time T1, a new set of three input data elements (fm0_25 * ~fm0_27 * ) is stored in memory 512. To avoid moving the rest of the initial set of input data elements, the new set of input data elements is stored in the memory addresses associated with rows 0, 1, and 2, and the input data element fm0_00 * ~fm0_02 * The first address for fetching the 25 sets of input data elements to perform the second speculation operation is at row 3, as indicated by pointer 902. Address generator 532 provides the initial input data address as the address associated with row 3, and can increment the input data address in response to subinstruction 606, allowing load / store controller 530 to fetch the 25 sets of input data elements from memory 512 from the input data address. However, to read a new set of input data elements, address generator 532 also implements a circular addressing scheme for the input data, and the input data address is incremented by one for input data elements fm024 and fm124. * ) wraps around after incrementing past the address associated with row 24, restarts at row 0, and ends at row 2.

[0090] At time T2, another new set of three input data elements (fm0_28* ~fm0_30 * ) is stored in memory 512. To avoid moving the rest of the initial set of input data elements, a new set of input data elements is stored in the memory addresses associated with rows 3, 4, and 5, and input data element fm0_03 * ~fm0_05 * The first address for fetching the 25 sets of input data elements to perform the third speculation operation is at row 6, as indicated by pointer 902. Address generator 532 provides the initial input data address as the address associated with row 6, and can increment the input data address in response to subinstruction 606, allowing load / store controller 530 to fetch the 25 sets of input data elements from memory 512 from the input data address. However, to read a new set of input data elements, address generator 532 also implements a circular addressing scheme for the input data, and the input data address is incremented by one for input data elements fm0_24 and fm1_24. * ) wraps around after incrementing past the address associated with row 24, restarts at row 0, and ends at row 5.

[0091] FIG. 10A illustrates an example of a circular buffer circuit 1000 that may be part of the address generator 532, while FIG. 10B presents a graph illustrating an example operation of the circular buffer 1000. Referring to FIG. 10B, the circular buffer circuit 1000 may receive an input / output data address ARw0. The circular buffer 1000 may forward ARw0 as an output address if circular addressing is disabled or if ARw0 is within the start address (start-addr in FIGS. 10A and 10B) and end address (end-addr in FIGS. 10A and 10B) of the memory region 1001 allocated for the input / output data. Referring to the left diagram, if circular addressing is enabled and ARw0 is below the start address by an offset (offset1) (by decrementing), the circular buffer circuit 1000 may perform a wraparound and provide a rotated version of ARw0 (ARw0cir0) by subtracting offset1 from the end address. Also, referring to the diagram on the right, if circular addressing is enabled and ARw0 exceeds (by increment) the starting address by an offset (offset2), circular buffer circuit 1000 can perform wraparound and provide a rotated version of ARw0 (ARw0_cir1) by subtracting offset2 from the starting address. On the other hand, if circular addressing is disabled or ARw0 is within memory region 1001, circular buffer circuit 1000 can forward ARw0 as the output address.

[0092] 10A, circular buffer circuit 1000 may include multiplexers 1002, 1004, 1006, and 1008, difference circuits 1012, 1014, and 1016, adder circuit 1020, incrementer circuit 1022, decrementer circuit 1024, and multiplexer control circuits 1030 and 1032. Circular buffer circuit 1000 has an address input 1040 coupled to an input / output data address register (e.g., ARin0 or ARout0) for receiving an input / output data address (ARw0).

[0093] Multiplexer 1002 can receive a starting input data address (“start-in” in FIG. 10 ) and a starting output data address (“start-out” in FIG. 10 ), and multiplexer 1004 can receive an ending input data address (“end-in” in FIG. 10 ) and an ending output data address (“end-out” in FIG. 10 ). The starting and ending input data addresses can define a memory region allocated for input data, and the starting and ending output data addresses can define a memory region allocated for output data, with the starting address being less than / than the ending address. Each of multiplexers 1002 and 1004 can also receive an indication M0 that indicates whether a circular addressing mode is provided for updating input data addresses (e.g., for always-on applications) or for output data addresses (e.g., for in-place operations). Indication M0 can also be received from configuration register 528c. If a circular addressing mode is provided to update the input data address, multiplexer 1002 may provide the starting input data address as the starting address (“start-addr” in FIG. 10 ), and multiplexer 1004 may provide the ending input data address as the ending address (“end-addr” in FIG. 10 ); otherwise, multiplexer 1002 may provide the starting output data address as the starting address, and multiplexer 1004 may provide the ending output data address as the ending address. Increment circuit 1022 has an input coupled to the output of multiplexer 1004 and may generate an incremented ending address 1060, and decrement circuit 1024 has an input coupled to the output of multiplexer 1002 and may generate a decremented starting address 1062.

[0094] The difference circuit 1012 has a first input coupled to the address input 1040 and a second input coupled to the output of the multiplexer 1002. The difference circuit 1014 has a first input coupled to the address input 1040 and a second input coupled to the output of the multiplexer 1004. The difference circuit 1012 can generate an offset 1050 between the starting address and ARwO, which also indicates whether ARwO is above or below the starting address. The difference circuit 1014 can generate an offset 1052 between the ending address and ARwO, which also indicates whether ARwO is above or below the starting address. The difference circuit 1016 can generate an address 1070 by subtracting the offset 1050 from the incremented ending address 1060, and the adder circuit 1020 can generate an address 1072 by adding the offset 1052 to the decremented ending address 1062. Address 1070 may be the wrap-around address ARw0_cir0 if ARw0 is below the starting address, and address 1072 may be the wrap-around address ARw0_cir1 if ARw0 is above the ending address.

[0095] Multiplexers 1006 and 1008 can selectively route one of input addresses ARw0, address 1070 (ARw0_cir0), or address 1072 (ARw0_cir1) to address output 1042. This selection is performed by multiplexer control circuits 1030 and 1032. Specifically, multiplexer control circuits 1030 and 1032 can each receive an indication of whether circular addressing is enabled ("circ enable" in FIG. 10), for example, from configuration register 528c. If circular addressing is disabled, or if ARw0 is greater than the starting address, or if ARw0 is less than the ending address, multiplexers 1006 and 1008 can connect address input 1040 to address output 1042 and provide as output the address stored in the input / output data address register. On the other hand, if the circular address is valid and ARw0 is less than the starting address, multiplexers 1006 and 1008 can provide ARw0_cir0 (address 1070) to address output 1042. If the circular address is valid and ARw0 is greater than the ending address, multiplexers 1006 and 1008 can provide ARw0_cir1 (address 1072) to address output 1042.

[0096] As described above, neural network processor 502 supports subinstructions containing flow control elements, such as loop elements and macro elements like those shown in FIG. 4, which enable compact representation of convolutional layer operations. FIG. 11 illustrates an example program containing a loop element executable by neural network processor 502. Referring to the top of FIG. 11, program 1100 includes instructions 1102, 1104, 1106, and 1108, each of which may include one or more subinstructions. Instruction 1102 may include subinstruction 604, which may have a field 604a identifying a setup loop operation (SETUP_LP) and a field 604c identifying a loop count register (LC_reg). Field 604a may include an opcode indicating the setup loop operation. Instruction 1102 is followed by instructions 1104 and 1106, each of which has a sub-instruction 612 that indicates a zero / deasserted end marker. Program 1100 also includes instruction 1108, which has a sub-instruction 612 that indicates an asserted end marker. Instructions 1102, 1104, 1106, and 1108 may define a loop operation that begins with instruction 1104 and ends with instruction 1108, where the loop operation is repeated a number of times defined in a loop count register identified by field 604c of instruction 1102. To implement the loop operation, the operation controller 522 can identify one or more pairs of sub-instructions including the opcode SETUP_LP and the asserted end marker by tracking the order / sequence in which the opcode SETUP_LP and the asserted end marker are extracted when the operation controller 522 analyzes the instruction, and store the program counter value of the pair of sub-instructions. After extracting the asserted end marker, the operation controller 522 can reset the program counter back to the count value of the sub-instruction having the paired SETUP_LP opcode, and repeat the execution of the instruction between the pair of the SETUP_LP sub-instruction and the asserted end marker.

[0097] 11 also shows program 1120, which illustrates a nested loop operation. Program 1120 includes instructions 1122, 1124, 1126, 1128, 1130, and 1132, each of which includes one or more sub-instructions. Instruction 1122 includes opcode SETUP_LP and identifies a first loop register LC-reg0 that stores a first loop count. Instruction 1124 includes opcode SETUP_LP and identifies a second loop register LC-reg1. Instruction 1130 includes an asserted end marker, and sub-instruction 1132 also includes an asserted end marker. Based on the order in which operation controller 522 extracts the SETUP_LP opcode and asserted end markers, operation controller 522 may pair the SETUP_LP of subinstruction 1122 with the asserted end marker of instruction 1132 and pair the SETUP_LP of subinstruction 1124 with the asserted end marker of instruction 1130. Operation controller 522 may repeat the execution of instructions 1124, 1126, and 1128 the number of times specified in LC-reg0. Also, upon extracting the asserted end marker of subinstruction 1130, operation controller 522 may reset the program counter to resume execution of subinstruction 1124 and repeat the execution of instructions 1126 and 1128 the number of times specified in LC-reg1, thereby implementing a nested loop operation.

[0098] FIG. 12 shows an example program 1200 including a macro element executable by the neural network processor 502. The program 1200 includes instructions 1201, 1202, 1204, 1206, and 1208, each including one or more subinstructions. Each instruction is also associated with a program counter value and has an asserted or deasserted end marker. The instruction 1206 includes fields 1206a, 1206b, and 1206c. The field 1206a may include a macro element opcode (MACRO) that identifies a macro including a set of instructions. The field 1206b may identify the program count value (e.g., PC#0) of the first instruction of the macro, and the last instruction of the macro may include an asserted end marker subinstruction 612. The field 1206c may indicate the number of times the macro is executed. Upon execution of instruction 1206, operation controller 522 resets the program counter to the count value in field 1206b and can execute all subsequent instructions until it receives instruction 1204 with an asserted end marker. Instruction 1204 may include other sub-instructions not shown in FIG. 12. Operation controller 522 can repeat execution of the macro the number of times specified in field 1206c. After executing instruction 1206, operation controller 522 can stop resetting the program counter and proceed to execute the subsequent instruction 1208.

[0099] 13 is a schematic diagram illustrating example internal components of the computation engine 524. Referring to FIG. 13, the computation engine 524 includes a MAC engine 1300 and a post-processing engine 1302. The MAC engine 1300 may receive input data elements and weight elements from input data registers (e.g., DIN0, DIN1) of the data registers 528a, weight registers (e.g., weight-0, weight-1) of the weight / parameter registers 528b, and scale shift registers (e.g., scale-shift-0, scale-shift-1) of the weight / parameter registers 528b, respectively. In response to a control signal 1301 from the computation controller 522, which generates the control signal in response to a subinstruction 602 indicating a MAC operation, the MAC engine 1300 may perform a MAC operation on the input data elements and weight elements. The MAC engine 1300 includes arithmetic circuits for performing MAC operations for neural network layers, such as CNN layers, DNN layers, and fully connected layers, and for storing intermediate output data in MAC registers (e.g., MACreg0-7) of the data registers 528a. The MAC engine 1300 can also fetch old partial sums from the MAC registers, update the old partial sums by adding the results of the MAC operations to the old partial sums, and store the updated partial sums back in the MAC registers. The arithmetic engine 524 also includes a MAC register multiplexer 1303 for selecting which MAC register is accessed by the MAC engine 1300 to perform the accumulation.

[0100] The post-processing engine 1302 can also receive intermediate output data elements from the MAC register. In response to a control signal 1305 from the operation controller 522, which generates a control signal in response to a subinstruction 602 indicating a BNorm operation, the post-processing engine 1302 can perform post-processing operations (e.g., BNorm and residual layer processing) on ​​the intermediate output data to generate output data and store the output data in an output data register (e.g., DOUT) of the data registers 528a. As described below, the post-processing engine 1302 can also perform a data packing operation on the output data register and, upon completing the data packing operation, send a signal to the load / store controller 530 to store the output data from the output data register back to the memory 512. Such a configuration can reduce accesses of the neural network process 502 to the memory 512 when writing back output data, thereby reducing memory usage and power consumption. The post-processing engine 1302 can also receive input data elements from the input data register and perform a residual mode operation based on the input data elements.

[0101] As described above, the computation engine 524 is configurable to perform MAC and post-processing operations on the weights and input data over a range of bit precisions (e.g., binary precision, ternary precision, 4-bit precision, 8-bit precision, etc.) based on parameters provided by the processor 514. The computation engine 524 includes a weight multiplexer 1304 and an input data multiplexer 1306 to support precision configurability. Specifically, depending on the input data and weight configuration 1310, the weight multiplexer 1304 can fetch all of the weight elements stored in a weight register (e.g., one of the weight-0 register or the weight-1 register) or half of the copies of the stored weight elements as packed data having a predetermined number of bits (e.g., 32 bits). Depending on the configuration 1311, the weight multiplexer 1304 can also perform processing on the weight elements to support various operations such as depthwise convolution and average pooling. In the case of a depthwise convolution, the weight multiplexer 1304 may select one of the 8-bit weights stored in the weight register, divide the 8-bit weights into groups (e.g., four groups) of weight elements, and pad each weight element group with zeros, so that the MAC engine 1300 can multiply the input data elements of a particular channel by zeros. Such a configuration may ensure that the intermediate output of a particular channel is based on a MAC operation on only the input data of that channel. As another example, in the case of an average pooling operation, the weight multiplexer 1304 may selectively forward zeros and ones as weight elements, such that input data elements paired with a weight element of one are represented in the intermediate output data elements, and input data elements paired with a weight element of zero are zeroed and not represented in the intermediate output data elements. The configuration 1311 may indicate a layer type, which may also indicate whether a depthwise convolution operation is performed. The configuration 1311 may also indicate whether an average pooling operation is performed. The configuration 1311 may be based on configuration data stored in configuration registers 528d and / or control signals from the operations controller 522 in response to commands.

[0102] Additionally, input data multiplexer 1306 can fetch all of the input data elements stored in an input data register (e.g., one of Din0 or Din1) or half of the stored input data elements as packed data having a predetermined number of bits (e.g., 32 bits). In some examples, input data and weight configuration 1310 can include an 8-bit mode (D8) or a 4-bit mode (D4), where D8 or D4 indicates whether computation engine 524 fetches input data in 8-bit or 4-bit format. Input data and weight configuration 1310 can be stored in and received from configuration register 528D. Whether computation engine 524 operates in D8 mode or D4 mode can depend on the input precision and weight precision, which can also determine the number of input data elements and weight elements fetched into MAC engine 1300 at a time (e.g., in one clock cycle).

[0103] The MAC engine 1300 and the post-processing engine 1302 may also receive an input precision and weight precision configuration 1312. Depending on the input data precision and weight precision, the arithmetic and logic circuits of the MAC engine 1300 and the post-processing engine 1302 may process operations differently. The post-processing engine 1302 also receives post-processing parameters 1314, which may define parameters for post-processing operations, some or all of which may depend on the input precision and / or weight precision. The input precision and weight precision configuration 1312 and some of the post-processing parameters 1314 may be received from configuration registers 528d. Some of the post-processing parameters 1314, such as shift and scale, may vary between different internal components of the post-processing engine 1302. These parameters may be fetched from the weight / parameter buffer 526 into weight / parameter registers 528b, and the post-processing engine 1302 may receive such parameters from the weight / parameter registers 528b.

[0104] The computation engine 524 may also include a max pooling engine 1320 for performing a max pooling operation on input data elements stored in the input data registers (DIN0, DIN1) and output data elements stored in the output data register (DOUT) and storing the result of the max pooling back in the output data register (DOUT). Specifically, the max pooling engine 1320 may overwrite the output data element in DOUT with the input data element in DIN0 / DIN1 if the input data element has a higher value than the output data element. The computation controller 522 may provide a control signal 1322 to the max pooling engine 1320 in response to, for example, a field 602c of the subinstruction 602 indicating the max pooling operation to be performed. The max pooling engine 1320 may then perform the max pooling operation in response to the control signal 1322. The max pooling engine 1320 may also receive post-processing parameters 1314 and configure the max pooling operation based on the parameters. In some examples, the max pooling engine 1320 can operate independently or in parallel with the MAC engine 1300 and post-processing engine 1302, thereby minimizing disruption of max pooling operations to operations by the rest of the computation engine 524 and improving efficiency.

[0105] FIG. 14 illustrates an example convolution operation performed by a MAC engine 1300. Referring to FIG. 14, the MAC engine 1300 may perform the convolution operation in a channel-major sequence. The MAC engine 1300 may first perform a convolution operation 1402 on input data elements 1404 associated with a first set of input channels at a particular height and a particular width (using a set of weights associated with a set of output channels) to generate a first partial sum of intermediate output data elements 1406. The input data elements 1404 may be stored in an input data register (e.g., Din0), and the intermediate output data elements 1406 may be stored in an intermediate output data register (e.g., MACreg0). The MAC engine 1300 may then perform a convolution operation 1412 on input data elements 1414 at the same height and width as the input data element 1404 but associated with a second set of input channels to generate a second partial sum for the intermediate output data element 1406, and add the second partial sum to the first partial sum. The input data element 1414 may be stored in another input data register (e.g., Din1). The MAC engine 1300 may perform additional convolutions on input data elements of other input channels at the same height and width, including input data element 1424, and accumulate the partial sums in intermediate output data registers to generate the intermediate output data element 1406. After the input data elements at a particular height and width position of all input channels have been processed, MAC engine 1300 can resume the convolution operation on input data elements 1404, 1414, and 1424 using another set of weights associated with a different set of output channels to calculate another set of intermediate output data elements, such as intermediate output data elements 1430 and 1432, associated with a different output channel.

[0106] Figures 15A, 15B, 15C, and 15D show example convolution and post-processing operations performed by MAC engine 1300 and post-processing engine 1302 using weight elements and input data elements of different bit precision. In Figure 15A, MAC engine 1300 can perform convolution operations between 256 8-bit input data elements (labeled in0, in1, in2, in3, ..., in254, and in255) and 8-bit weight elements to generate 24-bit intermediate output data elements, each generated by the sum of products of the input data elements and the weight elements. Post-processing engine 1302 performs post-processing operations, including BNorm and clamp / activation function operations (e.g., ReLU in Figure 14), on each intermediate output data element to generate an 8-bit output data element.

[0107] In FIG. 15B , MAC engine 1300 can perform convolution operations between 256 8-bit input data elements (labeled in0, in1, in2, in3, ..., in254, and in255) and 1-bit (binary) weight elements to generate 16-bit intermediate output data elements, each generated by the sum of multiplication products between the input data elements and the weight elements, followed by performing post-processing operations (e.g., BNorm and ReLU) on each intermediate output data element to generate an 8-bit output data element. Using binary weights (+1 or -1), MAC engine 1300 can generate multiplication products between input data elements and weight elements by simply forwarding the input data element if the weight element is 1 and forwarding the negative representation of the input data element if the weight element is -1, thereby reducing complexity and power and increasing the speed of operations. Weight storage can also be reduced by 1 / 8 compared to 8-bit weights. Although the precision of the intermediate output data elements is reduced (from 24 bits to 16 bits), the reduced precision may still be acceptable for certain applications, and the impact of the reduced precision can be further mitigated by having a neural network topology with mixed precision between the neural network layers.

[0108] In FIG. 15C , MAC engine 1300 can perform a convolution operation between an 8-bit input data element and a 2-bit ternary weight element (−1, 0, +1) to generate 16-bit intermediate output data elements, each generated by the sum of multiplication products between the input data element and the weight element, followed by performing post-processing operations (e.g., BNorm and ReLU) on each intermediate output data element to generate an 8-bit output data element. The 2-bit ternary weight element (which provides three alternative levels for representing the weight element) can be more accurate in representing the weight than the binary weight (which provides two alternative levels). MAC engine 1300 can also generate multiplication products between the input data element and the weight element by simply forwarding the input data element if the weight element is 1, forwarding a negative representation of the input data element if the weight element is −1, and forwarding zero if the weight element is zero, thereby reducing complexity and power and increasing computation speed.

[0109] In Figure 15D, MAC engine 1300 can perform a convolution operation between an 8-bit input data element and two sets of 2-bit ternary weight elements (-1, 0, +1) to generate 16-bit intermediate output data elements, where each intermediate output data element is generated by the sum of multiplication products between the input data element and the two sets of weight elements, and the intermediate output data elements generated from the two sets of weight elements are summed. The summed intermediate output data elements can then be post-processed (e.g., BNorm and ReLU) to generate an 8-bit output data element. Using two sets of 2-bit ternary weight elements (providing nine alternative levels) can be more accurate in representing weights than using a single set of 2-bit ternary weight elements (providing three alternative levels), providing similar benefits of simplified operation in MAC engine 1300 using ternary weight elements.

[0110] FIG. 16 illustrates example arithmetic operations performed by MAC engine 1300 to support the example convolution operation, for example, in the channel-first sequence of FIG. 14. The arithmetic operations shown in FIG. 16 may be performed in one clock cycle by MAC engine 1300. Referring to FIG. 16, MAC engine 1300 may receive four input values ​​X[0], X[1], X[2], and X[3] in each clock cycle, with each of the four input values ​​associated with a different input channel. MAC engine 1300 may also receive 16 weight values ​​associated with different input and output channels, such as W[0,0] (associated with output channel 0 and input channel 0) and W[1,3] (associated with output channel 1 and input channel 3). The example operation of FIG. 16 may be performed on 8-bit input data elements and 2-bit weight elements, or 4-bit input data elements and 4-bit weight elements. As described below, depending on the input precision and weight precision, the computation engine 524 can receive 8-bit input data elements or 4-bit input data elements. The computation engine 524 can also receive 4-bit weight elements, 2-bit weight elements, 8-bit weight elements, etc. The computation engine 524 can be configured to divide the weights / data and perform the example operations of FIG. 16 according to the input data and weight configuration 1310 and the precision configuration 1312. For example, the computation engine 524 can divide a 4-bit weight into two 2-bit weights and perform 8-bit data and 2-bit weight operations on two sets of weights, where each weight value can include a single 4-bit weight element, two 2-bit weight elements, or half an 8-bit weight element.

[0111] The MAC engine 1300 may generate partial sums for each of the intermediate output data elements Y[0], Y[1], Y[2], and Y[3], each associated with a different output channel, based on multiplication of the input data X[0]-X[3] by a particular output channel and a weight associated with the input channel of X[0]-X[3]. For example, for the intermediate output data element Y[0], the MAC engine 1300 may perform an accumulation operation by computing a multiplication product between W[0,3] and X[3], a multiplication product between W[0,2] and X[2], a multiplication product between W[0,1] and X[1], and a multiplication product between W[0,0] and X[0], and adding the multiplication products to a previous partial sum Y′ from one of the intermediate output data registers MACreg0-MACreg3 to generate a new partial sum Y, which may be stored in MACreg0-MACreg3 instead of the previous partial sum Y′. The addition may be a saturated addition. For the first instruction of a convolution operation, a bias value (e.g., Bias0) may be fetched from another set of intermediate output data registers MACreg4-MACreg7 via a multiplexer (labeled MUX in FIG. 16) into MAC engine 1300, and the multiplication product may be added to the bias value to generate the partial sum. Such a configuration may avoid performing additions in subsequent post-processing operations, thereby speeding up the post-processing operations. After the first instruction is executed, for subsequent instructions of the convolution operation, the multiplexer may route the outputs of MACreg0-MACreg3 to MAC engine 1300 to add the multiplication product to the previous partial sum.

[0112] The MAC engine 1330 can perform 16 multiplication operations, such as multiplication operation 1602, to generate a multiplication product in one clock cycle. In some examples, the MAC engine 1330 may include 16 multiplier circuits to perform the 16 multiplication operations in parallel. The MAC engine 1330 can also update the previous partial sum Y'[0] stored in the MAC register by adding the new partial sum to Y'[0].

[0113] Table 1 below shows the set of input precisions and weight precisions supported by the arithmetic engine 524. Each row also shows, for a given input precision and weight precision, the number of input data elements processed by the arithmetic engine 524 per clock cycle, the number of output data elements provided by the arithmetic engine 524 per clock cycle, and the number of multiply-accumulate (MAC) operations performed by the arithmetic engine 524 per clock cycle. The number of MACs per cycle may be equal to the product of the number of input data elements processed and the number of output data elements generated per clock cycle.

[0114] In some examples, as described below, the MAC engine 1330 may include an array of 32 4-bit by 2-bit multiplier circuits, each capable of performing a multiplication operation (or bitwise operation) between a 4-bit number and a 2-bit number per clock cycle, and the 32 multiplier circuits capable of performing 32 multiplications between 4-bit numbers and 2-bit numbers.

[0115] For 8-bit input data elements and 2-bit weight elements (binary or ternary), each input data element may be split into two 4-bit data values, and two multiplier circuits perform the operation on one 8-bit data element and one 2-bit weight element. Thus, an array of 32 multiplier circuits can perform 32 / 2 arithmetic operations on 8-bit input data and 2-bit weights per clock cycle.

[0116] For 8-bit input data elements and 4-bit weight elements, each input data element may be divided into two 4-bit data values, each 4-bit weight element may be divided into two 2-bit weight values, and four multiplier circuits perform an operation on one 8-bit data element and one 4-bit weight element. Thus, an array of 32 multiplier circuits can perform 32 / (2×2) operation of 8-bit input data and 4-bit weights per clock cycle.

[0117] For 8-bit input data elements and 8-bit weight elements, each input data element may be divided into two 4-bit data values, each weight element may be divided into four 2-bit weight values, and eight multiplier circuits perform an operation on one 8-bit data element and one 8-bit weight element. Thus, an array of 32 multiplier circuits can perform 32 / (2×4) arithmetic operations of 8-bit input data and 4-bit weights per clock cycle.

[0118] For 4-bit input data elements and 4-bit weight elements, each weight element may be divided into two 2-bit weight values, and two multiplier circuits perform the operation on one 4-bit data element and one 2-bit weight element. Thus, an array of 32 multiplier circuits can perform 32 / 2 arithmetic operations of 4-bit input data and 2-bit weights per clock cycle.

[0119] For binary weights and binary data, MAC engine 1330 (or arithmetic engine 524) can internally convert the binary weights to 2-bit weights. As described below, each multiplier circuit can perform a 4-bit arithmetic operation (e.g., XNOR) between the binary weights and the binary data, and an array of 32 multiplier circuits can perform 128 arithmetic operations of binary weights and binary data per clock cycle. [Table 1]

[0120] FIG. 17 illustrates the operation of an example multiplier circuit 1700 that may be part of MAC engine 1300. The example multiplier circuit 1700 may perform the multiplication operation shown in FIG. 16, such as multiplication operation 1602. The multiplier circuit 1700 may receive an 8-bit input value D[7:0], which may be one of X[0]-X[3] in FIG. 16, and a 4-bit weight value W[3:0], which may be one of the weight values ​​in FIG. 16 (e.g., W[0,0]). Each of the input value D and the weight value W may be signed or unsigned. The multiplier circuit 1700 may include two 4-bit by 2-bit multiplier circuits 1702 and 1704. In FIG. 17, the logical operations of the multiplier circuits 1702 and 1704 are represented in the form of instructions, which may be integrated into (and represent) logic circuits representing the multiplier circuits 1702 and 1704.

[0121] The multiplier circuit 1700 can receive a multiplier configuration 1710 to configure each of the multiplier circuits 1702 and 1704 to perform multiplication operations in various modes. The multiplier configuration 1710 can be part of the precision configuration 1312 and can include a first flag indicating whether D[7:4] is signed (e.g., D[7:4] is signed when the flag is asserted or represents a logical 1, and unsigned when the flag is deasserted or represents a logical 0), a second flag indicating whether D[3:0] is signed (both based on the input precision), a third flag indicating whether W[3:2] is signed, a fourth flag indicating whether W[1:0] is signed (both based on the weight precision), and a fifth flag indicating whether to operate in binary mode. Multiplier circuits 1702 and 1704 can operate in binary mode if precision configuration 1312 indicates that both the input data and the weights have 1-bit (binary) precision.

[0122] When the multiplier configuration 1710 indicates that the binary mode is disabled, the multiplier circuit 1702 can generate the output N0 as a 7-bit signed number by performing a multiplication operation between the four least significant bits D[3:0] and the two least significant bits W[1:0], and the multiplier circuit 1704 can generate the output N1 as a 7-bit signed number by performing a multiplication operation between the four most significant bits D[7:4] and the two most significant bits W[3:2]. Based on the multiplier configuration 1710, the multiplication operation for N0 can be a signed multiplication operation if at least one of D[3:0] or W[1:0] is signed, and the multiplication operation for N1 can be a signed multiplication operation if at least one of D[7:4] or W[3:2] is signed.

[0123] On the other hand, if the multiplier configuration 1710 indicates that the binary mode is enabled, the multiplier circuit 1702 performs a bitwise XNOR operation (e.g., 1710 in FIG. 17) between D[3:0] and W[3:0]. TIFF2025531347000004.tif6128) and summing the bitwise XNOR outputs to generate an 8-bit output N0. Also, multiplier circuit 1704 can perform a bitwise XNOR operation between D[7:4] and W[3:0] and summing the bitwise XNOR outputs to generate an 8-bit output N1. The truth table for the XNOR operation is shown below, where -1 is represented by a logical 0 and +1 is represented by a logical 1. As can be seen from the table, multiplication of two binary numbers represented in this way can be performed using the XNOR operation. [Table 2]

[0124] The MAC engine 1300 may include multiple arithmetic units, each including a multiplier circuit 1700 and a set of other logic circuits, for performing MAC operations to generate intermediate output data elements for a range of input precisions and weight precisions as listed in Table 1. FIG. 18 illustrates an example of an arithmetic unit 1800. Referring to FIG. 18, the arithmetic unit 1800 may have four arithmetic data inputs for receiving X0, X1, X2, and X3, respectively, as shown in FIG. 18. The arithmetic unit 1800 also has four arithmetic weight inputs for receiving W0, W1, W2, and W3, respectively, as shown in FIG. 18. The arithmetic unit 1800 also includes four multiplier circuits 1700, including 1700a, 1700b, 1700c, and 1700d. The input of the multiplier circuit 1700a is coupled to a first arithmetic data input and a first arithmetic weight input. The input of multiplier circuit 1700b is coupled to a second calculation data input and a second calculation weight input. The input of multiplier circuit 1700c is coupled to a third calculation data input and a third calculation weight input. The input of multiplier circuit 1700d is coupled to a fourth calculation data input and a fourth calculation weight input. As described above, each multiplier circuit 1700 generates two 7-bit outputs N0 and N1. Multiplier circuit 1700a generates 7-bit outputs N0a and N1a, multiplier circuit 1700b generates 7-bit outputs N0b and N1b, multiplier circuit 1700c generates 7-bit outputs N0c and N1c, and multiplier circuit 1700d generates 7-bit outputs N0d and N1d. Also, as described above, each multiplier circuit 1700a-d receives a multiplier configuration 1710.

[0125] The arithmetic unit 1800 also includes adders 1802a, 1802b, and 1802c and adders 1804a, 1804b, and 1804c. The adders 1802a, 1802b, and 1802c generate the sum of N1a, N1b, N1c, and N1d as MAC_L, which has 9 bits and is represented as a signed number when operating in non-binary mode. The adders 1804a, 1804b, and 1804c generate the sum of N0a, N0b, N0c, and N0d as MAC_R, which also has 9 bits and is represented as a signed number when operating in non-binary mode. The arithmetic unit 1800 also includes a bit shifter circuit 1806 that can perform a left shift of MAC_L by a number of bits specified in a shift control 1808 to generate MAC_L'. The arithmetic unit 1800 can receive the shift control 1808 from the configuration register 528c. As will be described below, the amount of the left bit shift is based on the input precision and the weight precision. The arithmetic unit 1800 can also perform a left shift of MAC_L' by a number of bits specified in a shift control 1808. R and MAC The sum of L' and MAC and an adder 1809 that generates a MAC out has 13 bits and can be represented as a signed number in non-binary mode as a MAC output. MAC_L and MAC_R can also be MAC outputs.

[0126] The arithmetic unit 1800 also includes an accumulator 1810, which calculates the MACreg as the old partial sum. in (18 bits) and updates the old partial sum by adding the MAC output to the old partial sum to create a new partial sum MACreg out. Figure 19 illustrates example internal components and their operation of accumulator 1810. In Figure 19, the logical operations of accumulator 1810 are represented in the form of instructions, which may be integrated into (and represent) logic circuits representing accumulator 1810.

[0127] 19, the accumulator 1810 may include two adders 1902 and 1904, each of which may be a 9-bit adder. If the MAC engine 1300 does not operate in binary mode, the adders 1902 and 1904 may form an 18-bit adder, which may be used to generate a 13-bit MAC. The adder 1904 first performs a 9-bit addition between the 9 LSBs of MAC_out and MACreg_in, and then updates the old partial sum (MACreg_in) by adding MAC_out to the 18-bit old partial sum. For the addition between the MSB of out and the MSB of MACreg_in, the carry (if any) may be propagated to adder 1902 to generate the 18-bit MACreg_out. The addition / summation may be a saturating addition and may be represented by the following equation: MACreg_out[17:0]=Saturate((MACreg_in[17:0]+MAC_out[12:0]),18bits) (Formula 2)

[0128] On the other hand, when MAC engine 1300 operates in binary mode, N0a, N0b, N0c, and N0d, and N1a, N1b, N1c, and N1d, can each have a maximum value of 4, and MAC_L and MAC_R can each have a maximum value of 16. Adders 1902 and 1904 can be two separate 9-bit adders. Adder 1902 can generate the 9 MSBs of MACreg_out by summing the 9 MSBs of MACreg_in and the 6 LSBs of MAC_L. Adder 1904 can generate the 9 LSBs of MACreg_out by summing the 9 LSBs of MACreg_in and the 6 LSBs of MAC_R. Because there is no carry from adder 1904 to adder 1902, adders 1902 and 1904 can perform additions in parallel, speeding up the update of partial sums.

[0129] 20A and 20B are schematic diagrams illustrating a MAC engine 1300 including the arithmetic unit 1800 of FIG. 18. Referring to FIGS. 20A and 20B, the MAC engine 1300 includes arithmetic units 1800a, 1800b, 1800c, and 1800d. The MAC engine 1300 has first, second, third, and fourth MAC data inputs for receiving 8-bit input data IN0, IN1, IN2, and IN3, respectively, from the input data multiplexer 1306. The first arithmetic data input of each arithmetic unit is coupled to the first MAC data input (for receiving IN0). The second arithmetic data input of each arithmetic unit is coupled to the second MAC data input (for receiving IN1). The third arithmetic data input of each arithmetic unit is coupled to the third MAC data input (for receiving IN2). The fourth arithmetic data input of each arithmetic unit is coupled to the fourth MAC data input (for receiving IN3). Input data multiplexer 1306 selectively forwards input data from one of the Din0 or Din1 input data registers and can forward all bits stored in the input data register (e.g., 32 bits including four 8-bit input data) or a half replica of the bits (16 bits) depending on D4 or D8 mode. For example, in D8 mode, input data multiplexer 1306 can forward X0 (associated with input channel 0) as IN0, X1 (associated with input channel 1) as IN1, X2 (associated with input channel 2) as IN2, and X3 (associated with input channel 3) as IN3. In D4 mode, input data multiplexer 1306 can forward a replica of the lower half (LSB) of the X2 bits as IN0, a replica of the upper half (MSB) of the X2 bits as IN1, a replica of the lower half of the X3 bits as IN2, and a replica of the upper half of the X3 bits as IN3. The input data multiplexer 1306 can also forward a copy of the lower half of the X0 bits as IN0, a copy of the upper half of the X0 bits as IN1, a copy of the lower half of the X1 bits as IN2, and a copy of the upper half of X1 as IN3.

[0130] MAC engine 1300 also includes first, second, third, and fourth groups of MAC weight inputs. Each computation unit is coupled to a respective group of first, second, third, and fourth MAC weight inputs, with the first computation weight input coupled to the first MAC weight input of the group, the second computation weight input coupled to the second MAC weight input of the group, the third computation weight input coupled to the third MAC weight input of the group, and the fourth computation weight input coupled to the fourth MAC weight input of the group. Each computation unit can receive four 4-bit weights from weight multiplexer 1304. For example, computation unit 1800a receives 4-bit weights W0a, W1a, W2a, and W3a at its first, second, third, and fourth computation weight inputs, respectively. Computation unit 1800b receives 4-bit weights W0b, W1b, W2b, and W3b at its first, second, third, and fourth computation weight inputs, respectively. Computation unit 1800c receives 4-bit weights W0c, W1c, W2c, and W3c at its first, second, third, and fourth computation weight inputs, respectively. Computation unit 1800d receives 4-bit weights W0d, W1d, W2d, and W3d at its first, second, third, and fourth computation weight inputs, respectively. In Figure 20, a first group of first, second, third, and fourth MAC weight inputs can receive W0a, W1a, W2a, and W3a, respectively. A second group of first, second, third, and fourth MAC weight inputs may receive W0b, W1b, W2b, and W3b, respectively. A third group of first, second, third, and fourth MAC weight inputs may receive W0c, W1c, W2c, and W3c, respectively. A fourth group of first, second, third, and fourth MAC weight inputs may receive W0d, W1d, W2d, and W3d, respectively.

[0131] Weight multiplexer 1304 can selectively transfer weights from one of the weight-0 register or the weight-1 register. In D4 mode, operation units 1800a and 1800b can receive the 32 upper half bits of a selected weight register, and operation units 1800c and 1800d can receive the 32 lower half bits of the selected register. In D8 mode, operation units 1800a and 1800b can receive half copies (16 bits) of the upper half bits of a selected weight register, and operation units 1800c and 1800d can receive half copies (16 bits) of the lower half bits of a selected weight register. Operation units 1800a-d can store intermediate output data elements in one MAC register, and each intermediate output data element can have 18 bits.

[0132] 20A and 20B also illustrate a data merge circuit 2002, which may be part of the MAC engine 1300 or the post-processing engine 1302. As described below, for a given input precision and weight precision, the outputs of some or all of the arithmetic units 1800a-d may be merged to generate a particular intermediate output data element. The data merge circuit 2002 includes bit shifters 2004a, 2004b, and 2006. The bit shifter 2004a is configured to perform a 2-bit left shift on the 18-bit output of the arithmetic unit 1800a, and the bit shifter 2004b is configured to perform a 2-bit left shift on the 18-bit output of the arithmetic unit 1800c. The data merge circuit 2002 also includes adders 2008a, 2008b, and 2030. Adder 2008a can add a left-shifted version of the output of arithmetic unit 1800a to the output of arithmetic unit 1800b to generate partial sum 2020a. Adder 2008b can add a left-shifted version of the output of arithmetic unit 1800c to the output of arithmetic unit 1800d to generate partial sum 2020b. Bit shifter 2006 is configured to perform a 4-bit left shift on partial sum 2020a. Data merge circuit 2002 also includes adder 2030 to generate partial sum 2032 by adding the left-shifted version of partial sum 2020a to partial sum 2020b. MAC register multiplexer 1303 receives the 18-bit outputs of arithmetic units 1800a-1800d and partial sums 2020a, 2020b, and 2032 and can select one of them to store in a MAC register based on the input precision and weight precision, as described below.

[0133] 21A-21F also illustrate different configurations of the arithmetic engines 1800a-d, data merge circuit 2002, weight multiplexer 1304, input data multiplexer 1303, and MAC register multiplexer 1303 to support MAC operations with different input precisions and weight precisions.

[0134] 21A-1 and 21A-2 illustrate an example configuration in which each input data element is an 8-bit number and each weight element is a 2-bit number. The input data multiplexer 1306 can operate in D8 mode and provide 32-bit input data including four 8-bit input data elements each cycle. The 2-bit weight elements can be ternary weights (-1, 0, 1) or can be extended from 1-bit binary weights (e.g., +1 is mapped to bit 01 and -1 is mapped to bit 11). The weight multiplexer 1304 can also operate in D8 mode and replicate half of the weight elements stored in the weight registers to provide 32-bit weights (including 16 x 2-bit weight elements). The computation unit 1800a can receive four 8-bit input data elements at its first, second, third, and fourth computation data inputs and four replicas of the 2-bit weight elements at its first, second, third, and fourth weight inputs. The computation unit 1800a may compute the partial sum for one of the intermediate output data elements, such as Y0, as follows: Y0=(W0×X0 H +W1×X1 H +W2×X2 H +W3×X3 H )<<4+(W0×X0 L +W1×X1 L +W2×X2 L +W3×X3 L ) (Formula 3)

[0135] In equation (3), W0, W1, W2, and W3 are each 2-bit weight elements. X0, X1, X2, and X3 are each 8-bit input data elements. X0 H , X1 H , X2 H , and X3 H are the 4 most significant bits (also represented as D[7:4] bits) of X0, X1, X2, and X3, respectively, while X0 L , X1 L , X2 L , and X3 Lare the four LSBs (also represented as D[3:0] bits) of X0, X1, X2, and X3, respectively. The multiplication circuit 1700a (of the arithmetic unit 1800a) multiplies W0 by X0 H Let N0a, and W0×X0 L is generated as N0b, and the multiplication circuit 1700b generates W1×X1 H Let N1a be W1×X1 L is generated as N1b, and the multiplication circuit 1700c generates W2 × X2 H Let N2a be W2×X2 L is generated as N2b, and the multiplication circuit 1700d generates W3 × X3 H Let N3a be W3×X3 L as N3b. Shift control 1808 can control bit shifter circuit 1806a to left shift MAC_L by 4 bits. MAC register multiplexer 1303 can selectively connect the output of accumulator 1810a to the MAC register, bypassing data merge circuit 2002.

[0136] 21B-1 and 21B-2 illustrate an example configuration in which each input data element is a 4-bit number and each weight element is a 4-bit number. The input data multiplexer 1306 can operate in D4 mode to provide 16-bit input data containing four 4-bit input data elements (e.g., by duplicating half of the bits in the Din0 / Din1 registers) each cycle. The weight multiplexer 1304 can also operate in D4 mode to provide all of the weight elements stored in the weight registers to provide 64-bit weights (containing 16 x 4-bit weight elements). The arithmetic unit 1800a can receive 4-bit copies of X0, X1, X2, and X3 (each represented as D[3:0] bits) at its first, second, third, and fourth arithmetic data inputs and four 4-bit weight elements at its first, second, third, and fourth arithmetic weight inputs. The calculation unit 1800a can calculate the partial sum for Y0 as follows: Y0=(W0 H ×X0+W1 H ×X1+W2H ×X2+W3 H ×X3)<<2+(W0 L ×X0+W1 L ×X1+W2 L ×X2+W3 L ×X3) (Formula 4)

[0137] In Equation 4, W0, W1, W2, and W3 are each 4-bit weight elements. X0, X1, X2, and X3 are each 4-bit input data elements. H , W1 H , W2 H , and W3 H are the two most significant bits of W0, W1, W2, and W3, respectively, while W0 L , W1 L , W2 L , and W3 L are the 2 LSBs of W0, W1, W2, and W3, respectively. Multiplication circuit 1700a receives X0 and X4 as 8-bit inputs and W0 as a 4-bit input (from input data multiplexer 1306 operating in D4 mode), and multiplies X0 × W0 H Let N0a be X4×W0 L The multiplication circuit 1700b generates X1×W1 as N0b. H Let N1a be X5×W1 L is generated as N1b, and the multiplication circuit 1700c generates X2 × W2 H Let N2a be X6×W2 L is generated as N2b, and the multiplication circuit 1700d generates X3 × W3 H Let N3a be X7×W3 L as N3b. Shift control 1808 can control bit shifter circuit 1806a to left shift MAC_L by 2 bits. MAC register multiplexer 1303 can selectively connect the output of accumulator 1810a to the MAC register, bypassing data merge circuit 2002.

[0138] In a subsequent cycle (not shown in Figures 21B-1 and 21B-2), the calculation unit 1800a can receive 4-bit copies of X4, X5, X6, and X7 and four 4-bit weight elements W4, W5, W6, and W7 at the first, second, third, and fourth calculation weight inputs, calculate partial sums based on Equation 2, and add the partial sums to Y0.

[0139] 21C-1 and 21C-2 illustrate an example configuration in which each input data element is a 4-bit number and each weight element is a 2-bit number. The input data multiplexer 1306 can operate in D8 mode and provide 32-bit input data including eight 4-bit input data elements in each cycle. The weight multiplexer 1304 can also operate in D4 mode and provide all of the stored weight elements in the weight register to provide a 64-bit weight (including 32 x 2-bit weight elements). The arithmetic unit 1800a can receive eight 2-bit weight elements along with two 2-bit weight elements at each of its first, second, third, and fourth weight inputs. The arithmetic unit 1800a can receive eight 4-bit input data elements along with two 4-bit input data elements at each of its first, second, third, and fourth input data inputs. The arithmetic unit 1800a can calculate the partial sum for Y0 as follows: Y0=(W0×X0+W2×X2+W4×X4+W6×X6)+(W1×X1+W3×X3+W5×X5+W7×X7) (Formula 5)

[0140] In Equation 5, W0, W1, W2, and W3 are each 2-bit weighting elements. X0, X1, X2, X3, X4, X5, X6, and X7 are each 4-bit input data elements (represented as D[3:0] bits). Multiplication circuit 1700a receives X0 and X1 as 8-bit inputs and A0 and A1 as 4-bit inputs, and generates A0 × X0 as N0a and A1 × X1 as N0b. Multiplication circuit 1700b generates A2 × X2 as N1a and A3 × X3 as N1b. Multiplication circuit 1700c generates A4 × X4 as N2a and A5 × X5 as N2b. Multiplication circuit 1700d generates A6 × X6 as N3a and A7 × X7 as N3b. The shift control 1808 can control the bit shifter circuit 1806a not to left-shift MAC_L. Therefore, the bit shifter circuit 1806a is omitted in Figures 21C-1 and 21C-2. The MAC register multiplexer 1303 can selectively connect the output of the accumulator 1810a to the MAC register, bypassing the data merge circuit 2002.

[0141] 21D-1 and 21D-2 illustrate an example configuration in which each input data element is an 8-bit number and each weight element is a 4-bit number. The input data multiplexer 1306 can operate in D8 mode and provide 32-bit input data including four 8-bit input data elements each cycle. The weight multiplexer 1304 can also operate in D8 mode and replicate half of the weight elements stored in the weight registers to provide 64-bit weights (including 8 x 4-bit weight element replications). Each of the arithmetic units 1800 can receive four sets of 2-bit replicas of the 4-bit weight elements and four 8-bit input data elements. The arithmetic units 1800a, 1800b, 1800c, and 1800d can compute partial sums for Y0 and Y1 as follows: Y0 H =(W0 H ×X0 H +W1 H ×1 H +W2 H ×X2H +W3 H ×X3 H )<<4+(W0 H ×X0 L +W1 H ×X1 L +W2 H ×X2 L +W3 H ×X3 L ) (Equation 6) Y0 L =(W0 L ×X0 H +W1 L ×X1 H +W2 L ×X2 H +W3 L ×X3 H )<<4+(W0 L ×X0 L +W1 L ×X1 L +W2 L ×X2 L +W3 L ×X3 L ) (Equation 7) Y0 = Y0 H <<2+Y0 L (Equation 8) Y1 H =(W4 H ×X0 H +W5 H ×X1 H +W6 H ×X2 H +W7 H ×X3 H )<<4+(W4 H ×X0 L +W5 H ×X1 L +W6 H ×X2 L +W7 H ×X3 L ) (Equation 9) Y1 L =(W4 L ×X0 H +W5 L ×X1 H +W6 L ×X2 H +W7 L ×X3 H )<<4+(W4L ×X0 L +W5 L ×1 L +W6 L ×X2 L +W7 L ×X3 L ) (Equation 10) Y1=Y1 H <<2+Y1 L (Formula 11)

[0142] In Equations 6 to 11, W0, W1, W2, W3, W4, W5, W6, and W7 are each 4-bit weighting elements. X0, X1, X2, and X3 are each 8-bit input data elements. W0 H , W1 H , W2 H , W3 H , W4 H , W5 H , W6 H , W7 H are the two most significant bits of W0, W1, W2, W3, W4, W5, W6, and W7, respectively, while W0 L , W1 L , W2 L , W3 L , W4 L , W5 L , W6 L , W7 L are the LSBs of the two bits of W0, W1, W2, W3, W4, W5, W6, and W7, respectively. H , X1 H , X2 H , and X3 H are the 4 most significant bits (also represented as D[7:4] bits) of X0, X1, X2, and X3, respectively, while X0 L , X1 L , X2 L , and X3 L are the LSBs of the 4 bits (also represented as D[3:0] bits) of X0, X1, X2, and X3, respectively.

[0143] A plurality of arithmetic units 1800 are H and Y0 LFor example, the first computation weight input of the computation unit 1800a is W0 H and the second computation weight input of computation unit 1800a can receive a copy of W1 H and the third computation weight input of computation unit 1800a can receive a copy of W2 H and the fourth computation weight input of computation unit 1800a can receive a copy of W3 H The first computation weight input of computation unit 1800b can receive a copy of W0 L and the second computation weight input of computation unit 1800b can receive a copy of W1 L and the third computation weight input of computation unit 1800b can receive a copy of W2 L and the fourth computation weight input of computation unit 1800b can receive a copy of W3 L You may receive a copy of the

[0144] Also, the first calculation weight input of the calculation unit 1800c is W4 H and the second computation weight input of computation unit 1800c can receive a copy of W5 H and the third computation weight input of computation unit 1800c can receive a copy of W6 H and the fourth computation weight input of computation unit 1800c can receive a copy of W7 H The first computation weight input of computation unit 1800d can receive a copy of W4 L and the second computation weight input of computation unit 1800d can receive a copy of W5 L and the third computation weight input of computation unit 1800d can receive a copy of W6 L and the fourth computation weight input of computation unit 1800d can receive a copy of W7 LA first operation data input of each of the operation units 1800a to 1800d can receive X0, a second operation data input of each of the operation units 1800a to 1800d can receive X1, a third operation data input of each of the operation units 1800a to 1800d can receive X2, and a fourth operation data input of each of the operation units 1800a to 1800d can receive X3.

[0145] The calculation unit 1800a is W0 H ×X0 H +W1 H ×1 H +W2 H ×X2 H +W3 H ×X3 H and W0 H ×X0 L +W1 H ×X1L+W2 H ×X2 L +W3 H ×X3 L The calculation unit 1800b can calculate W0 L ×X0 H +W1 L ×X1H+W2 L ×X2 H +W3 L ×X3 H and W0 L ×X0 L +W1 L ×1 L +W2 H ×X2 L +W3H×X3 L The bit shifters 1806A and 1806B of each arithmetic unit can perform a 4-bit left shift. H The partial sum of Y is stored in the first MAC register (e.g., MACreg0) and L The partial sum of W4 may be stored in a second MAC register (e.g., MACreg1). Also, in the same cycle, arithmetic unit 1800c stores W4 H ×X0 H +W5 H ×1H +W6 H ×X2 H +W7 H ×X3 H and W4 H ×X0 L +W5 H ×1 L +W6 H ×X2 L +W7 H ×X3 L Calculate Y1 H The calculation unit 1800d can generate a partial sum of W4 L ×X0 H +W5 L ×1 H +W6 L ×X2 H +W7 L ×X3 H and W4 L ×X0 L +W5 L ×1 L +W6 L ×X2 L +W7 L ×X3 L Calculate Y1 L It is possible to generate partial sums of Y1 H Partial sum of Y1 L The partial sums of Y0 and Y1 can be stored in a third MAC register (e.g., MACreg2) and a fourth MAC register (e.g., MACreg3). Thus, partial sums of two outputs (Y0 and Y1) can be generated every cycle.

[0146] 21E-1 and 21E-2 illustrate an example configuration in which each input data element is 8 bits and each weight element is 8 bits. The input data multiplexer 1306 can operate in D8 mode and provide 32-bit input data including four 8-bit input data elements each cycle. The weight multiplexer 1304 can also operate in D8 mode and replicate half of the weight elements stored in the weight registers to provide 64-bit weights (including four 8-bit weight element replicas). Each arithmetic unit 1800 can receive four sets of 2-bit replicas of the 8-bit weight elements and four 8-bit input data elements. The arithmetic units 1800a-1800d can compute partial sums for Y0 as follows: Y0 HH =(W0 HH ×X0 H +W1 HH ×1 H +W2 HH ×X2 H +W3 HH ×X3 H )<<4+(W0 HH ×X0 L +W1 HH ×1 L +W2 HH ×X2 L +W3 HH ×X3 L ) (Equation 12) Y0 HL =(W0 HL ×X0 H +W1 HL ×1 H +W2 HL ×X2 H +W3 HL ×X3 H )<<4+(W0 HL ×X0 L +W1 HL ×1 L +W2 HL ×X2 L +W3 HL ×X3 L ) (Equation 13) Y0 LH =(W0 LH ×X0 H +W1LH ×1 H +W2 LH ×X2 H +W3 LH ×X3 H )<<4+(W0 LH ×X0 L +W1 LH ×1 L +W2 LH ×X2 L +W3 LH ×X3 L ) (Equation 14) Y0 LL =(W0 LL ×X0 H +W1 LL ×1 H +W2 LL ×X2 H +W3 LL ×X3 H )<<4+(W0 LL ×X0 L +W1 LL ×1 L +W2 LL ×X2 L +W3 LL ×X3 L ) (Equation 15) Y0=Y0 HH <<6+Y0 HL <<4+Y0 LH <<2+Y0 LL (Formula 16)

[0147] In equations 12 to 16, W0 HH , W1 HH , W2 HH and W3 HH are bits [7:6] of W0, W1, W2 and W3, respectively, and W0 HL , W1 HL , W2 HL and W3 HL are bits [5:4] of W0, W1, W2 and W3, respectively, and W0 LH , W1 LH , W2 LH and W3 LH are bits [3:2] of W0, W1, W2, and W3, respectively, and W0 LL , W1 LL , W2LL and W3 LL are bits [1:0] of W0, W1, W2, and W3, respectively. H , X1 H , X2 H , and X3 H are the 4 most significant bits (also represented as D[7:4] bits) of X0, X1, X2, and X3, respectively, while X0 L , X1 L , X2 L , and X3 L are the LSBs (also represented as D[3:0] bits) of the 4 bits of X0, X1, X2, and X3, respectively. HH , Y0 HL , Y0 LH , Y0 LL are bits [7:6], bits [5:4], bits [3:2], and bits [1:0] of Y0, respectively.

[0148] A plurality of arithmetic units 1800 are HH , Y0 HL , Y0 LH , and Y0 LL For example, the first computation weight input of the computation unit 1800a is W0 HH and the second computation weight input of computation unit 1800a can receive a copy of W1 HH and the third computation weight input of computation unit 1800a can receive a copy of W2 HH and the fourth computation weight input of computation unit 1800a can receive a copy of W3 HH The first computation weight input of computation unit 1800b can receive a copy of W0 HL and the second computation weight input of computation unit 1800b can receive a copy of W1 HL and the third computation weight input of computation unit 1800b can receive a copy of W2 HL and the fourth computation weight input of computation unit 1800b can receive a copy of W3 HL You may receive a copy of the

[0149] Also, the first calculation weight input of the calculation unit 1800c is W0 LH and the second computation weight input of computation unit 1800c can receive a copy of W1 LH and the third computation weight input of computation unit 1800c can receive a copy of W2 LH and the fourth computation weight input of computation unit 1800c can receive a copy of W3 LH The first computation weight input of computation unit 1800d can receive a copy of W0 LL and the second computation weight input of computation unit 1800d can receive a copy of W1 LL and the third computation weight input of computation unit 1800d can receive a copy of W2 LL and the fourth computation weight input of computation unit 1800d can receive a copy of W3 LL A first operation data input of each of the operation units 1800a to 1800d can receive X0, a second operation data input of each of the operation units 1800a to 1800d can receive X1, a third operation data input of each of the operation units 1800a to 1800d can receive X2, and a fourth operation data input of each of the operation units 1800a to 1800d can receive X3.

[0150] The calculation unit 1800a is W0 HH ×X0 H +W1 HH ×1 H +W2 HH ×X2 H +W3 HH ×X3 H , and W0 HH ×X0 L +W1 HH ×1 L +W2 HH ×X2 L +W3 HH ×X3 L The calculation unit 1800b can calculate W0 HL ×X0 H +W1 HL ×1H +W2 HL ×X2 H +W3 HL ×X3 H , and W0 HL ×X0 L +W1 HL ×1 L +W2 HL ×X2 L +W3 HL ×X3 L The calculation unit 1800c can calculate W0 LH ×X0 H +W1 LH ×1 H +W2 LH ×X2 H +W3 LH ×X3 H , and W0 LH ×X0 L +W1 LH ×1 L +W2 LH ×X2 L +W3 LH ×X3 L The calculation unit 1800d can calculate W0 LL ×X0 H +W1 LL ×1 H +W2 LL ×X2 H +W3 LL ×X3 H , and W0 LL ×X0 L +W1 LL ×1 L +W2 LL ×X2 L +W3 LL ×X3 L The bit shifters 1806a to 1806d of each arithmetic unit can perform a 4-bit left shift. HH may be stored in a first MAC register (e.g., MACreg0), and Y0 HL The partial sum of Y0 may be stored in a second MAC register (e.g., MACreg1). LH The partial sum of Y0 may be stored in a third MAC register (e.g., MACreg2). LLThe partial sum of Y0 may be stored in a fourth MAC register (e.g., MACreg3). HH is calculated as a signed number, and Y0 HL , Y0 LH , and Y0 LL can be operated on as an unsigned number.

[0151] 21F, 21G, and 21H illustrate example configurations for supporting depthwise convolution operations. As described above, in a depthwise convolution operation, intermediate output data elements may be generated from input data elements of the same channel. To support a depthwise convolution operation, weight multiplexer 1304 can selectively forward zeros to certain multipliers assigned to perform multiplication operations on input data elements of different channels, so that the outputs of such multipliers are zero and do not contribute to the intermediate output data elements.

[0152] 21F-1 and 21F-2 illustrate example configurations for supporting depthwise convolution between 8-bit input data elements and 8-bit weight elements. To support depthwise convolution, the computation units 1800a-d receive non-zero weights (e.g., W0) for a particular channel from the weight multiplexer 1304. HH , W0 HL , W0 LH , W0 LL ) and zero weights for the other channels. In the configuration of FIG. 21F, the calculation units 1800a-d, together with the data merge circuit 2002, can calculate Y0 as follows: Y0 HH =(W0 HH ×X0 H )<<4+(W0 HH ×X0 L ) (Equation 17) Y0 HL =(W0 HL ×X0 H )<<4+(W0 HL ×X0 L ) (Equation 18) Y0 LH=(W0 LH ×X0 H )<<4+(W0 LH ×X0 L ) (Equation 19) Y0 LL =(W0 LL ×X0 H )<<4+(W0 LL ×X0 L ) (Equation 20) Y0=Y0 HH <<6+Y0 HL <<4+Y0 LH <<2+Y0 LL (Formula 21)

[0153] 21G-1 and 21G-2 illustrate example configurations for supporting depthwise convolution operations between 8-bit input data elements and 2-bit weight elements. In FIGS. 21G-1 and 21G-2, computation unit 1800a may be assigned to compute Y (of a first channel) and may receive only the non-zero weight (W) of the first channel at a first computation weight input and zero weights for the other channels at other computation weight inputs. Computation unit 1800b may be assigned to compute Y (of a second channel) and may receive only the non-zero weight (W) of the second channel at a second computation weight input and zero weights for the other channels at other computation weight inputs. Computation unit 1800c may be assigned to compute Y (of a third channel) and may receive only the non-zero weight (W) of the third channel at a third computation weight input and zero weights for the other channels at other computation weight inputs. Also, computation unit 1800d may be assigned to compute Y3 (for the fourth channel) and may receive only the non-zero weight (W3) for the fourth channel at its fourth computation weight input and zero weights for the other channels at its other computation weight inputs. In the configuration of Figure 21G, computation units 1800a-d may compute Y0, Y1, Y2, and Y3 as follows: Y0=(W0×X0 H )<<4+(W0×X0 L) (Equation 22) Y1=(W1×X1 H )<<4+(W1×X1 L ) (Formula 23) Y2=(W2×X2 H )<<4+(W2×X2 L ) (Formula 24) Y3=(W3×X2 H )<<4+(W3×X3 L ) (Formula 25)

[0154] 21H-1 and 21H-2 illustrate example configurations for supporting depthwise convolution operations between 8-bit input data elements and 4-bit weight elements. In FIG. 21H, computation units 1800a and 1800b can be assigned to compute Y (of a first channel), receiving only the non-zero weight (W) of the first channel at a first computation weight input and zero weights for the other channels at their other computation weight inputs. Computation units 1800c and 1800d can be assigned to compute Y (of a second channel), receiving only the non-zero weight (W) of the second channel at a second computation weight input and zero weights for the other channels at their other computation weight inputs. In the configuration of FIG. 21H, computation units 1800a-d, together with data merge circuit 2002, can compute Y and Y as follows: Y0 H =(W0 H ×X0 H )<<4+(W0 H ×X0 L ) (Formula 26) Y0 L =(W0 L ×X0 H )<<4+(W0 L ×X0 L ) (Formula 27) Y1 H =(W1 H ×1 H )<<4+(W1 H ×1 L ) (Formula 28) Y1 L =(W1 L×1 H )<<4+(W1 L ×1 L ) (Formula 29) Y0=Y0 H <<2+Y0 L (Formula 30) Y1=Y1 H <<2+Y1 L (Formula 31)

[0155] In a subsequent cycle (not shown), calculation units 1800a and 1800b can receive a non-zero weight (W2) for the third channel at the third calculation weight input and zero weights for the other channels at the other calculation weight inputs, and calculation units 1800c and 1800d can receive a non-zero weight (W3) for the fourth channel at the fourth calculation weight input and zero weights for the other channels at the other calculation weight inputs, and can calculate Y2 and Y3 as follows: Y2 H =(W2 H ×X2 H )<<4+(W2 H ×X2 L ) (Formula 32) Y2 L =(W2 L ×X2 H )<<4+(W2 L ×X2 L ) (Formula 33) Y3 H =(W3 H ×X3 H )<<4+(W3 H ×X3 L ) (Formula 34) Y3 L =(W3 L ×X3 H )<<4+(W3 L ×X3 L ) (Formula 35) Y2=Y2 H <<2+Y2 L (Formula 36) Y3=Y3 H <<2+Y3 L (Formula 37)

[0156] 22A, 22B, and 22C illustrate example internal components of MAC register multiplexer 1303, input data multiplexer 1306, and weight multiplexer 1304. FIG. 22A illustrates example internal components of MAC register multiplexer 1303. Referring to FIG. 22A, MAC register multiplexer 1303 may include a demultiplexer circuit 2200 and a multiplexer circuit 2202. Demultiplexer circuit 2200 is coupled between the outputs of the set of arithmetic units 1800 (multiple MACreg_out[17:0], such as four MACreg_out, eight MACreg_out, etc.) and the inputs of the MAC registers, and multiplexer circuit 2202 is coupled between the outputs of the MAC registers and the inputs of the set of arithmetic units 1800 (multiple MACreg_in[17:0]). The demultiplexer circuit 2200 may be controlled by the operation controller 522 (e.g., based on a subinstruction) to connect the outputs of the operation units 1800 to the inputs of the MAC registers to store the partial sums generated by the operation units 1800, or to store bias values ​​(Bias0, Bias1, Bias2, and Bias3) in some of the MAC registers. The bias values ​​may be added to the partial sums as part of a post-processing operation. The operation controller 522 may control the demultiplexer circuit 2200 to store the bias values ​​in the MAC registers in response to executing a first subinstruction of a neural network layer operation. By introducing the bias values ​​into the MAC registers as part of an initialization operation, latency caused by adding bias as part of a post-processing operation by the post-processing engine 1302 may be avoided. The multiplexer 2202 may also be controlled by the operation controller 522 based on a subinstruction to select which MAC register provides the previous partial sum for updating by a particular operation unit 1800.

[0157] 22B illustrates example internal components of input data multiplexer 1306. Referring to FIG. 22B, input data multiplexer 1306 may have inputs coupled to a first input data register (Din0) and a second input data register (Din1), and outputs coupled to MAC engine 1300 to provide four input data IN0, IN1, IN2, and IN3. Input data multiplexer 1306 also includes multiplexers 2210, 2212, a replica circuit 2214, and a multiplexer 2216. Multiplexers 2210, 2212, and 2216 are controlled by control signals M0, M1, and M2, some of which may be derived from input data and weight configuration 1310 and some of which may be provided by operation controller 522. For example, multiplexer 2210 can be controlled by control signal M0 to selectively provide data from one of registers Din0 or Din1 as IN0, IN1, IN2, and IN3. Multiplexer 2212 can select 8 bits from the upper 16 bits (bits 31-16) or the lower 16 bits (bits 15-0) of the selected register to duplicate the bits in D4 mode. Duplicator circuit 2214 generates two 8-bit values ​​by splitting the selected 8-bit value into two 4-bit values ​​and duplicating each 4-bit value. Multiplexer 2216 can be controlled by control signal M2, which is set based on D4 mode or D8 mode, to forward the selected 8-bit value as IN0, IN1, IN2, and IN3 (in D8 mode), or to forward 8-bit values ​​as IN0, IN1, IN2, and IN3, each containing a respective duplicated 4-bit value.

[0158] 22C illustrates example internal components of weight multiplexer 1304. Referring to FIG. 22C, weight multiplexer 1304 may have inputs coupled to a first weight register (weight-0 register) and a second weight register (weight-1 register), and outputs coupled to four groups of weight inputs of MAC engine 1300 to provide a first group of weights W0a, W1a, W2a, and W3a, a second group of weights W0b, W1b, W2b, and W3b, a third group of weights W0c, W1c, W2c, and W3c, and a fourth group of weights W0d, W1d, W2d, and W3d. Weight multiplexer 1304 also includes multiplexers 2220, 2222, a replication circuit 2224, and a multiplexer 2226. Multiplexers 2220, 2222, and 2226 are controlled by control signals M0, M1, and M2, some of which may be derived from input data and weight configuration 1310 and some of which may be provided by operation controller 522. For example, multiplexer 2210 may be controlled by control signal M0 to selectively provide 64-bit data from one of the weight-0 or weight-1 registers as a group of weights. In D8 mode, multiplexer 2222 can select 16 bits from the upper 32 bits (one of Wt1H or Wt0H) or the lower 32 bits (one of Wt1L or Wt0L) of the selected register to duplicate the bits. Duplicator circuit 2224 generates two 32-bit values ​​by splitting the selected 32-bit value into two 16-bit values ​​and duplicating each 16-bit value. The multiplexer 2226 can be controlled by a control signal M2 that is set based on the D4 or D8 mode to transfer a 64-bit value from a selected register as a group of weights in D4 mode, or to transfer two 32-bit values ​​from a replica of a 16-bit value as a group of weights in D8 mode.

[0159] In addition, weight multiplexer 1304 also includes depth-wise and average pooling logic 2228. When enabled by a depth-wise convolution enable signal (e.g., from configuration 1311), logic 2228 can select one of 8 bits from one of the 64-bit weight-0 or weight-1 registers, split the 8 bits into four groups of 2 bits, pad the weight groups with zeros, and send the weight groups to computation units 1800a, 1800b, 1800c, and 1800d. Weight multiplexer 1304 can also pad different weights in each group with zeros, as shown in Figures 21F-21H, depending on the input data and the weight precision from weight configuration 1310. Additionally, when enabled by an average pooling enable signal (e.g., from configuration 1311), weight multiplexer 1304 can provide copies of weight elements W0, W1, W2, and W3 in groups of first, second, third, and fourth weight elements, as in Figures 21G-1 and 21G-2, where each of W0, W1, W2, and W3 is equal to logic 1.

[0160] FIG. 23 illustrates exemplary internal components of the post-processing engine 1302. Referring to FIG. 23, the post-processing engine 1302 includes a post-processing circuit 2302 and a data packing circuit 2304. The post-processing circuit 2302 can receive intermediate output data of different channels, such as Y0, Y1, Y2, and Y3, from the MAC registers. The post-processing circuit 2302 includes circuits, such as arithmetic circuits and various logic circuits, to perform post-processing operations (e.g., BatchNorm, residual layer processing, etc.) on the intermediate output data in response to control signals 1305 from the operation controller 522, which generates control signals in response to sub-instructions 602 indicating the post-processing operations. The processing circuit 2302 also receives a precision configuration 1312 indicating the input precision, output precision, and weight precision, and post-processing parameters 1314 defining various parameters of the post-processing operations, including the output data bit size / precision, and configures the post-processing operations based on the precision configuration 1312 and the post-processing parameters 1314. In some examples, the processing circuit 2302 can perform multiple post-processing operations in parallel on the intermediate output data Y0, Y1, Y2, and Y3 to generate the output data Out0, Out1, Out2, and Out3 in one clock cycle. In some examples, the processing circuit 2302 can include the data merging circuit 2002 of Figures 20A and 20B to provide intermediate output data elements for different input precisions and weight precisions, as described above.

[0161] The data packing circuit 2304 can store output data, such as Out0, Out1, Out2, and Out3, in a DOUT register. In some examples, the DOUT register has 32 bits. The data packing circuit 2304 is configured to store the output data in specific bit positions of the DOUT register, and in response to the DOUT register being filled with the output data (e.g., 32 bits of output data being stored), send a control signal 2305 to the load / store controller 530 to fetch the output data from the DOUT register back to the memory 512, thereby overwriting the DOUT register with the new output data.

[0162] Specifically, for a certain set of input precision, output precision, and weight precision, the data packing circuit 2304 may generate four 8-bit output data elements Out0, Out1, Out2, and Out3 in one clock cycle. The data packing circuit 2304 may store the four 8-bit output data elements in the DOUT register after one clock cycle and then send a control signal 2305 to the load / store controller 530. For a different set of input, output, and weight precision, the processing circuit 2302 may generate 16-bit output data per clock cycle, such as four 4-bit output data elements or two 8-bit output data elements. In such a case, the processing circuit 2302 may store a first set of 16-bit output data in first bit positions of the DOUT register after a first clock cycle, and then store a second set of 16-bit output data in second bit positions of the DOUT register after a second clock cycle. Such a configuration allows load / store controller 530 to send output data in chunks of a particular number of bits (e.g., 32 bits) that can be optimized for write operations of memory 512. Such a configuration can reduce accesses by neural network process 502 to memory 512 when writing back output data, reducing memory usage and power consumption.

[0163] In some examples, the data packing circuit 2304 may also receive a control signal 2306 from the operation controller 522 and send the control signal 2305 in response to the control signal 2306. The operation controller 522 may generate the control signal 2306 in response to an instruction from the instruction buffer 520.

[0164] 24 illustrates example internal components of processing circuit 2302. Referring to FIG. 24, processing circuit 2302 may include a series of processing circuits coupled between a data input for receiving intermediate output data elements and a data output for providing an output data element. The processing circuits may include a multiplier circuit 2402, a bit shifter 2404, a saturation circuit 2406, an adder 2408, and a clamp circuit 2410, each of which may include arithmetic circuits (e.g., adders and multipliers) and / or logic circuits (e.g., mapping tables implemented with multiplexers and demultiplexers for the bit shifter circuit, saturation circuit, and clamp circuit) to process the intermediate output data elements and provide an output data element. For example, processing circuit 2302 includes a multiplier circuit 2402a, a bit shifter 2404a, a saturation circuit 2406a, an adder 2408a, and a clamp circuit 2410a coupled between a first input of processing circuit 2302 (e.g., for receiving Y0) and a first output of processing circuit 2302 (e.g., for providing Out0), and a multiplier circuit 2402b, a bit shifter 2404b, a saturation circuit 2406b, an adder 2408b, and a clamp circuit 2410a coupled between a second input of processing circuit 2302 (e.g., for receiving Y1) and a second output of processing circuit 2302 (e.g., for providing Out1). 10b, and may include a multiplier circuit 2402c, a bit shifter 2404c, a saturation circuit 2406c, an adder 2408c, and a clamp circuit 2410c coupled between a third input of processing circuit 2302 (e.g., for receiving Y2) and a third output of processing circuit 2302 (e.g., for providing Out2), and a multiplier circuit 2402d, a bit shifter 2404d, a saturation circuit 2406d, an adder 2408d, and a clamp circuit 2410d coupled between a fourth input of processing circuit 2302 (e.g., for receiving Y3) and a fourth output of processing circuit 2302 (e.g., for providing Out3).

[0165] Processing circuit 2302 also includes data routing circuit 2412 coupled between the input of processing circuit 2302 and the inputs of multiplier circuits 2402a-d, and data routing circuit 2414 coupled between the outputs of clamp circuits 2410a-d and the output of processing circuit 2302. As described below, based on the input precision, output precision, and weight precision, data routing circuit 2412 can route intermediate output data elements to the multiplier circuits, and data routing circuit 2414 can route the outputs of the clamp circuits to the output of processing circuit 2302. Further processing circuit 2302 includes multiplexer circuit 2416a coupled between the output of adder 2408a and the input of adder 2408b, and multiplexer circuit 2416c coupled between the output of adder 2408c and the input of adder 2404d. Multiplexer circuit 2416 can be controlled based on the weight precision.

[0166] Each multiplier circuit 2402 is coupled to a respective input of 2302 for scaling the intermediate output data element by a scaling factor (labeled scale[0], scale[1], scale[2], and scale[3] in FIG. 24). The input of a bit shifter 2404 is coupled to the output of the multiplier for right shifting the scaled intermediate output data element by a number of bits specified by a shift parameter (labeled shift[0], shift[1], shift[2], and shift[3] in FIG. 24). The input of a saturation circuit 2406 is coupled to the output of the bit shifter. The saturation circuit 2406 is configured to scale the intermediate output data element by a number of bits specified by a shift parameter (labeled shift[0], shift[1], shift[2], and shift[3] in FIG. 24). 15 and (2 15 -1), the scaled and right-shifted intermediate output data element can be forwarded; otherwise, the data element is -2 15 If it is less than the lower limit -2 15 and the data element is (2 15 -1) if it exceeds the upper limit (2 15The adder 2408a may output a scaled, right-shifted, and limited intermediate output data element (-1). In some examples, the upper and lower limits of the saturation circuit 2406 may be fixed / hardwired and not programmable. The adder 2408a may add a value to the scaled, right-shifted, and limited intermediate output data element. The added value may be from the residual layer output (labeled in[0], in[1], in[2], and in[3]), which may be fetched from the input data registers DIN0 / DIN1 to provide residual layer processing. In some examples, the added value may also be from the output of another adder circuit 2408 in the processing circuit 2302 based on control over the multiplexer circuit 2416. The output of the adder 2408a is coupled to the input of a clamp circuit 2410. The clamp circuit 2410 may implement a clamp function, which may be based on a rectified linear unit (ReLU) of the activation function. The clamp function may also clamp / limit the output between an upper value (labeled clamp_high) and a lower value (labeled clamp_low). The scale, shift, clamp_high, and clamp_low values ​​and connections may be defined in precision configuration 1312 and / or post-processing parameters 1314. For example, different clamp_high and clamp_low values ​​may be set based on the output data precision as follows: [Table 3]

[0167] For each intermediate output data element Y_n (e.g., Y0, Y1, Y2, and Y3), the processing circuit 2302 may generate an output data element Out_n (e.g., Out0, Out1, Out2, and Out3) based on the following equation: Scaling and shifting may be different for different channels. Also, as described above in FIG. 22, a bias value bias[n] may be introduced into the MAC register to initialize the intermediate output data elements. Out n=clamp(Scale[n]×(Y_n+bias[n])>>shift[n], clamp_high, clamp_low) (Equation 38)

[0168] 24, scale values ​​(e.g., scale[0] through scale[3]) and shift values ​​(e.g., shift[0] through shift[3]) may vary between different neural network layers and may be received from weight / parameter register 529b (as well as weight and parameter buffer 526). Clamp_high and clamp_low values ​​may be relatively static between different neural network layers and may be received from configuration register 528d.

[0169] Figures 25A, 25B, and 25C are schematic diagrams illustrating different configurations of processing circuitry 2302 to support post-processing operations for different input precisions and weight precisions. For simplicity, data routing circuits 2412 and 2414 and multiplexer circuits 2416a-c are omitted, but their data routing operations are shown in Figures 25A-C.

[0170] Figure 25A is an example configuration of processing circuit 2302 for supporting 8-bit data and 2-ternary (2T) weights, also shown in Figure 15D. In Figure 25A, data routing circuit 2412 can connect a first input to multiplier circuit 2402a, a second input to multiplier circuit 2402b, a third input to multiplier circuit 2402c, and a fourth input to multiplier circuit 2402d. Also, to support 2T weight operations, multiplexer circuit 2416a can route the output of adder 2408a to the input of adder 2408b, so that adder 2408b adds the output of saturation circuit 2406a (for Y0) and the output of saturation circuit 2406b (for Y1) and provides the sum to clamp circuit 2410b. Additionally, multiplexer circuit 2416c can route the output of adder 2408c to the input of adder 2408d, so that adder 2408d adds the output of saturation circuit 2406c (for Y2) and the output of saturation circuit 2406d (for Y3) and provides the sum to clamp circuit 2410d. Data routing circuit 2414 can route the output of clamp circuit 2410b to its first and third outputs to provide replicated Out0 and Out2, and route the output of clamp circuit 2410d to its second and fourth outputs to provide replicated Out1 and Out3. In some examples, the data packing circuit 2304 can store one of Out0 or Out2 and one of Out1 or Out3 generated by the processing circuit 2302 in a first clock cycle in a first bit position of the DOUT register, then store one of Out0 or Out2 and one of Out1 or Out3 generated by the processing circuit 2302 in a second clock cycle in a second bit position of the DOUT register, and then send a control signal 2305 to start fetching output data from the DOUT register to memory 512. In some examples, the data packing circuit 2304 can also store Out0, Out2, Out1, and Out3 generated by the processing circuit 2302 in a clock cycle and send a control signal 2305 to start fetching output data from the DOUT register to memory 512.

[0171] Figures 25B and 25C illustrate an example configuration of processing circuit 2302 including a data merging circuit 2002, which may be part of data routing circuit 2412. In Figure 25B, data merging circuit 2002 is configured to support 8-bit input data and 4-bit weights, as in Figures 21D-1 and 21D-2. Data routing circuit 2412 may route the output of adder 2008a to the inputs of multiplier circuits 2402a and 2402c, and the output of adder 2008b to the inputs of multiplier circuits 2402b and 2402d. Processing circuit 2302 may generate Out0 and Out2 as duplicates, and generate Out1 and Out3 as duplicates. The data packing circuit 2304 may store Out0 and Out2, Out1 and Out3 in the DOUT register as duplicate data generated in one clock cycle, or may store different sets of Out0 / Out2 and Out1 / Out3 from two different clock cycles in the DOUT register, as described above.

[0172] Also in FIG. 25C, the data merging circuit 2002 is configured to support 8-bit input data and 8-bit weights, as in FIG. 21E. The data routing circuit 2412 can route the output of the adder 2008a to the inputs of the multiplier circuits 2402a and 2402c, and route the output of the adder 2030 to the inputs of the multiplier circuits 2402a-d. The processing circuit 2302 can generate Out0, Out1, Out2, and Out4(3?) as duplicates. The data packing circuit 2304 can store Out0-Out3 in the DOUT register as duplicate data generated in one clock cycle, as described above, or can store different Out0 / Out1 / Out2 / Out3 from four different clock cycles in the DOUT register.

[0173] 26A-1, 26A-2, and 26B illustrate examples of the internal components and their operation of the max pooling engine 1320. The max pooling engine 1320 compares input data elements in one of the DIN0 or DIN1 input data registers with output data elements in the DOUT register and may replace a particular output data element if the particular output data element in the DOUT register has a lower value than the input data element being compared. The max pooling engine 1320 may be bypassed so as not to overwrite the DOUT register or to output zero as the result of the max pooling operation.

[0174] 26A-1 and 26A-2, the max pooling engine 1320 may include a multiplexer 2602, a comparison circuit element 2604, and a multiplexer 2606. The multiplexer 2602 may be controlled (e.g., based on the post-processing parameters 1314) to select which of the input data registers DIN0 or DIN1 performs the max pooling operation. The comparison circuit element 2604 can receive configuration data via terminals cnfg-hi[1:0] and cnfg-lo[1:0] indicating, for example, whether to perform a 4-bit or 8-bit comparison, whether the comparison is signed or unsigned, etc., and compare the input data (from one of DIN0 or DIN1) and the output data (at DOUT) in 8-bit chunks (e.g., between DIN[31:24] and DOUT[31:24], between DIN[23:16] and DOUT[23:16], between DIN[15:8] and DOUT[15:8], and between DIN[7:0] and DOUT[7:0]) based on the configuration data. The configuration data can be part of post-processing parameters 1314 from configuration register 528d.

[0175] For each 8-bit chunk, a comparison between the 4 MSBs may be performed by an MSB comparison circuit such as comparison circuit 2604a (also labeled CMP-hi in FIGS. 26A and 26B), and a comparison between the 4 LSBs may be performed by an LSB comparison circuit such as comparison circuit 2604b (also labeled CMP-lo in FIGS. 26A and 26B). The CMP-hi circuit may also send the comparison result of the 4 MSBs to the CMP-lo circuit to ensure that, for an 8-bit comparison, when the MSB of DOUT has the same value as or greater than the MSB of DIN but the LSB of DOUT has a value smaller than the LSB of DIN, the CMP-lo circuit does not indicate that DOUT is smaller than DIN. Depending on the result of the comparison, multiplexers (e.g., multiplexers 2604c and 2604d) may overwrite the 4 MSBs and / or the 4 LSBs of the 8-bit chunk of output data with the input data. In the case of an 8-bit comparison, the multiplexer overwrites the 8-bit DOUT data with the 8-bit DIN data only if the entire 8-bit DOUT data has a lower value than the 8-bit DIN data. The multiplexer 2606 can also provide either an 8-bit chunk of DOUT data or zeros. In the case of a 4-bit comparison, the multiplexer can compare the 4-bit DOUT data with the 4-bit DIN data and replace the 4-bit DOUT data if the 4-bit DIN data exceeds the 4-bit DOUT data, without considering the comparison result of the other 4-bit DIN / DOUT data.

[0176] Figure 26B is charts 2610 and 2612 illustrating example operations of the CMP-hi and CMP-lo circuits. In Figure 26B, charts 2610 and 2612 illustrate the logical operations of the CMP-hi and CMP-lo circuits, respectively, which are expressed in the form of instructions that may be integrated into (and represent) the logic circuits that represent the CMP-hi and CMP-lo circuits.

[0177] Referring to chart 2610, a CMP-hi circuit (e.g., compare circuit 2604a) can receive configuration data such as whether the 4 MSBs of the 8-bit chunks of DIN and DOUT data being compared are signed or unsigned data. If the data is signed, CMP-hi can sign-extend both the 4 MSBs of the DIN data and the 4 MSBs of the DOUT data; if not, it can prepend them with zeros (by adding zeros before the MSBs).

[0178] If the four MSBs of the DIN data have a value greater than the four MSBs of the DOUT data, the CMP-hi circuit can set a control signal to cause a multiplexer (e.g., multiplexer circuit 2604c) to overwrite the four MSBs of the DOUT data with the four LSBs of the DIN data. The CMP-hi circuit can also provide a first comparison signal (set cmp-res="GT") to the CMP-lo circuit. On the other hand, if the four MSBs of the DIN data and the four MSBs of the DOUT data are the same, the CMP-hi circuit can provide a second comparison signal (set cmp-res="EQ"). The first and second comparison signals are provided to the CMP-lo circuit to avoid making an erroneous comparison decision that the MSB of DOUT has a higher value than DIN but the LSB of DOUT has a lower value than DIN. In both cases where the four MSBs of the DIN data have the same value as the DOUT data or have a lower value, the CMP-hi circuit can maintain the four MSBs of the DOUT data in the DOUT register.

[0179] Also, referring to chart 2612, a CMP-lo circuit (e.g., compare circuit 2604b) can receive configuration data such as whether the 4 LSBs of the 8-bit chunks of DIN and DOUT data being compared are signed or unsigned data, and whether an 8-bit or 4-bit comparison is performed. If the data is signed, CMP-lo can sign-extend both the 4 LSBs of the DIN data and the 4 LSBs of the DOUT data; otherwise, it can prepend them with zeros. Also, if a 4-bit comparison is performed, the CMP-lo circuit can ignore the control signal from the CMP-hi circuit and overwrite the 4 LSBs of the DOUT data with the 4 LSBs of the DIN data if the 4 LSBs have a higher value. Also, when an 8-bit comparison is performed, the CMP-lo circuit can (using multiplexer circuit 2604d) overwrite the 4 LSBs of the DOUT data with the 4 LSBs of the DIN data only if the first control signal indicates that the 4 MSBs of the DOUT data have a higher value than the 4 MSBs of the DIN data, or if the second control signal indicates that the 4 MSBs are equal and the 4 LSBs of the DOUT data have a lower value than the 4 LSBs of the DIN data.

[0180] 27 shows a flowchart of a method 2700 of operation of a neural network processor. The method 2700 may be performed by various components of the neural network processor 502, such as the operation controller 522, the load / store controller 530, the operation engine 524, etc.

[0181] In operation 2702, the operation controller 522 receives a first instruction from the instruction buffer 520. An example syntax of the first instruction is illustrated in Figures 6A and 6B. The first instruction may include a sub-instruction 610 that indicates a fetch of an input data element into an input data register and a sub-instruction 608 that indicates a fetch of a weight element into a weight register.

[0182] In operation 2704, in response to a first instruction (e.g., subinstruction 610), load / store controller 530 fetches an input data element from a memory external to the neural network processor (e.g., memory 512) into an input data register (e.g., Din0 or Din1 of data register 528a) of the neural network processor. Load / store controller 530 may perform a read operation via memory interface 534 to fetch the input data element. The read address may be generated by address generator 532 and may be based on a circular addressing scheme such as described in FIGS. 8A-10B.

[0183] Also in operation 2706, in response to the first instruction (e.g., subinstruction 608), load / store controller 530 fetches weight elements from the neural network processor's weight buffer (e.g., weight and parameter buffer 526) to the neural network processor's weight register (e.g., weight-0 register or weight-1 register of weight / parameter registers 528b). The fetching of weight elements and input data elements may be performed in parallel.

[0184] In operation 2708, the computation controller 522 receives a second instruction from the instruction buffer 520. The second instruction may include sub-instructions 602 that indicate computation operations (e.g., MAC operations, post-processing operations, etc.) to be performed by the computation engine 524 on the weight elements stored in the weight registers and the input data elements stored in the input data registers.

[0185] At operation 2710, the computation engine may fetch the input data element and the weight element from the input data register and the weight register, respectively, in response to the second instruction.

[0186] In operation 2712, the computation engine may, in response to the second instruction, perform computation operations including MAC operations and post-processing operations on the input data elements and weight elements to generate output data elements.

[0187] In operation 2714, the computation engine may store the output data element in an output data register (eg, Dout).

[0188] 28 shows a flowchart of a method 2800 of operation of a neural network processor. The method 2800 may be performed by various components of the neural network processor 502, such as the arithmetic controller 522, the load / store controller 530, and the arithmetic engine 524, which includes the MAC engine 1300.

[0189] In operation 2802, the operation controller 522 receives a first indication of a particular input precision and a second indication of a particular weight precision. The first and second indications may be received from the configuration register 528c.

[0190] In operation 2804, the computation controller 522 configures the computation engines of the neural network processor based on the first instruction and the second instruction, such as by setting the weight multiplexer 1304 and the input data multiplexer 1306 to D4 / D8 mode, setting the configuration 1710 including (or excluding) binary mode for the computation units 1800 of the MAC engine 1300, etc.

[0191] In operation 2806, the operation controller 522 receives an instruction from the instruction buffer. The instruction may include sub-instructions 602 that indicate a set of MAC operations to be performed by the MAC engine 1300 on the weight elements stored in the weight registers and the input data elements stored in the input data registers.

[0192] In operation 2808, in response to the instruction, the computation engine configured based on the first and second instructions can fetch input data elements and weight elements from the input data registers and weight registers, respectively. As described above, depending on whether the weights are 4-bit or 8-bit precise and whether the input data elements are 4-bit or 8-bit precise, a D8 / D4 mode can be set, and the weight multiplexer 1304 and the data multiplexer 1306 can fetch weight elements of different bit precisions and input data elements of different bit precisions to the computation unit 1800, as shown in Figures 21A-21H. Depending on the bit precision, the weight multiplexer 1304 can provide different bits of the weight elements to different computation units, or provide different weight elements to different computation units. The weight multiplexer 1304 can also duplicate some of the bits in the D8 mode. Also, the data multiplexer 1306 can provide duplicates of the input data elements in the D4 mode.

[0193] In operation 2810, the arithmetic engine configured based on the first instruction and the second instruction can, in response to the instruction, perform a multiply-accumulate (MAC) operation between input data elements of a particular input precision and weight elements of a particular weight precision to generate an intermediate output data element. For example, in binary mode, the arithmetic engine can perform a bitwise XNOR operation between the input data elements and the weight elements. Also, for certain data and weight precisions (e.g., 8-bit input precision and 2-bit weight precision, 4-bit input precision and weight precision, etc.), each arithmetic unit can generate an intermediate output data element. For other data and weight precisions (e.g., 8-bit input precision and 4-bit weight precision, 8-bit input precision and 8-bit weight precision), the arithmetic engine can merge outputs from different arithmetic units using a data merge circuit (e.g., data merge circuit 2002) to generate an intermediate output data element, as shown in Figures 21D and 21E.

[0194] In operation 2812, the computation engine may store the intermediate output data element in an intermediate output data register (eg, a MAC register).

[0195] 29A and 29B show a flowchart of a method 2900 of operation of a neural network processor. Method 2900 may be performed by various components of neural network processor 502, such as, for example, arithmetic controller 522, load / store controller 530, and arithmetic engine 524, which includes MAC engine 1300.

[0196] At operation 2902, the operation controller 522 receives a first indication of a particular output precision and a second indication of a particular weight precision. The operation controller 522 may also receive a third indication of a particular input precision. The first, second, and third indications may be received from the configuration register 528c.

[0197] In operation 2904, the operation controller 522 configures the post-processing engine 1302 based on the first instruction and the second instruction (third instruction). Such configuration may be based on, for example, setting clamp high and clamp low values ​​based on the output precision (8 bits or 4 bits), configuring multiplexer circuit 2416a to route the output of adder 2408a to the input of adder 2408b and multiplexer circuit 2416b to route the output of adder 2408c to the input of adder 2408d to support 2-ternary weights, as shown in Figures 25A-25C, and configuring data routing circuit 2412 to route the output of data merge circuit 2002 to multipliers 2402a-d to support 8-bit input precision and 4-bit weight precision or 8-bit input precision and 8-bit weight precision.

[0198] At operation 2906, the operation controller 522 receives a first instruction from the instruction buffer. The first instruction may include sub-instructions 602 that indicate a set of MAC operations to be performed by the MAC engine 1300 on the weight elements stored in the weight registers and the input data elements stored in the input data registers.

[0199] In operation 2908, the MAC engine 1300 of the computation engine 524 can fetch input data elements and weight elements from the input data registers and weight registers, respectively, in response to the first instruction. In some examples, the computation engine 524 can also be configured based on the input precision and weight precision. As described above, depending on whether the weights are 4-bit or 8-bit precise and whether the input data elements are 4-bit or 8-bit precise, a D8 / D4 mode can be set, and the weight multiplexer 1304 and the data multiplexer 1306 can fetch weight elements of different bit precisions and input data elements of different bit precisions to the computation unit 1800, as shown in Figures 21A-21H. Depending on the bit precision, the weight multiplexer 1304 can provide different bits of the weight elements to different computation units or provide different weight elements to different computation units. The weight multiplexer 1304 can also duplicate some of the bits in the D8 mode. Additionally, the data multiplexer 1306 can provide a duplicate of the input data element in D4 mode.

[0200] In operation 2910, the MAC engine 1300 can, in response to an instruction, perform a multiply-accumulate (MAC) operation between input data elements of a particular input precision and weight elements of a particular weight precision to generate intermediate output data elements. For example, for binary mode, the computation engine can perform a bitwise XNOR operation between the input data elements and the weight elements. Also, for certain data and weight precisions (e.g., 8-bit input precision and 2-bit weight precision, 4-bit input precision and weight precision, etc.), each computation unit can generate an intermediate output data element. For other data and weight precisions (e.g., 8-bit input precision and 4-bit weight precision, 8-bit input precision and 8-bit weight precision), the computation engine can merge outputs from different computation units using a data merge circuit (e.g., data merge circuit 2002) to generate intermediate output data elements, as shown in Figures 21D and 21E.

[0201] At operation 2912, the MAC engine 1300 may store the intermediate output data element in an intermediate output data register (eg, a MAC register).

[0202] 29B, in operation 2914, the operation controller 522 receives a second instruction from the instruction buffer. The second instruction may include a sub-instruction 602 that indicates a post-processing operation (e.g., BNorm) to be performed by the post-processing engine 1302 on the intermediate output data.

[0203] In operation 2916, the post-processing engine 1302 configured based on the first instruction and the second instruction (and the third instruction) fetches the intermediate output data element from the intermediate output data register in response to the second instruction.

[0204] In operation 2918, the post-processing engine 1302 configured based on the first instruction and the second instruction (and the third instruction) performs post-processing operations, such as BNorm operations and residual layer processing, on the intermediate data elements to generate output data elements, as shown in Figures 24 and 25A-25C. Post-processing operations, including scaling, bit shifting, clamping, etc., may be performed in one clock cycle.

[0205] At operation 2920, the post-processing engine 1302 stores an output data element in an output data register (e.g., Dout) of the data registers 528a in response to the second instruction. The storing of the output data element may be performed by the data packing circuit 2304. Upon storing the threshold size of data (e.g., 32 bits) in Dout, the data packing circuit 2304 may send a control signal 2305 to the load / store controller 530 to fetch the output data into the memory 512.

[0206] In this description, the term "couple" may encompass a connection, communication, or signal path that enables a functional relationship consistent with this description. For example, if device A provides a signal to control device B to perform a certain action, (a) in a first example, device A is coupled to device B; or (b) in a second example, if intervening component C does not substantially change the functional relationship between device A and device B, device A is coupled to device B via an intervening component C, such that device B is controlled by device A via a control signal provided by device A. Also, in this description, a device "configured to" perform a certain task or function may be configured (e.g., programmed and / or hardwired) at the time of manufacture by a manufacturer to perform that function and / or may be configurable (or reconfigurable) by a user after manufacture to perform that function and / or other additional or alternative functions. Such configuration may be via firmware and / or software programming of the device, via the construction and / or layout of the device's hardware components and interconnections, or a combination thereof. Also, in this description, circuits or devices including certain components may instead be coupled to those components and adapted to form the described circuit element or device. For example, a structure described as including one or more semiconductor elements (e.g., transistors), one or more passive elements (e.g., resistors, capacitors, and / or inductors), and / or one or more sources (e.g., voltage and / or current sources) may instead include only the semiconductor elements in a single physical device (e.g., a semiconductor die and / or integrated circuit (IC) package) and be adapted to be coupled to at least some of the passive elements and / or sources, either during or after manufacture, by an end user, a third party, etc., to form the described structure.

[0207] Although specific transistor structures are mentioned above, other transistor or device structures may be substituted. For example, p-type MOSFETs may be substituted for n-type MOSFETs with little or no additional modification. Also, other types of transistors (such as bipolar transistors) may be utilized in place of the transistors shown. Capacitors may be implemented using different device structures (such as metal structures formed on top of each other to form a parallel plate capacitor) or may be formed on layers (metal or doped semiconductor) closer to or further from the semiconductor substrate surface.

[0208] As used above, the terms "terminal," "node," "interconnect," and "pin" are used interchangeably. Unless specifically stated to the contrary, these terms are used generally to refer to an interconnection between, or termination of, a device element, circuit element, integrated circuit, device, or other electronic or semiconductor component.

[0209] Although certain components may be described herein as being of a particular process technology, such components may be interchanged with components of other process technologies. Circuits described herein are reconfigurable to include the replaced component to provide functionality at least partially similar to that available prior to the component replacement. A component shown as a resistor, unless otherwise specified, generally represents any one or more elements coupled in series and / or parallel to provide the amount of impedance represented by the depicted resistor. For example, a resistor or capacitor shown and described herein as a single component may instead be multiple resistors or capacitors coupled in series or parallel between the same two nodes as the single resistor or capacitor, respectively. Additionally, the use of the phrase “ground terminal” in this description includes chassis ground, earth ground, floating ground, virtual ground, digital ground, common ground, and / or any other form of ground connection applicable or suitable to the teachings of this description. Unless otherwise specified, “about,” “approximately,” or “substantially” preceding a value means + / - 10 percent of the stated value.

[0210] Modifications in the described examples are possible, and other examples are possible, within the scope of the claims.

Claims

1. 1. A neural network processor, comprising: A memory interface; an instruction buffer; a weight buffer; an input data register; a weight register; an output data register; Computing engine and A controller; Including, The controller: receiving a first instruction from the instruction buffer; fetching input data elements from the memory interface into the input data registers and weight elements from the weight buffer into the weight registers in response to the first instruction; receiving a second instruction from the instruction buffer; In response to the second command, fetching the input data elements and the weight elements from the input data registers and the weight registers, respectively, into the computing engine; performing an arithmetic operation between the input data elements and the weight elements using the computing engine to generate output data elements; storing the output data element in the output data register; It is configured as follows: Neural network processor.

2. 2. The neural network processor of claim 1, wherein the controller: extracting from the first instruction a first sub-instruction indicating a fetch of the input data element into the input data register; extracting from the first instruction a second sub-instruction indicating fetching the weight elements from the weight buffer to the weight register; It is structured as follows: the controller is configured to fetch the input data element from the memory interface to the input data register in response to the first sub-instruction, and to fetch the weight element from the weight buffer to the weight register in response to the second sub-instruction; Neural network processor.

3. 3. The neural network processor of claim 2, wherein the controller is configured to fetch, in parallel, the input data elements from the memory interface to the input data registers and the weight elements from the weight buffer to the weight registers.

4. 3. The neural network processor of claim 2, the input data element is a first input data element; the input data register is a first input data register; the neural network processor further includes a second input data register; the second instruction includes a third sub-instruction indicating a fetch of a second input data element into the second input data register; the controller is configured to fetch the second input data element from the memory interface into the second input data register in response to the third sub-instruction; Neural network processor.

5. 5. The neural network processor of claim 4, wherein the controller is configured to perform the computational operations using the computing engine and fetch the second input data element from the memory interface in parallel.

6. 3. The neural network processor of claim 2, further comprising an input data address register configured to store an input data memory address; the first sub-instruction includes a reference to the input data address register; the controller is configured to perform a memory read operation at the input data memory address via the memory interface to fetch the input data element in response to the first sub-instruction; Neural network processor.

7. 7. The neural network processor of claim 6, the input data memory address is a first input data memory address, the first instruction includes a third sub-instruction, the third sub-instruction indicates that the first input data memory address is to be incremented or decremented; The controller: extracting the third sub-instruction from the first instruction; generating a second input data memory address by incrementing or decrementing the first input data memory address in response to the third sub-instruction; storing the second input data memory address in the input data address register to overwrite the first input data memory address; It is configured as follows: Neural network processor.

8. 8. The neural network processor of claim 7, wherein the controller is configured to generate the second input data memory address based on a starting input data memory address and an ending input data memory address, the starting input data memory address and the ending input data memory address defining an input data memory region.

9. 9. The neural network processor of claim 8, further comprising a configuration register configured to store an indication of an address management mode and the starting and ending input data memory addresses; the controller is configured to indicate a circular addressing mode in response to the instruction; If the second input data memory address is lower than the starting input data memory address, determining a first offset between the second input data memory address and the ending input data memory address; updating the second input data memory address based on the ending input data memory address and the first offset; storing the updated second input data memory address in the input data address register to overwrite the first input data memory address; If the second input data memory address is higher than the ending input data memory address, determining a second offset between the second input data memory address and the starting input data memory address; updating the second input data memory address based on the starting input data memory address and the second offset; storing the updated second input data memory address in the input data address register to overwrite the first input data memory address; if the second input data memory address is between the starting input data memory address and the ending input data memory address, storing the second input data memory address in the input data address register to overwrite the first input data memory address; It is configured as follows: Neural network processor.

10. 7. The neural network processor of claim 6, further comprising an output data address register configured to store an output data memory address; The controller: receiving a third instruction from the instruction buffer indicating writing the output data element from the output data register to memory via the memory interface and including a reference to the output data address register; performing a memory write operation at the output data memory address via the memory interface in response to the third instruction; It is configured as follows: Neural network processor.

11. 11. The neural network processor of claim 10, wherein the output data memory address is a first output data memory address, and the third instruction includes a third sub-instruction, the third sub-instruction indicating that the first output data memory address is to be incremented or decremented; The controller: extracting the third sub-instruction from the third instruction; generating a second output data memory address by incrementing or decrementing the first output data memory address in response to the third sub-instruction; storing the second output data memory address in the output data address register to overwrite the first output data memory address; It is configured as follows: Neural network processor.

12. 12. The neural network processor of claim 11, wherein the controller is configured to generate the second output data memory address based on a starting output data memory address and an ending output data memory address, the starting output data memory address and the ending output data memory address defining an output data memory area.

13. 13. The neural network processor of claim 12, further comprising a configuration register configured to store an indication of an address management mode, the starting output data memory address, and the ending output data memory address; the controller is configured to indicate a circular addressing mode in response to the instruction; If the second output data memory address is lower than the starting output data memory address, determining a first offset between the second output data memory address and the ending output data memory address; updating the second output data memory address based on the ending output data memory address and the first offset; storing the updated second output data memory address in the output data address register to overwrite the first output data memory address; If the second output data memory address is higher than the last output data memory address, determining a second offset between the second output data memory address and the starting output data memory address; updating the second output data memory address based on the starting output data memory address and the second offset; storing the updated second output data memory address in the output data address register to overwrite the first output data memory address; if the second output data memory address is between the starting output data memory address and the ending output data memory address, storing the second output data memory address in the output data address register to overwrite the first output data memory address; It is configured as follows: Neural network processor.

14. 12. The neural network processor of claim 11, wherein the controller is configured to generate the second output data memory address based on a starting input data memory address and an ending input data memory address, the starting input data memory address and the ending input data memory address defining an input data memory region.

15. 15. The neural network processor of claim 14, If the second output data memory address is lower than the starting input data memory address, determining a first offset between the second output data memory address and the ending input data memory address; updating the second output data memory address based on the ending input data memory address and the first offset; storing the updated second output data memory address in the output data address register to overwrite the first output data memory address; If the second output data memory address is higher than the ending input data memory address, determining a second offset between the second output data memory address and the starting input data memory address; updating the second output data memory address based on the starting input data memory address and the second offset; storing the updated second output data memory address in the output data address register to overwrite the first output data memory address; storing said second output data memory address in said input data address register to overwrite said first output data memory address if said second output data memory address is between said starting input data memory address and said ending input data memory address; It is configured as follows: Neural network processor.

16. 10. The neural network processor of claim 1, further comprising a configuration register configured to store a loop count value; The controller: receiving a third instruction from the instruction buffer, the third instruction including a loop start indicator and a reference to the configuration register; receiving a fourth instruction from the instruction buffer; Execute the fourth instruction; receiving a fifth instruction from the instruction buffer, the fifth instruction including an end loop indicator; repeating execution of the fourth instruction a number of times based on the loop count value in response to the end loop indicator; It is configured as follows: Neural network processor.

17. 2. The neural network processor of claim 1, wherein the first and second instructions are associated with first and second program count (PC) values, respectively, and the controller: receiving a third instruction from the instruction buffer, the third instruction including an end indicator; receiving a fourth instruction from the instruction buffer, the fourth instruction including a macroinstruction indicator, a macroinstruction PC value equal to the first PC value, and a repeat count value; executing the first and second instructions a number of times equal to the repeat count value in response to the macroinstruction indicator and the end indicator; It is configured as follows: Neural network processor.

18. 2. The neural network processor of claim 1, wherein the controller: receiving a third instruction from the instruction buffer; extracting a first instruction type indicator from the third instruction; extracting a first number of sub-instructions from the third instruction in response to the first instruction type indicator; receiving a fourth instruction from the instruction buffer; extracting a second instruction type indicator from the fourth instruction; extracting a second number of sub-instructions from the fourth instruction in response to the second instruction type indicator; wherein the first number and the second number are different. Neural network processor.

19. 20. The neural network processor of claim 18, a neural network processor, wherein the first instruction type and the second instruction type indicate different ones of a first type instruction including one sub-instruction, a second type instruction including two sub-instructions, a third type instruction including three sub-instructions, and a fourth type instruction including five sub-instructions.

20. 1. A method comprising: receiving a first instruction from an instruction buffer of the neural network processor; In response to the first command, fetching input data elements from a memory external to said neural network processor into input data registers of said neural network processor; fetching weight elements from a weight buffer of said neural network processor into a weight register of said neural network processor; receiving a second instruction from the instruction buffer; In response to the second command, fetching the input data elements and the weight elements from the input data registers and the weight registers, respectively, into a computing engine of the neural network processor; performing an arithmetic operation between the input data elements and the weight elements using the computing engine to generate output data elements; storing the output data elements in an output data register of the neural network processor; A method comprising:

21. 21. The method of claim 20, determining an input data memory address based on a circular addressing scheme; fetching the input data element from the memory by performing a memory read operation using the input data memory address; The method further comprises:

22. 21. The method of claim 20, determining an output data memory address based on a circular addressing scheme; storing the output data element in a memory by performing a memory write operation using the output data memory address; The method further comprises:

23. 1. A neural network processor, comprising: A memory interface; an instruction buffer; a weight buffer; an input data register; a weight register; an output data register; an address register; an address generation engine configurable to set input data addresses, output data addresses, and weight addresses in the address registers; a data load / store engine configurable to fetch input data from the memory interface to the input data register based on the input data address, fetch output data from the output data register to the memory interface based on the output data address, and fetch weights from the weight buffer to the weight register based on the weight address; a computing engine configurable to perform an operation based on the input data and the weights to generate the output data; A controller; Including, the controller responding to one or more instructions from the instruction buffer: extracting a first sub-instruction directed to the address generation engine to set one of the input data address or the output data address; extracting a second sub-instruction directed to the address generation engine to set the weighted address; extracting a third sub-instruction directed to the computing engine to perform the operation; extracting a fourth sub-instruction directed to the data load / store engine to fetch the weight at the weight address in the weight buffer; extracting a fifth sub-instruction directed to the data load / store engine to fetch the input data at the input data address via the memory interface; in parallel, configuring the address generation engine using the first and second sub-instructions, configuring the computing engine using the third sub-instruction, and configuring the data load / store engine using the fourth and fifth sub-instructions; It is configured as follows: Neural network processor.

24. 24. The neural network processor of claim 23, wherein the one or more instructions include a first instruction and a second instruction, the first instruction having a different bit length than the second instruction.

Images